python3爬虫环境搭建
安装python3
sudo apt-get install python3-dev build-essential libssl-dev libffi-dev libxml2 libxml2-dev libxslt1-dev zlib1g-dev
sudo apt-get install python3
sudo apt-get install ptyhon3-pip
ubuntu安装mongo
sudo apt-get install mongodb
mongod
mongo
> show dbs
admin (empty)
local 0.078GB
> use local
switched to db local
> db.test.insert({'a':'b'})
WriteResult({ "nInserted" : 1 })
mac安装mongo
brew install mongodb
brew services start mongodb
ubuntu安装redis
sudo apt-get install redis-server
redis-cli
127.0.0.1:6379> set 'a' 'b'
OK
127.0.0.1:6379> get 'a'
"b"
127.0.0.1:6379>
root@iZwz91u9ywulp6n78yyed6Z:~# sudo vi /etc/redis/redis.conf
requirepass myredisserver
sudo service redis restart
root@iZwz91u9ywulp6n78yyed6Z:~# redis-cli -a myredisserver
127.0.0.1:6379> get 'a'
"b"
127.0.0.1:6379>
mac 安装redis
brew install redis
qunzhudeMacBook-Air:/ qunzhupu$ brew services start redis
==> Successfully started `redis` (label: homebrew.mxcl.redis)
qunzhudeMacBook-Air:/ qunzhupu$ redis-cli
127.0.0.1:6379> set 'a' 'b'
OK
127.0.0.1:6379> get 'a'
"b"
cd /usr/local/etc
sudo vim redis.conf
requirepass foobared
ubuntu配置mysql
root@iZwz91u9ywulp6n78yyed6Z:~# cd /etc/mysql/
root@iZwz91u9ywulp6n78yyed6Z:/etc/mysql# ls
conf.d debian.cnf debian-start my.cnf my.cnf.fallback mysql.cnf mysql.conf.d
root@iZwz91u9ywulp6n78yyed6Z:/etc/mysql# cd mysql.conf.d/
root@iZwz91u9ywulp6n78yyed6Z:/etc/mysql/mysql.conf.d# ls
mysqld.cnf mysqld_safe_syslog.cnf
root@iZwz91u9ywulp6n78yyed6Z:/etc/mysql/mysql.conf.d# vi mysqld.cnf
root@iZwz91u9ywulp6n78yyed6Z:/etc/mysql/mysql.conf.d# sudo service mysql restart
GRANT ALL PRIVILEGES ON *.* TO 'root'@'%' IDENTIFIED BY '123456' WITH GRANT OPTION;
FLUSH PRIVILEGES;
selenium库使用
>>> import selenium
>>> from selenium import webdriver
>>> driver - webdriver.Chrome()
>>> driver.get('http://www.baidu.com')
安装beautifulsoup4和pyquery
pip3 install beautifulsoup4
pip3 install pyquery
qunzhudeMacBook-Air:bin qunzhupu$ python3
Python 3.7.0 (default, Jun 29 2018, 20:13:13)
[Clang 9.1.0 (clang-902.0.39.2)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from pyquery import PyQuery as pq
>>> dec =pq('<html></html>')
>>> dec =pq('<html>Hello</html>')
>>> result = dec('html').text()
>>> result
'Hello'
>>>
pymysql连接Mysql
qunzhudeMacBook-Air:bin qunzhupu$ python3
Python 3.7.0 (default, Jun 29 2018, 20:13:13)
[Clang 9.1.0 (clang-902.0.39.2)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import pymysql
>>> conn = pymysql.connect(host='123.206.204.80',user='puqunzhu',password='123456',port=3306,db='puqunzhu')
>>> cursor = conn.cursor()
>>> cursor.execute('select * from article')
1
pymongo连接mongoDB
pip3 install pymongo
qunzhudeMacBook-Air:bin qunzhupu$ python3
Python 3.7.0 (default, Jun 29 2018, 20:13:13)
[Clang 9.1.0 (clang-902.0.39.2)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import pymongo
>>> client = pymongo.MongoClient('localhost')
>>> db = client['newtestdb']
>>> db['table'].insert({'name':'Bob'})
__main__:1: DeprecationWarning: insert is deprecated. Use insert_one or insert_many instead.
ObjectId('5bc5b059e18a3620cc46bdb4')
>>> db['table'].find_one({'name':'Bob'})
{'_id': ObjectId('5bc5b059e18a3620cc46bdb4'), 'name': 'Bob'}
redis连接redis
pip3 install redis
qunzhudeMacBook-Air:bin qunzhupu$ python3
Python 3.7.0 (default, Jun 29 2018, 20:13:13)
[Clang 9.1.0 (clang-902.0.39.2)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import redis
>>> r = redis.Redis('localhost',6379)
>>> r.set('name','puqunzhu')
True
>>> r.get('name')
b'puqunzhu'
flask代理获取
pip3 install flask
django
pip3 install django
jupyter
pip3 install jupyter
qunzhudeMacBook-Air:bin qunzhupu$ jupyter notebook
python3爬虫环境搭建的更多相关文章
- selenium+python爬虫环境搭建
前言: 准备使用selenium爬取网站数据,先搭建selenium+python爬虫环境搭建 系统环境: 64位win10系统,同时装python2.7和python3.6两个版本,IDE为pych ...
- Python基础之Windows下Python3.x环境搭建
本文介绍使用Python3.x搭建Python环境,文章最后将使用Windows自带的cmd写一个简单的Python程序. 下载Python环境安装包 在https:/www.python.org/下 ...
- python3之环境搭建以及基础语法
早些时候,有一段时间吧,为了想免费下载网易云的收费音乐,也是翻了下python的教程的,语法方面没细致地去看,跟java.php.javascript这些都大同小异,也是面向对象的.那么,近期准备快速 ...
- 云效（阿里云）流水线 + nginx + uWsgi + flask + python3 基础环境搭建 --备忘
一.开发环境搭建 1.安装python3 yum -y groupinstall "Development tools" yum -y install zlib-devel bzi ...
- Python3+selenium3环境搭建笔记
系统:win7 64位浏览器:ie9 64位 chrome70 32位 firefox63 64位python版本:3.6.5 Windows x86 executable installersele ...
- 2 Selenium3.0+Python3.6环境搭建
[说明] 再次搭建一次环境,是因为遇到怎么都打不开IE的问题了,环境信息为:Selenium3.0+Python3.6+win7+ie10 [搭建步骤] 1.下载Python3.6,并点击安装和配置环 ...
- 金玉良缘易配而木石前盟难得|M1 Mac os(Apple Silicon)天生一对Python3开发环境搭建(集成深度学习框架Tensorflow/Pytorch)
原文转载自「刘悦的技术博客」https://v3u.cn/a_id_189 笔者投入M1的怀抱已经有一段时间了,俗话说得好,但闻新人笑,不见旧人哭,Intel mac早已被束之高阁,而M1 mac已经 ...
- python——python3.6环境搭建（Windows10,64位）
1.python软件资源下载 1.1 打开python官网地址:https://www.python.org 1.2 根据自己电脑的设置选择下载合适的python3.6.2 1.3 此处选择windo ...
- vs2015 + Python3.5 环境搭建
1. vs2015只支持Python3.5及以前的版本,对应Anaconda3.4.2之前的版本. 2. 卸载掉所有安装过的Python 3. 建议重装VS2015, 因为增量升级Python Too ...
随机推荐
- I don't Blame You that You don't Understand Me
I don't Blame You that You don't Understand Me Every one has a dead corner in himself, with no entry ...
- (转载)非常完善的Log4net配置详细说明
一.前言 在项目中,对项目的日志收集是非常重要的,这里我就说说代码的异常日志收集.收集异常日志,有很多第三方成熟的框架,我这里就介绍一下我常用的Log4net. Log4Net介绍 Log4net 是 ...
- IntelliJ IDEA 设置 vue 支持
一.IntelliJ IDEA支持.vue文件 安装vue.js file --> settings --> plugins,输入vue,点击搜索结果里的vue.js右边的install按 ...
- 利用HBuilder打包Vue开发的webapp为app
众所周知,前端开发完成的webapp只能运行在浏览器上,对运行环境有一定的限制,也就是除了浏览器其他的环境下不支持:那么现在如果有个需求是这样的呢?需要一套代码三端运行呢?三端运行(黑人脸??)是的, ...
- SqlServer还原数据库时提示：异常终止，不能在此版本的SQL Server中启动，因为它包含分区函数
场景 在SqlServer Management中进行数据库还原时提示: 数据库不能在此版本的SQL Server中启动,因为它包含分区函数. 点击左下角的查看详细信息 实现 电脑上安装的是SQL S ...
- Winforn中实现ZedGraph自定义添加右键菜单项(附源码下载)
场景 Winform中实现ZedGraph中曲线右键显示为中文: https://blog.csdn.net/BADAO_LIUMANG_QIZHI/article/details/100115292 ...
- Linux 笔记 - 第十三章 Linux 系统日常管理之（一）系统状态监控
博客地址:http://www.moonxy.com 一.前言 如果你是一名 Linux 运维人员,最主要的工作是优化系统配置,使应用在系统上以最优的状态运行.系统运行状态主要包括:系统负载.内存状态 ...
- mybatis中Insert后主键返回
1.Mapper的写法,返回的这个int是受影响的行号 int insertNewUser(User newUser); 2.xml的写法 <!--返回主键 形式1 --> <ins ...
- Django+MySQL Dashboard 网页端数据库可视化
1. Overview Python+MySQL+Django, 有些数据托管在 MySQL 的数据库,然后我们希望进行可视化,通过 web 的方式对数据库的信息去进行展示/检索/维护/.. 这个项目 ...
- Splitting into digits CodeForce#1104A
题目链接:Splitting into digits 题目原文 Vasya has his favourite number