python3爬虫入门程序

适用于有且只有一点Python3和网页基础的朋友,大牛&路人请绕道

(本文很多废话,第一次在网上长篇大论,所以激动的停不下来,如果有大佬路过,也希望不要直接绕道,烦请指点一二)

感谢博客园给了我一个机会,我喜欢的id还没有被抢注,真的是太可怕了

*注:这是一段废话,正文请直接跳过这一段. 大二的时候因为爱好,自己学了点python(当初学主要是因为语法简洁美观,还没有大括号,代码对齐?反正java代码也要对齐啊~),还好我学python的时候py3已经流行起来了,没有学py2,不然又得好一阵折腾. *

写这个代码的背景:记得高二的时候逛骗子网站,出于好奇在网站上留下了同桌的手机号,结果这都过了大概大半年了吧,同桌还是接二连三的可以收到骗子的电话,但是原来的网址已经找不到了,于是就在百度随便搜索了关键词"牛股"挨个查看的,只找到了一个可以输入手机号的,剩下的都是让加微信的,本来想整整我们老师呢,想想还是免了,干脆往他们数据库填点东西玩吧,其实最终是否成功我也不能确定

• 操作环境
• win10 1803 64位
• Chrome 68.0.3440.106（正式版本） （64 位）
• pycharm-UI(pycharm专业版) 2018.2
• python-365
• 库(非自带库用pip直接安装就行):
  • pymysql :import pymysql
  • requests :import requests
  • json(自带) :import json
  • Faker: :from faker import Faker

首先选取目标

目标网站是这个,url为:http://gpyd.gp241.com/nyqpc/bd2.html?id=20110052,

1.首先肯定是抓取一下post/get地址

进入首页后点击"点击领取9月牛股"弹出对话框后,按F12弹出开发者工具



在开发者工具中选中"Network",随后点击网页中的点击领取,会看到network中多出来一条文件信息

然后提取一下我们需要的数据放到pycharm中,并整理成这种json格式:

2.这样我们就得到了这些数据:

url = r"https: // download.zslxt.com / tinterface.php"
headers = {
    "Host": "download.zslxt.com",
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36",
    "Accept": "*/*",
    "Accept-Language": "zh-CN,zh;q=0.8,zh-TW;q=0.7,zh-HK;q=0.5,en-US;q=0.3,en;q=0.2",
    "Accept-Encoding": "gzip, deflate, br",
    "Referer": "http:/gpyd.gp241.com/nyqpc/bd2.html?id=20110052",
    "Content-Type": "application/x-www-form-urlencoded; charset=UTF-8",
    "Content-Length": "107",
    "Origin": "http://gpyd.gp241.com",
    "Connection": "keep-alive"
}
data = {
    "bm": "gbk",
    "gpdm": "",
    "id": "20110052",
    "phone": "15666668888",
    "qudao": 98,
    "remarks": "牛有圈百度2)"
}

这里看data的参数也应该明白了,这里就是我们刚才输入的手机号了,别的代码可以不动,我刚才切换浏览器发现并没有影响,不知道是怎么来的,可能是跟百度推广有关吧

然后,这样之后就可以向网站发送一条数据了

首先我们要使用requests库,这里就不介绍了,是一个可以用来请求get/post...还可以使用session保持登陆,用途很广

这里有一点需要注意,就是data数据不可以直接传送,需要用json.dumps()方法转为字符串

requests可以返回请求数据,这段代码并没有体现出来,但是请不要被误导

import requests
import json

url = r"https: // download.zslxt.com / tinterface.php"
headers = {
    "Host": "download.zslxt.com",
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36",
    "Accept": "*/*",
    "Accept-Language": "zh-CN,zh;q=0.8,zh-TW;q=0.7,zh-HK;q=0.5,en-US;q=0.3,en;q=0.2",
    "Accept-Encoding": "gzip, deflate, br",
    "Referer": "http:/gpyd.gp241.com/nyqpc/bd2.html?id=20110052",
    "Content-Type": "application/x-www-form-urlencoded; charset=UTF-8",
    "Content-Length": "107",
    "Origin": "http://gpyd.gp241.com",
    "Connection": "keep-alive"
}
data = {
    "bm": "gbk",
    "gpdm": "",
    "id": "20110052",
    "phone": "15666668888",
    "qudao": 98,
    "remarks": "牛有圈百度2)"
}

requests.post(url=url, headers=headers, data=json.dumps(data))

可是,总不能只发送一次吧

这里先介绍一下python中最假的库--Faker

其实这个库的"造假"功能出乎意料的强大,有兴趣的可以去了解一下

在这个例子中大概只需要两个功能:生成随机user-agent和手机号码(甚至这个网站也没必要随机user-agent,因为我没有使用代理ip提交了大概两千条数据,都没有被封)

这样之后,我们的代码就学会了一点伪装的皮毛

(这里插个题外话,那天在某论坛看到一个朋友问为什么一直在更换代理还是被封号了,,,当时我用的手机也没有哪个论坛的帐号因此不方便回复他,同一个账号一直在更换ip这种行为不正常吖)

这样之后我们的代码便成了如下代码:

import requests
import json
from faker import Faker

f = Faker(locale="zh-CN")

user_agent = f.user_agent()
phone = f.phone_number()
url = r"https: // download.zslxt.com / tinterface.php"
headers = {
    "Host": "download.zslxt.com",
    "User-Agent": user_agent,
    "Accept": "*/*",
    "Accept-Language": "zh-CN,zh;q=0.8,zh-TW;q=0.7,zh-HK;q=0.5,en-US;q=0.3,en;q=0.2",
    "Accept-Encoding": "gzip, deflate, br",
    "Referer": "http:/gpyd.gp241.com/nyqpc/bd2.html?id=20110052",
    "Content-Type": "application/x-www-form-urlencoded; charset=UTF-8",
    "Content-Length": "107",
    "Origin": "http://gpyd.gp241.com",
    "Connection": "keep-alive"
}
data = {
    "bm": "gbk",
    "gpdm": "",
    "id": "20110052",
    "phone": phone,
    "qudao": 98,
    "remarks": "牛有圈百度2)"
}

req = requests.post(url=url, headers=headers, data=json.dumps(data))

然后就快要完成了,为了方便循环发送数据,我们再把它整理成一段函数:

其实我一开始学python真的不喜欢写函数,毕竟那么两行代码就能写完了,包装成一个函数简直就是在凑代码行数,毫无用途,但是我今天看到了一个故事:

为了检测空的奶盒子,博士后和农民用两种方式解决了这个问题:发明一台机器,使用了一台风扇

但是很多时候我们新学东西时遇到的问题都可以用以前就会的方法解决这个问题,但是随着问题的深入,有时候就只能使用新学的只是来解决以后遇到的问题了,写写函数(包装成类)总是没错的,前提是这个代码你是用来练手的,而不是用来应急的.

import requests
import json
from faker import Faker

f = Faker(locale="zh-CN")

def duang():
    user_agent = f.user_agent()
    phone = f.phone_number()
    url = r"https: // download.zslxt.com / tinterface.php"
    headers = {
        "Host": "download.zslxt.com",
        "User-Agent": user_agent,
        "Accept": "*/*",
        "Accept-Language": "zh-CN,zh;q=0.8,zh-TW;q=0.7,zh-HK;q=0.5,en-US;q=0.3,en;q=0.2",
        "Accept-Encoding": "gzip, deflate, br",
        "Referer": "http:/gpyd.gp241.com/nyqpc/bd2.html?id=20110052",
        "Content-Type": "application/x-www-form-urlencoded; charset=UTF-8",
        "Content-Length": "107",
        "Origin": "http://gpyd.gp241.com",
        "Connection": "keep-alive"
    }
    data = {
        "bm": "gbk",
        "gpdm": "",
        "id": "20110052",
        "phone": phone,
        "qudao": 98,
        "remarks": "牛有圈百度2)"
    }

    req = requests.post(url=url, headers=headers, data=json.dumps(data))
	return user_agent, phone, req

这样我们就可以方便的进行调用了,写个main函数来调用它

import requests
import json
from faker import Faker

f = Faker(locale="zh-CN")

def duang():
    user_agent = f.user_agent()
    phone = f.phone_number()
    url = r"https: // download.zslxt.com / tinterface.php"
    headers = {
        "Host": "download.zslxt.com",
        "User-Agent": user_agent,
        "Accept": "*/*",
        "Accept-Language": "zh-CN,zh;q=0.8,zh-TW;q=0.7,zh-HK;q=0.5,en-US;q=0.3,en;q=0.2",
        "Accept-Encoding": "gzip, deflate, br",
        "Referer": "http:/gpyd.gp241.com/nyqpc/bd2.html?id=20110052",
        "Content-Type": "application/x-www-form-urlencoded; charset=UTF-8",
        "Content-Length": "107",
        "Origin": "http://gpyd.gp241.com",
        "Connection": "keep-alive"
    }
    data = {
        "bm": "gbk",
        "gpdm": "",
        "id": "20110052",
        "phone": phone,
        "qudao": 98,
        "remarks": "牛有圈百度2)"
    }

    req = requests.post(url=url, headers=headers, data=json.dumps(data))
    return user_agent, phone, req

if __name__ == '__main__':
    for i in range(100000):
        user_agent, phone, req = duang()
        print(i, '\t', phone, '\t', req.status_code, '\n', user_agent)

这里就是输出一下信息啦,刚才出去吃饭的时候断网了,只跑了3000多,这里就不截图了(如果真有用来练手的朋友可以尝试自己完善一下代码,断网后也可以等待并继续执行)

附上全部代码(写到mysql了):

import pymysql
import requests
import json
from faker import Faker

f = Faker(locale="zh-CN")

def duang():
    user_agent = f.user_agent()
    phone = f.phone_number()
    url = r"https: // download.zslxt.com / tinterface.php"
    headers = {
        "Host": "download.zslxt.com",
        "User-Agent": user_agent,
        "Accept": "*/*",
        "Accept-Language": "zh-CN,zh;q=0.8,zh-TW;q=0.7,zh-HK;q=0.5,en-US;q=0.3,en;q=0.2",
        "Accept-Encoding": "gzip, deflate, br",
        "Referer": "http:/gpyd.gp241.com/nyqpc/bd2.html?id=20110052",
        "Content-Type": "application/x-www-form-urlencoded; charset=UTF-8",
        "Content-Length": "107",
        "Origin": "http://gpyd.gp241.com",
        "Connection": "keep-alive"
    }
    data = {
        "bm": "gbk",
        "gpdm": "",
        "id": "20110052",
        "phone": phone,
        "qudao": 98,
        "remarks": "牛有圈百度2)"
    }

    req = requests.post(url=url, headers=headers, data=json.dumps(data))
    return user_agent, phone, req.status_code

if __name__ == '__main__':
    for i in range(100000):
        user_agent, phone, status_code = duang()
        db = pymysql.connect("localhost", "root", "xiaoyan", "python")
        cur = db.cursor()
        cur.execute(f"INSERT INTO python1duang VALUES(default,'{user_agent}','{phone}','{status_code}')")
        db.commit()
        print(i, '\t', phone, '\t', status_code, '\n', user_agent)
    db.close()