爬取知乎热榜标题和连接 （python，requests，xpath）

用python爬取知乎的热榜，获取标题和链接。

环境和方法：ubantu16.04、python3、requests、xpath

1.用浏览器打开知乎，并登录

2.获取cookie和User—Agent

3.上代码

 import requests
 from lxml import etree

 def get_html(url):
     headers={
             'Cookie':'你的Cookie',
             #'Host':'www.zhihu.com',
             'User-Agent':'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.102 Safari/537.36'
             }

     r=requests.get(url,headers=headers)

     if r.status_code==200:
         deal_content(r.text)

 def deal_content(r):
     html = etree.HTML(r)
     title_list = html.xpath('//*[@id="TopstoryContent"]/div/section/div[2]/a/h2')
     link_list = html.xpath('//*[@id="TopstoryContent"]/div/section/div[2]/a/@href')
     for i in range(0,len(title_list)):
         print(title_list[i].text)
         print(link_list[i])
         with open("zhihu.txt",'a') as f:
             f.write(title_list[i].text+'\n')
             f.write('\t链接为：'+link_list[i]+'\n')
             f.write('*'*50+'\n')

 def main():
     url='https://www.zhihu.com/hot'
     get_html(url)

 main()

4.爬取结果