标签：python 22% detail 3A% url 源码 2C% infos 挑中
前言
老司机带你去看车，网上的几千条的二手车数据，只需几十行代码，就可以统统获取，保存数据到我们本地电脑上
知识点：
1.python基础知识
2.函数
3.requests库
4.xpath适合零基础的同学
环境：
windows + pycharm + python3
爬虫流程：
1.目标网址
2. 发送请求，获取响应
3. 解析网页 提取数据
4. 保存数据
加企鹅群695185429即可免费获取，资料全在群文件里。资料可以领取包括不限于Python实战演练、PDF电子文档、面试集锦、学习资料等
步骤：
1.导入工具
import io import sys import requests # pip install requests from lxml import etree # pip
2.获取汽车详情页面的url，解析网站
def get_detail_urls(url):
# 目标网址
# url = 'https://www.guazi.com/cs/buy/o3/'
# 发送请求，获取响应
resp = requests.get(url,headers=headers)
text = resp.content.decode('utf-8')
# 解析网页
html = etree.HTML(text)
ul = html.xpath('//ul[@class="carlist clearfix js-top"]')[0]
# print(ul)
lis = ul.xpath('./li')
detail_urls = []
for li in lis:
detail_url = li.xpath('./a/@href')
# print(detail_url)
detail_url = 'https://www.guazi.com' + detail_url[0]
# print(detail_url)
detail_urls.append(detail_url)
return detail_urls
3.添加请求头
headers = {
'User-Agent':'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.130 Safari/537.36',
'Cookie':'uuid=5a823c6f-3504-47a9-8360-f9a5040e5f23; ganji_uuid=4238534742401031078259; lg=1; Hm_lvt_936a6d5df3f3d309bda39e92da3dd52f=1590045325; track_id=79952087417704448; antipas=q7222002m3213k0641719; cityDomain=cs; clueSourceCode=%2A%2300; user_city_id=204; sessionid=38afa34e-f972-431b-ce65-010f82a03571; close_finance_popup=2020-05-23; cainfo=%7B%22ca_a%22%3A%22-%22%2C%22ca_b%22%3A%22-%22%2C%22ca_s%22%3A%22pz_baidu%22%2C%22ca_n%22%3A%22pcbiaoti%22%2C%22ca_medium%22%3A%22-%22%2C%22ca_term%22%3A%22-%22%2C%22ca_content%22%3A%22%22%2C%22ca_campaign%22%3A%22%22%2C%22ca_kw%22%3A%22-%22%2C%22ca_i%22%3A%22-%22%2C%22scode%22%3A%22-%22%2C%22keyword%22%3A%22-%22%2C%22ca_keywordid%22%3A%22-%22%2C%22ca_transid%22%3A%22%22%2C%22platform%22%3A%221%22%2C%22version%22%3A1%2C%22track_id%22%3A%2279952087417704448%22%2C%22display_finance_flag%22%3A%22-%22%2C%22client_ab%22%3A%22-%22%2C%22guid%22%3A%225a823c6f-3504-47a9-8360-f9a5040e5f23%22%2C%22ca_city%22%3A%22cs%22%2C%22sessionid%22%3A%2238afa34e-f972-431b-ce65-010f82a03571%22%7D; preTime=%7B%22last%22%3A1590217273%2C%22this%22%3A1586866452%2C%22pre%22%3A1586866452%7D',
}
4.提取每辆汽车详情页面的数据
def parse_detail_page(url):
resp = requests.get(url,headers=headers)
text = resp.content.decode('utf-8')
html = etree.HTML(text)
# 标题
title = html.xpath('//div[@class="product-textbox"]/h2/text()')[0]
title = title.strip()
print(title)
# 信息
info = html.xpath('//div[@class="product-textbox"]/ul/li/span/text()')
# print(info)
infos = {}
cardtime = info[0]
km = info[1]
displacement = info[2]
speedbox = info[3]
infos['title'] = title
infos['cardtime'] = cardtime
infos['km'] = km
infos['displacement'] = displacement
infos['speedbox'] = speedbox
print(infos)
return infos
5.保存数据
def save_data(infos, f):
f.write('{},{},{},{},{}\n'.format(infos['title'],infos['cardtime'],infos['km'],infos['displacement'],infos['speedbox']))
if __name__ == '__main__':
base_url = 'https://www.guazi.com/cs/buy/o{}/'
with open('guazi.csv','a',encoding='utf-8') as f:
for x in range(1,51):
url = base_url.format(x)
detail_urls = get_detail_urls(url)
for detail_url in detail_urls:
infos = parse_detail_page(detail_url)
save_data(infos, f)
最后运行代码，效果如下图
标签：python,22%,detail,3A%,url,源码,2C%,infos,挑中 来源： https://www.cnblogs.com/python0921/p/12968265.html
本站声明： 1. iCode9 技术分享网（下文简称本站）提供的所有内容，仅供技术学习、探讨和分享； 2. 关于本站的所有留言、评论、转载及引用，纯属内容发起人的个人观点，与本站观点和立场无关； 3. 关于本站的所有言论和文字，纯属内容发起人的个人观点，与本站观点和立场无关； 4. 本站文章均是网友提供，不完全保证技术分享内容的完整性、准确性、时效性、风险性和版权归属；如您发现该文章侵犯了您的权益，可联系我们第一时间进行删除； 5. 本站为非盈利性的个人网站，所有内容不会用来进行牟利，也不会利用任何形式的广告来间接获益，纯粹是为了广大技术爱好者提供技术内容和技术思想的分享性交流网站。