标签：多种 网页 rqq text li soup html import 解析
1.使用正则表达式获取网页标题内容
In [ ]:
import requests
import re
url = 'http://www.tipdm.com/tipdm/index.html'
rqq = requests.get(url)
rqq.encoding = 'utf-8'
rqq.text
In [ ]:
re.findall('<li><a href="[a-z0-9.:/]+" target=".*">(.+)</a></li>', rqq.text)
In [ ]:
re.findall('<li[ a-z="]*><a[a-z0-9 = "]* href="[a-z0-9.:/]+".*>(.+)</a></li>', rqq.text)
2.使用XPath解析网页
In [ ]:
import requests
from lxml import etree
url = 'http://www.tipdm.com/tipdm/index.html'
rqq = requests.get(url)
html = etree.HTML(rqq.content, etree.HTMLParser())
html
In [ ]:
etree.tostring(html, encoding='utf-8').decode('utf-8')
rqq.encoding = 'utf-8'
rqq.text
In [ ]:
html.xpath('/html/head/title')
In [ ]:
html.xpath('//title')
In [ ]:
html.xpath('/html/body/header/div/nav/ul/li/a/text()')
html.xpath('//header/div/nav/ul/li/a/text()')
html.xpath('//*[@id="menu"]/li/a/text()')
In [ ]:
html.xpath('/html/body/header/div/nav/ul/li[last()]')
In [ ]:
html.xpath('/html/body/header/div/nav/ul/li/a[@target="_blank"]/text()')
In [ ]:
import requests
from lxml import etree
url = 'https://weixin.sogou.com/'
rqq = requests.get(url)
In [ ]:
html = etree.HTML(rqq.content, etree.HTMLParser(encoding='utf-8'))
html.xpath('//*[@id="topwords"]/li/a[@title]/text()')
In [ ]:
[html.xpath('//*[@id="topwords"]/li['+str(i)+']/a[@title]/text()') for i in range(1, 11)]
html.xpath('//*[@id="topwords"]/li/a[@title]/text()')
In [ ]:
print(rqq.text[:20])
print(rqq.content[:20])
3.使用Beautiful Soup解析网页
In [ ]:
import requests
from bs4 import BeautifulSoup
rqq = requests.get('http://www.tipdm.com/tipdm/index.html')
soup = BeautifulSoup(rqq.content, 'lxml')
In [ ]:
soup.head
soup.body
soup.li
soup.find_all('li')
In [ ]:
a = soup.link
a.name
a.attrs
a['href']
In [ ]:
a = soup.find_all('nav')
for i in a[0].find_all('li'):
print(i.string)
In [ ]:
a=soup.select('html > head > title')[0]
a.text
In [ ]:
soup.select('.menu > li') # class
soup.select('#menu > li') # id
[i.text for i in soup.select('.menu > li')]
In [ ]:
soup.select('#menu > li > a')
In [ ]:
from bs4 import BeautifulSoup
import requests
rqq = requests.get('https://weixin.sogou.com/')
soup = BeautifulSoup(rqq.content, 'lxml')
soup.select('#topwords')
[i.text for i in soup.select('.hot-news > li > a')]
[i['title'] for i in soup.select('.hot-news > li > a')]
In [ ]:
a = soup.find_all(id='topwords')[0]
[i.text for i in a.find_all('a')]
[i.string for i in a.find_all('a')]
4.参考文章
【创作不易，望点赞收藏，若有疑问，请留言，谢谢】
标签：多种,网页,rqq,text,li,soup,html,import,解析 来源： https://www.cnblogs.com/dongxuelove/p/16434804.html
本站声明： 1. iCode9 技术分享网（下文简称本站）提供的所有内容，仅供技术学习、探讨和分享； 2. 关于本站的所有留言、评论、转载及引用，纯属内容发起人的个人观点，与本站观点和立场无关； 3. 关于本站的所有言论和文字，纯属内容发起人的个人观点，与本站观点和立场无关； 4. 本站文章均是网友提供，不完全保证技术分享内容的完整性、准确性、时效性、风险性和版权归属；如您发现该文章侵犯了您的权益，可联系我们第一时间进行删除； 5. 本站为非盈利性的个人网站，所有内容不会用来进行牟利，也不会利用任何形式的广告来间接获益，纯粹是为了广大技术爱好者提供技术内容和技术思想的分享性交流网站。