- Python Web Scraping Cookbook
- Michael Heydt
- 257字
- 2021-06-30 18:44:03
How to do it...
We start by importing Selector from scrapy, and also requests so that we can retrieve the page:
In [1]: from scrapy.selector import Selector
...: import requests
...:
Next we load the page. For this example we are going to retrieve the most recent questions on StackOverflow and extract their titles. We can make this query with the the following:
In [2]: response = requests.get("http://stackoverflow.com/questions")
Now create a Selector and pass it the response object:
In [3]: selector = Selector(response)
...: selector
...:
Out[3]: <Selector xpath=None data='<html>\r\n\r\n <head>\r\n\r\n <title>N'>
Examining the content of this page we can see that questions have the following structure to their HTML:
The HTML of a StackOverflow Question
With the selector we can find these using XPath:
In [4]: summaries = selector.xpath('//div[@class="summary"]/h3')
...: summaries[0:5]
...:
Out[4]:
[<Selector xpath='//div[@class="summary"]/h3' data='<h3><a href="/questions/48353091/how-to-'>,
<Selector xpath='//div[@class="summary"]/h3' data='<h3><a href="/questions/48353090/move-fi'>,
<Selector xpath='//div[@class="summary"]/h3' data='<h3><a href="/questions/48353089/java-la'>,
<Selector xpath='//div[@class="summary"]/h3' data='<h3><a href="/questions/48353086/how-do-'>,
<Selector xpath='//div[@class="summary"]/h3' data='<h3><a href="/questions/48353085/running'>]
And now we drill a little further into each to get the title of the question.
In [5]: [x.extract() for x in summaries.xpath('a[@class="question-hyperlink"]/text()')][:10]
Out[5]:
['How to convert stdout binary file to a data URL?',
'Move first letter from sentence to the end',
'Java launch program and interact with it programmatically',
'How do I build vala from scratch',
'Running Sql Script',
'Mysql - Auto create, update, delete table 2 from table 1',
'how to map meeting data corresponding calendar time in java',
'Range of L*a* b* in Matlab',
'set maximum and minimum number input box in js,html',
'I created generic array and tried to store the value but it is showing ArrayStoreException']