Scrapy Shell Commands
Fetch Command
fetch('http://chaistudy.com')
We can also fetch the website using the below command, it will be same as the above fetch command
scrapy shell "yourwebsite.com"
Response, response command is to get the response from the scrapped content.
response.xpath() or response.css()
Number of items in response
len(response.xpath( '//*[@id="your_css_id"]'))
While using Python scrapy, the xpath is the best option and when we want to fetch the id or class of css, use the below format.
ID:
response.xpath('//*[@id="your_css_id"]')
Class:
response.xpath('//*[@class="your_css_class_name"]')
To Get the text,
response.xpath('//*[@class="your_css_class_name"]/text()')
Extract all data,
response.xpath('//*[@id="your_css_id"]/text()').extract()
response.xpath('//*[@id="your_css_id"]/text()').extract_all()
Extract First Element,
response.xpath('//*[@id="your_css_id"]/text()').extract_first()