尝试刮取snapdeal数据时的输出如下：
scrapy shell "https://www.snapdeal.com"
response.text
u'<HTML><HEAD>\n<TITLE>Access Denied</TITLE>\n</HEAD><BODY>\n<H1>Access Denied</H1>\n \nYou don\'t have permission to access "http://www.snapdeal.com/" on this server.<P>\nReference #18.1dd70b17.1514632273.17456300\n</BODY>\n</HTML>\n'
有帮助吗？
如果我使用User-Agent然后我得到正确的页面
scrapy shell
fetch("https://www.snapdeal.com", headers={'User-Agent': "Mozilla/5.0"})
response.text
或者使用脚本
import scrapy
#from scrapy.commands.view import open_in_browser
class MySpider(scrapy.Spider):
name = 'myspider'
start_urls = ['https://www.snapdeal.com/']
def parse(self, response):
print('url:', response.url)
#open_in_browser(response)
for item in response.xpath('//*[@class="catText"]/text()').extract():
print(item)
# --- it runs without project ---
from scrapy.crawler import CrawlerProcess
c = CrawlerProcess({
'USER_AGENT': 'Mozilla/5.0',
})
c.crawl(MySpider)
c.start()