如何通过抓斗外壳爬行特定的网站

问题描述 投票:0回答:1

我想通过刮板抓取该网站:https://aaav2.hinet.net/A1/AuthScreen.jsp

我尝试过:

$ scrapy shell'https://aaav2.hinet.net/A1/AuthScreen.jsp'

以及用户代理

$ scrapy shell -s USER_AGENT ='Mozilla / 5.0(Macintosh; Intel Mac OS X10_11_2)AppleWebKit / 601.3.9(KHTML,如Gecko)版本/9.0.2Safari / 601.3.9''https://aaav2.hinet.net/A1/AuthScreen.jsp'

然后查看(响应)但一无所获

有人会帮助我获得正确的观点(回应),就像我直接在浏览器中打开此网址一样?

python scrapy
1个回答
0
投票

您正在重定向到另一个URL。做:

$ scrapy shell "https://aaav2.hinet.net/A1/error.jsp?aa-eurl=edc68fe62571d6617ef5f42113d9068aa9f6600e320d55084d75fbf2cd244155e02b9b684284ed94c52ee591d2edde9a&mesg=aa-version+parameter+is+required%21%3Cbr+%2F%3Eaa-productid+parameter+is+required%21%3Cbr+%2F%3Eaa-curl+parameter+is+required%21%3Cbr+%2F%3Eaa-eurl+parameter+is+required%21%3Cbr+%2F%3Eaa-fee+parameter+is+required%21%3Cbr+%2F%3E&aa-eurlDesc=&aa-device=pc&aa-usage=&aa-propertiesKey=&aa-language="

view(response)此页面将向您显示一个页面,就像您的浏览器一样。

© www.soinside.com 2019 - 2024. All rights reserved.