我想通过刮板抓取该网站:https://aaav2.hinet.net/A1/AuthScreen.jsp
我尝试过:
$ scrapy shell'https://aaav2.hinet.net/A1/AuthScreen.jsp'
以及用户代理
$ scrapy shell -s USER_AGENT ='Mozilla / 5.0(Macintosh; Intel Mac OS X10_11_2)AppleWebKit / 601.3.9(KHTML,如Gecko)版本/9.0.2Safari / 601.3.9''https://aaav2.hinet.net/A1/AuthScreen.jsp'
然后查看(响应)但一无所获
有人会帮助我获得正确的观点(回应),就像我直接在浏览器中打开此网址一样?
您正在重定向到另一个URL。做:
$ scrapy shell "https://aaav2.hinet.net/A1/error.jsp?aa-eurl=edc68fe62571d6617ef5f42113d9068aa9f6600e320d55084d75fbf2cd244155e02b9b684284ed94c52ee591d2edde9a&mesg=aa-version+parameter+is+required%21%3Cbr+%2F%3Eaa-productid+parameter+is+required%21%3Cbr+%2F%3Eaa-curl+parameter+is+required%21%3Cbr+%2F%3Eaa-eurl+parameter+is+required%21%3Cbr+%2F%3Eaa-fee+parameter+is+required%21%3Cbr+%2F%3E&aa-eurlDesc=&aa-device=pc&aa-usage=&aa-propertiesKey=&aa-language="
view(response)
此页面将向您显示一个页面,就像您的浏览器一样。