访问被拒绝而刮

问题描述 投票:0回答:1

我想创建一个脚本去到https://www.size.co.uk/featured/footwear/和刮含量,但不知何故,当我运行该脚本,我得到了访问被拒绝。下面是代码:

from urllib import urlopen
from bs4 import BeautifulSoup as BS
url = urlopen('https://www.size.co.uk/')
print BS(url, 'lxml')

输出是

<html><head>
<title>Access Denied</title>
</head><body>
<h1>Access Denied</h1>

You don't have permission to access "http://www.size.co.uk/" on this server.
<p>
Reference #18.6202655f.1498945327.11002828
</p></body>
</html>

当我与其他网站的尝试,代码工作正常,也当我使用硒,什么都不会发生,但我还是想知道如何在不使用Selenium绕过这个错误。但是,当我不同的网站上使用Selenium像http://www.footpatrol.co.uk/shop我得到了同样的访问被拒绝的错误,这里是footpatrol的代码:

from selenium import webdriver

driver = webdriver.PhantomJS('C:\Users\V\Desktop\PY\web_scrape\phantomjs.exe')
driver.get('http://www.footpatrol.com')
pageSource = driver.page_source
soup = BS(pageSource, 'lxml')
print soup

输出是:

<html><head>
<title>Access Denied</title>
</head><body>
<h1>Access Denied</h1>

You don't have permission to access "http://www.footpatrol.co.uk/" on this 
server.<p>
Reference #18.6202655f.1498945644.110590db


</p></body></html>
python web-scraping beautifulsoup
1个回答
2
投票
import requests
from bs4 import BeautifulSoup as BS

url = 'https://www.size.co.uk/'
agent = {"User-Agent":'Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.115 Safari/537.36'}
page = requests.get(url, headers=agent)
print (BS(page.content, 'lxml'))
© www.soinside.com 2019 - 2024. All rights reserved.