Python Web Scrape - 403错误

Question

我正在尝试使用python beautifulsoup和urllib打开这个网站，但我一直收到403错误。有人可以指导我这个错误吗？

我目前的代码是这样的;

from urllib.request import urlopen as uReq
from bs4 import BeautifulSoup as soup

my_url = 'https://www.cubesmart.com/florida-self-storage/st--petersburg-self-storage/3337.html?utm_source=local&utm_medium=organic&utm_campaign=googlemybusiness&utm_term=3337'

uClient = uReq(my_url)

但我得到403错误。

我四处搜索并尝试使用下面的方法，但它也给了我同样的错误。

from urllib.request import Request, urlopen
url="https://www.cubesmart.com/florida-self-storage/st--petersburg-self-storage/3337.html?utm_source=local&utm_medium=organic&utm_campaign=googlemybusiness&utm_term=3337"
req = Request(url, headers={'User-Agent': 'Mozilla/5.0'})

web_byte = urlopen(req).read()

webpage = web_byte.decode('utf-8')

任何帮助表示赞赏。

Answer 1

尝试使用session()的requests如下：

import requests

my_session = requests.session()
for_cookies = my_session.get("https://www.cubesmart.com")
cookies = for_cookies.cookies
headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:57.0) Gecko/20100101 Firefox/57.0'}
my_url = 'https://www.cubesmart.com/florida-self-storage/st--petersburg-self-storage/3337.html?utm_source=local&utm_medium=organic&utm_campaign=googlemybusiness&utm_term=3337'

response = my_session.get(my_url, headers=headers, cookies=cookies)
print(response.status_code)  # 200