如何绕过HTTP错误403:使用Python 3禁止使用urllib.request

问题描述 投票:0回答:2

嗨不是每次,但有时当试图获得访问LSE代码时,我抛出了每个恼人的HTTP错误403:禁止消息。

任何人都知道如何只使用标准的python模块来克服这个问题(很遗憾没有美丽的汤)。

import urllib.request

url = "http://www.londonstockexchange.com/exchange/prices-and-markets/stocks/indices/ftse-indices.html"
infile = urllib.request.urlopen(url) # Open the URL
data = infile.read().decode('ISO-8859-1') # Read the content as string decoded with ISO-8859-1

print(data) # Print the data to the screen

但是偶尔会出现这个错误:

Traceback (most recent call last):
  File "/home/ubuntu/workspace/programming_practice/Assessment/Summative/removingThe403Error.py", line 5, in <module>
    webpage = urlopen(req).read().decode('ISO-8859-1')
  File "/usr/lib/python3.4/urllib/request.py", line 161, in urlopen
    return opener.open(url, data, timeout)
  File "/usr/lib/python3.4/urllib/request.py", line 469, in open
    response = meth(req, response)
  File "/usr/lib/python3.4/urllib/request.py", line 579, in http_response
    'http', request, response, code, msg, hdrs)
  File "/usr/lib/python3.4/urllib/request.py", line 507, in error
    return self._call_chain(*args)
  File "/usr/lib/python3.4/urllib/request.py", line 441, in _call_chain
    result = func(*args)
  File "/usr/lib/python3.4/urllib/request.py", line 587, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 403: Forbidden


Process exited with code: 1

链接到所有可以使用的模块列表:https://docs.python.org/3.4/py-modindex.html

提前谢谢了。

python python-3.x urllib http-status-code-403 urllib3
2个回答
1
投票

看来你的利率有限。尝试入睡并重试。例如:

import urllib
import urllib.request
from time import sleep

LSE_URL = "http://www.londonstockexchange.com/exchange/prices-and-markets/stocks/indices/ftse-indices.html"
WAIT_PERIOD = 15

def stock_data_reader():
    stock_data = get_stock_data()
    while True:
        if not stock_data:
            sleep(WAIT_PERIOD) # sleep for a while until next retry
            stock_data = get_stock_data()                
        else:
            break

    print(stock_data) # do something with stock data



def get_stock_data():
    try:
        infile = urllib.request.urlopen(LSE_URL) # Open the URL
    except urllib.error.HTTPError as http_err:
        print("Error: %s" % http_err)
        return None
    else:
        data = infile.read().decode('ISO-8859-1') # Read the content as string decoded with ISO-8859-1
        return data


stock_data_reader()

0
投票

这可能是由于mod_security造成的。你需要通过打开URL作为浏览器而不是python urllib来欺骗。

在这里,我更正了你的代码:

import urllib.request

url = "http://www.londonstockexchange.com/exchange/prices-and-markets/stocks/indices/ftse-indices.html"

# Open the URL as Browser, not as python urllib
page=urllib.request.Request(url,headers={'User-Agent': 'Mozilla/5.0'}) 
infile=urllib.request.urlopen(page).read()
data = infile.decode('ISO-8859-1') # Read the content as string decoded with ISO-8859-1

print(data) # Print the data to the screen

接下来,您可以使用BeautifulSoup来抓取HTML。

© www.soinside.com 2019 - 2024. All rights reserved.