使用保存的html页面使用scrapy进行报废

Question

我正在寻找一种方法来使用我在计算机上保存的html页面使用scrapy。就我而言，我收到了一个错误：

requests.exceptions.InvalidSchema: No connection adapters were found for 'file:///home/stage/Guillaume/scraper_test/mypage/details.html'

SPIDER_START_URLS = [“file：///home/stage/Guillaume/scraper_test/mypage/details.html”]

Answer 1

我使用request_fingerprint将现有的HTML文件注入HTTPCACHE_DIR（几乎总是.scrapy/httpcache/${spider_name}）取得了巨大的成功。然后，打开上述默认为基于文件的缓存存储的http cache middleware，以及认为磁盘文件具有权威性的“Dummy Policy”，如果它在缓存中找到URL，则不会发出网络请求。

我希望脚本会像（这只是一般的想法，并不保证甚至运行）：

import sys
from scrapy.extensions.httpcache import FilesystemCacheStorage
from scrapy.http import Request, HtmlResponse
from scrapy.settings import Settings

# this value is the actual URL from which the on-disk file was saved
# not the "file://" version
url = sys.argv[1]
html_filename = sys.argv[2]
with open(html_filename) as fh:
    html_bytes = fh.read()
req = Request(url=url)
resp = HtmlResponse(url=req.url, body=html_bytes, encoding='utf-8', request=req)
settings = Settings()
cache = FilesystemCacheStorage(settings)
spider = None  # fill in your Spider class here
cache.store_response(spider, req, resp)