如何运行多线程刮板

Question

我想运行刮板。

from threading import Thread
from scraper import Scraper

def start(i: int):
    s = Scraper("de",i,10)
    s.scrape()


d = dict()


for i in range(10):
    d[i] = Thread(target=start, args=(i,))

以上脚本仅执行一个刮板。我要运行10个刮板。每个抓取工具都使用requests抓取目标网址列表的十分之一。我究竟做错了什么？您需要什么信息？

Answer 1

from threading import Thread
from scraper import Scraper

def start(i: int):
    s = Scraper("de",i,10)
    s.scrape()
d = {i:Thread(target=start, args=(i,)) for i in range(10)}
[d[i].start() for i in range(10)]