那么BeautifulSoup和Python 3中的多线程/多处理呢？

Question

所以我搞乱了BeautifulSoup。我写了一些代码，并在此处通过它。有了以下问题 - 有没有办法使用多线程或多处理来加速它？打赌这段代码远非理想:)池应该用于这样的场合吗？

PS。我以这个网站为例。

先感谢您。

import requests
from bs4 import BeautifulSoup
import csv
import sys
reload(sys)
sys.setdefaultencoding('utf-8')

pages = [str(i) for i in range(100,2000)]
for page in pages:
    html = requests.get('https://statesassembly.gov.je/Pages/Members.aspxMemberID='+page).text
    def get_page_data():
    soup = BeautifulSoup(html, 'lxml')
    name = soup.find('h1').text
    title = soup.find(class_='gel-layout__item gel-2/3@m gel-1/1@s').find('h2').text
    data = {'name': name,
            'title': title,
            }

    return (data)

data = get_page_data()
with open('Members.csv','a') as output_file:
    writer = csv.writer(output_file, delimiter=';')
    writer.writerow((data['name'],
                    data['title'],
                    ))

Answer 1

蛮力政府网站在某些国家可能是非法的。请确保您已阅读您所在国家/地区的版权法以及您从中获取数据的国家/地区。

首先，请将您的列表分成几部分，然后将它的线程并行执行。

Python program to illustrate the concept of threading

import threading 
import os 

def task1(): 
    print("Task 1 assigned to thread: {}".format(threading.current_thread().name)) 
    print("ID of process running task 1: {}".format(os.getpid())) 

def task2(): 
    print("Task 2 assigned to thread: {}".format(threading.current_thread().name)) 
    print("ID of process running task 2: {}".format(os.getpid())) 

if __name__ == "__main__": 

    # print ID of current process 
    print("ID of process running main program: {}".format(os.getpid())) 

    # print name of main thread 
    print("Main thread name: {}".format(threading.main_thread().name)) 

    # creating threads 
    t1 = threading.Thread(target=task1, name='t1') 
    t2 = threading.Thread(target=task2, name='t2')   

    # starting threads 
    t1.start() 
    t2.start() 

    # wait until all threads finish 
    t1.join() 
    t2.join()