Selenium Python一次处理多个Webdriver

问题描述 投票:0回答:1

嗨,我想问一下如何同时处理多个Chrome浏览器。我尝试使用多个chromedriver.exe,对于每个我想打开的浏览器,都有一个自己的chromedriver.exe]

driverPlace2 = "C:/Users/M/PycharmProjects/Test/chromedriver2.exe"
driverPlace3 = "C:/Users/M/PycharmProjects/Test/chromedriver3.exe"
driverPlace4 = "C:/Users/M/PycharmProjects/Test/chromedriver4.exe"
driverPlace5 = "C:/Users/M/PycharmProjects/Test/chromedriver5.exe"
driverPlace6 = "C:/Users/M/PycharmProjects/Test/chromedriver6.exe"
driverPlace7 = "C:/Users/M/PycharmProjects/Test/chromedriver7.exe"
driverPlace8 = "C:/Users/M/PycharmProjects/Test/chromedriver8.exe"

第六个之后,它开始关闭第二个,第三个...

它在控制台中什么也没有给出,它只是开始关闭。

我该如何解决?

非常感谢您阅读和帮助我

python python-3.x selenium selenium-webdriver webdriver
1个回答
0
投票

您可以将程序与Thread :(以下脚本一起使用,打开2个chrome浏览器,它们同时在网上爬行)

from selenium import webdriver
from random import shuffle
from concurrent import futures
from urllib.parse import urlparse


def driver_crawl_web(first_page):
    driver = webdriver.Chrome()
    driver.get(first_page)
    while True:
        a_tags = driver.find_elements_by_xpath("//a[@href]")
        shuffle(a_tags)
        for tag in a_tags:
            href = tag.get_attribute('href')
            if check_url(href):
                selected_link = href
                break
        if selected_link == driver.current_url:
            driver.back()
            continue
        driver.get(selected_link)

def check_url(url):
    parse = urlparse(url)
    if "https" in parse.scheme:
        if "youtube" not in parse.netloc:
            return True
    return False

def crawl_with_concurrency(links):
    executor = futures.ThreadPoolExecutor(max_workers=2)
    executions = executor.map(driver_crawl_web, links)


links = ["https://www.google.com/search?q=python",
        "https://www.google.com/search?q=javascript"]

crawl_with_concurrency(links)
© www.soinside.com 2019 - 2024. All rights reserved.