在Puppeteer中循环播放一组网址

问题描述 投票:2回答:1

如何使用Puppeteer从多个网址中删除内容?

我创建了一个循环,但我只看到第一个url的结果。

我怀疑它与我声明结果变量的地方有关,但我没有运气,有人知道怎么做吗?

const puppeteer = require('puppeteer');

function run() {
    return new Promise(async (resolve, reject) => {
        try {

            const browser = await puppeteer.launch();
            const page = await browser.newPage();
            const urls = ["https://www.marksandspencer.com/high-neck-long-sleeve-blouse/p/p60260040?image=SD_01_T43_5168_HD_X_EC_90&color=LIGHTDENIM&prevPage=plp", "https://www.marksandspencer.com/pure-cotton-printed-short-sleeve-t-shirt/p/p60263529?image=SD_01_T41_8030Z_Z4_X_EC_90&color=WHITEMIX&prevPage=plp"];
            

              for (let i = 0; i < urls.length; i++) {
                const url = urls[i];
                await page.goto(url);
                let products = await page.evaluate(() => {
                    let product = document.querySelector('h1[itemprop=name]').innerText;
                    let results = [];
                    let items = document.querySelectorAll('[data-ttip-id=sizeGridTooltip] tbody tr td label');
                    items.forEach((element) => {
                        let size = element.getAttribute('for');
                        let stockLevel = "";
                        let nearest_td = element.closest('td');
                        if (nearest_td.classList.contains('low-stock')) {
                            stockLevel = "Low stock"
                        } else if (nearest_td.classList.contains('out-of-stock')) {
                            stockLevel = "Out of stock"
                        } else {
                            stockLevel = "In stock"
                        }
                        results.push({
                            product: product,
                            size: size,
                            stock: stockLevel
                        })
                    });
                    return results
                })
                browser.close();
                return resolve(products);
            }
            
        } catch (e) {
            return reject(e);
        }
    })
}
run().then(console.log).catch(console.error);
javascript puppeteer
1个回答
2
投票

这些行在你的for循环中:

                browser.close();
                return resolve(products);

因此,作为第一次迭代的一部分,您关闭浏览器并返回该函数。你应该将它移出你的for循环并将products存储在一个数组中,如下所示:

              const urls = /* ... */;
              const productsList = [];

              for (let i = 0; i < urls.length; i++) {
                const url = urls[i];
                await page.goto(url);
                let products = await page.evaluate(/* ... */);
                productsList.push(products);
              }
              browser.close();
              return resolve(productsList); // resolve with an array containing the aggregated products

如果您正在寻找更优雅的解决方案(用于并行抓取页面),您可能需要查看库puppeteer-cluster(免责声明:我是作者)。

© www.soinside.com 2019 - 2024. All rights reserved.