Python + Selenium firefox webdriver - 从网站中提取图像

Question

我试图使用以下内容从网页中提取图像：Python 2.7 + Selenium（使用FireFox）+ Beautiful Soup。

页面动态加载，因此，我使用Selenium进行屏幕抓取。前端的一切看起来都很棒，但是，当我加载所有图像时，我看到HTML，我看不到图像的链接。有什么想法可以在这里发生什么？

网站是https://flipp.com/flyers?postal_code=97035，从那里，导航到https://flipp.com/weekly_ad/1550082-big-5-sporting-goods-weekly-ad，以查看第一个每周广告（我的工作代码在下面）。

为了让事情更加奇怪，我能够看到图像在检查器窗口中加载......但我仍然无法在HTML中看到它们。有什么想法在这里发生，以及如何获取更新的HTML（图像加载后？）

这是我能够从HTML中提取的一组图像（通过附加jpg）。这些仅适用于将鼠标悬停在画布上的弹出窗口。

我想要达到的实际上是构成实际页面/画布的图像。我可以看到它们通过（使用firefox中的流量选项），但由于某些原因它们没有出现在HTML中。有什么想法在这里发生？

工作代码：

#import packages
from time import gmtime, strftime,sleep, time
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium import webdriver
from selenium.webdriver.common.proxy import Proxy, ProxyType
from selenium.webdriver.common.desired_capabilities import DesiredCapabilities
#scraping packages
from bs4 import BeautifulSoup


USAPROXY = "177.84.23.122:3128"
def launch_webdriver(PROXY):
    PROXY = PROXY
    PROXY_HOST = PROXY.rpartition(':')[0]
    PROXY_PORT = PROXY.rpartition(':')[2]
    fp = webdriver.FirefoxProfile()
    # Direct = 0, Manual = 1, PAC = 2, AUTODETECT = 4, SYSTEM = 5
    fp.set_preference("network.proxy.type", 1)
    fp.set_preference("network.proxy.http",PROXY_HOST)
    fp.set_preference("network.proxy.http_port",int(PROXY_PORT))
    fp.set_preference("network.proxy.ssl",PROXY_HOST)
    fp.set_preference("network.proxy.ssl_port",int(PROXY_PORT))
    fp.set_preference("general.useragent.override","whater_useragent")    
    fp.update_preferences()
    return webdriver.Firefox(firefox_profile=fp)




def test():
    driver = launch_webdriver(USAPROXY)
    driver.set_page_load_timeout(11)
    driver.get("https://flipp.com/flyers?postal_code=97035")
    sleep(15)
    driver.get("https://flipp.com/weekly_ad/1550082-big-5-sporting-goods-weekly-ad")
    sleep(5)
    my_html = driver.page_source
    soup = BeautifulSoup(my_html,'lxml')
    tags=soup.findAll('img')  #prints only 3 imgs, there should be 100s
    for tag in tags:print tag
    print soup.prettify()
#execute script
test()

Answer 1

我用soup = BeautifulSoup(my_html,'lxml')替换soup = BeautifulSoup(my_html,'html.parser')的代码做了一些小改动，如下所示：

代码： driver.set_page_load_timeout(11) driver.get("https://flipp.com/flyers?postal_code=97035") sleep(15) driver.get("https://flipp.com/weekly_ad/1550082-big-5-sporting-goods-weekly-ad") sleep(5) my_html = driver.page_source soup = BeautifulSoup(my_html,'html.parser') tags=soup.findAll('img') for tag in tags:print (tag)
输出： <img alt="" src="/94815ec0/images/page-favourites.svg"/> <img alt="" src="/94815ec0/images/page-flyers.svg"/> <img alt="" src="/94815ec0/images/page-coupons.svg"/> <img alt="" src="/94815ec0/images/profile.png"/> <img alt="" src="/94815ec0/images/signin-google-en.png"/> <img alt="" src="/94815ec0/images/signin-facebook-en.png"/> <img alt="" class="sl-icon" src="/94815ec0/images/sl/list-icon.svg"/> <img alt="" class="logo" contain="true" fit="" href="https://images.wishabi.net/merchants/2143/1399408035/large" is="flipp-lazy-image" src="data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAQAAAC1HAwCAAAAC0lEQVQYV2NgYAAAAAMAAWgmWQ0AAAAASUVORK5CYII=" style='background-image: url("https://images.wishabi.net/merchants/2143/1399408035/large");'/> <img src="/94815ec0/images/location.svg"/> <img alt="" class="flyer-thumbnail" cover="true" fit="" href="https://f.wishabi.net/flyers/1568365/web_premium/1519664612.jpg" is="flipp-lazy-image" src="data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAQAAAC1HAwCAAAAC0lEQVQYV2NgYAAAAAMAAWgmWQ0AAAAASUVORK5CYII=" style='background-image: url("https://f.wishabi.net/flyers/1568365/web_premium/1519664612.jpg");'/> <img alt="" class="logo" contain="true" fit="" href="https://images.wishabi.net/merchants/1417562816/1417562816/large" is="flipp-lazy-image" src="data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAQAAAC1HAwCAAAAC0lEQVQYV2NgYAAAAAMAAWgmWQ0AAAAASUVORK5CYII=" style='background-image: url("https://images.wishabi.net/merchants/1417562816/1417562816/large");'/> <img alt="" class="flyer-thumbnail" cover="true" fit="" href="https://f.wishabi.net/flyers/1570217/web_premium/1519767026.jpg" is="flipp-lazy-image" src="data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAQAAAC1HAwCAAAAC0lEQVQYV2NgYAAAAAMAAWgmWQ0AAAAASUVORK5CYII=" style='background-image: url("https://f.wishabi.net/flyers/1570217/web_premium/1519767026.jpg");'/> <img alt="" class="logo" contain="true" fit="" href="https://images.wishabi.net/merchants/2217/1399408048/large" is="flipp-lazy-image" src="data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAQAAAC1HAwCAAAAC0lEQVQYV2NgYAAAAAMAAWgmWQ0AAAAASUVORK5CYII=" style='background-image: url("https://images.wishabi.net/merchants/2217/1399408048/large");'/> <img alt="" class="flyer-thumbnail" cover="true" fit="" href="https://f.wishabi.net/flyers/1548763/web_premium/1519408077.jpg" is="flipp-lazy-image" src="data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAQAAAC1HAwCAAAAC0lEQVQYV2NgYAAAAAMAAWgmWQ0AAAAASUVORK5CYII=" style='background-image: url("https://f.wishabi.net/flyers/1548763/web_premium/1519408077.jpg");'/> <img alt="" class="logo" contain="true" fit="" href="https://images.wishabi.net/merchants/2392/1412008375/large" is="flipp-lazy-image" src="data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAQAAAC1HAwCAAAAC0lEQVQYV2NgYAAAAAMAAWgmWQ0AAAAASUVORK5CYII=" style='background-image: url("https://images.wishabi.net/merchants/2392/1412008375/large");'/> <img alt="" class="flyer-thumbnail" cover="true" fit="" href="https://f.wishabi.net/flyers/1558209/web_premium/1519940192.jpg" is="flipp-lazy-image" src="data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAQAAAC1HAwCAAAAC0lEQVQYV2NgYAAAAAMAAWgmWQ0AAAAASUVORK5CYII=" style='background-image: url("https://f.wishabi.net/flyers/1558209/web_premium/1519940192.jpg");'/> <img alt="" class="logo" contain="true" fit="" href="https://images.wishabi.net/merchants/2175/1399558010/large" is="flipp-lazy-image" src="data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAQAAAC1HAwCAAAAC0lEQVQYV2NgYAAAAAMAAWgmWQ0AAAAASUVORK5CYII=" style='background-image: url("https://images.wishabi.net/merchants/2175/1399558010/large");'/> <img alt="" class="flyer-thumbnail" cover="true" fit="" href="https://f.wishabi.net/flyers/1553653/web_premium/1519086192.jpg" is="flipp-lazy-image" src="data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAQAAAC1HAwCAAAAC0lEQVQYV2NgYAAAAAMAAWgmWQ0AAAAASUVORK5CYII=" style='background-image: url("https://f.wishabi.net/flyers/1553653/web_premium/1519086192.jpg");'/> <img alt="" class="logo" contain="true" fit="" href="https://images.wishabi.net/merchants/1415661435/1415661435/large" is="flipp-lazy-image" src="data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAQAAAC1HAwCAAAAC0lEQVQYV2NgYAAAAAMAAWgmWQ0AAAAASUVORK5CYII=" style='background-image: url("https://images.wishabi.net/merchants/1415661435/1415661435/large");'/> <img alt="" src="/94815ec0/images/email_notices.png"/> <img alt="Flipp logo"/> <img alt="image of a sad ice cream" class="sad-cream"/> <img alt="Google Chrome Logo" class="browser-img chrome"/> <img alt="Mozilla Firefox Logo" class="browser-img ff"/> <img alt="Microsoft Edge Logo" class="browser-img edge"/> <img alt="Apple Safari Logo" class="browser-img safari"/> <img alt="" height="0" id="batBeacon0.08041384361820791" src="https://bat.bing.com/action/0?ti=5463843&Ver=2&mid=e698c347-3982-6279-c6a5-5e5b764b55dd&evt=pageLoad&sid=ab7428c0-1&lt=1647&pi=0&lg=en-US&sw=1366&sh=768&sc=24&tl=Big%205%20Sporting%20Goods%20Weekly%20Ad%20for%20Lake%20Oswego%20this%20week%20(Feb%2025,%202018%20-%20Mar%203,%202018)%20-%20Flipp&kw=flyers,%20coupons,%20shopping%20list,%20deals,%20circulaires,%20coupons,%20liste%20d%E2%80%99achats,%20offres&p=https%3A%2F%2Fflipp.com%2Fweekly_ad%2F1550082-big-5-sporting-goods-weekly-ad&r=&msclkid=N&rn=558478" style="width:0px; height:0px; display:none; visibility:hidden;" width="0"/>

Answer 2

你没有在my_html=driver.page_source中看到更新的HTML的原因是因为page_source在页面动态加载之前抓取了HTML。尝试这样做以在页面加载后获取HTML：

my_html = driver.execute_script("return document.getElementsByTagName('html')[0].innerHTML")
# or
my_html = driver.find_element_by_tag_name('html').get_attribute('innerHTML')

编辑：

好吧，我想我想出了你想要的东西。我找到了一种方法来访问network资源并获取浏览器正在记录的性能数据。调用此函数并在加载所需页面后传递驱动程序，它应该以您正在寻找的格式返回图像：

def getNetworkImages(driver):
    ImageList = []
    Resources = driver.execute_script("return window.performance.getEntriesByType('resource');")
    for resource in Resources:
        if resource['initiatorType'] == 'img': ImageList.append(resource['name'])
    for image in ImageList: print(image)
    return ImageList

注意：这是使用Chrome 64和Chromedriver 2.35测试的。

Python + Selenium firefox webdriver - 从网站中提取图像

问题描述投票：2回答：2

2个回答

最新问题

Python + Selenium firefox webdriver - 从网站中提取图像

问题描述 投票：2回答：2

2个回答

最新问题

问题描述投票：2回答：2