通过具有嵌入式传单svg的RSelenium提取基础数据，等等

问题描述 投票：2回答：2

我想在此link中提取有关每个广告的信息。现在，我可以自动单击See Ad Details了，但是有很多基础数据并不容易将它们整理成整洁的数据框。

library(RSelenium)
rs <- rsDriver()
remote <- rs$client
remote$navigate(
  paste0(
    "https://www.facebook.com/ads/library/?", 
    "active_status=all&ad_type=political_and_issue_ads&country=US&", 
    "impression_search_field=has_impressions_lifetime&", 
    "q=actblue&view_all_page_id=38471053686"
  )
)

test <- remote$findElement(using = "xpath", "//*[@class=\"_7kfh\"]")
test$clickElement()
## Manually figured out element
test <- remote$findElement(using = "xpath", "//*[@class=\"_7lq0\"]")
test$getElementText()

输出文本本身很乱，但是我相信经过一些时间和努力，可以将其整理成有用的东西。问题是在

中处理基础数据
  1. 图形，它似乎只是一个图像，和
  2. 传单svg，当光标悬停在其上时显示数据。

我不知道如何系统地提取此图像，尤其是传单svg。在这种情况下，我将如何处理每个广告，然后提取详细信息中可用的全部数据？

r reactjs web-scraping leaflet rselenium
2个回答
2
投票

年龄和性别图形是Canva元素。要将其获取为图像，可以对元素进行截图。 Python示例：

driver.find_element_by_tag_name('canvas').screenshot("age_and_gender.png")

显示此广告的位置是SVG，您可以用相同的方法将其保存为图像。结果将不是很准确，因为SVG的可见部分与实际部分不同。但是您可以在之后裁剪图像。 Python示例：

driver.find_element_by_tag_name('svg').screenshot("where_this_ad_was_shown.png")

要从中提取全部数据，您不能使用Selenium。获取数据的方式是配置代理服务器，捕获API请求并获取JSON格式的数据。是的，有可能。

简单的方法是使用某些请求来获取广告和详细信息，而无需使用Selenium。 Python工作示例：

import json
import requests

params = (
    ('q', 'actblue'),
    ('count', '1000'), # default is 30, for 38471053686 it will return about 300 results.
    ('active_status', 'all'),
    ('ad_type', 'political_and_issue_ads'),
    ('countries/[0/]', 'US'),
    ('impression_search_field', 'has_impressions_lifetime'),
    ('view_all_page_id', '38471053686'),
)

data = {'__a': '1', }

with requests.session() as s:
    response = s.post('https://www.facebook.com/ads/library/async/search_ads/', params=params, data=data)
    ads = json.loads(response.text.replace('for (;;);', ''))['payload']['results']
    for ad in ads:
        ad_details_params = (
            ('ad_archive_id', ad[0]['adArchiveID']),
            ('country', 'US'),
        )
        response = s.post('https://www.facebook.com/ads/library/async/insights/', params=ad_details_params, data=data)
        print('parse json from response')

不是：Facebook不允许未经书面同意就自动收集数据权限https://www.facebook.com/apps/site_scraping_tos_terms.php

但是众所周知，Facebook不会拒绝收集我们的数据。

每个广告详细信息的响应将类似于：

{
  "__ar": 1,
  "payload": {
    "ageGenderData": [
      {
        "age_range": "18-24",
        "female": 0.03,
        "male": 0.05,
        "unknown": 0
      },
      {
        "age_range": "25-34",
        "female": 0.12,
        "male": 0.12,
        "unknown": 0.01
      },
      {
        "age_range": "35-44",
        "female": 0.16,
        "male": 0.09,
        "unknown": 0
      },
      {
        "age_range": "45-54",
        "female": 0.11,
        "male": 0.05,
        "unknown": 0
      },
      {
        "age_range": "55-64",
        "female": 0.09,
        "male": 0.04,
        "unknown": 0
      },
      {
        "age_range": "65+",
        "female": 0.09,
        "male": 0.03,
        "unknown": 0
      }
    ],
    "currency": "USD",
    "currencyMatched": true,
    "impressions": "35\u00a0B - 40\u00a0B",
    "locationData": [
      {
        "reach": 0,
        "region": "Alabama"
      },
      {
        "reach": 0,
        "region": "Utah"
      },
      {
        "reach": 0,
        "region": "Maine"
      },
      {
        "reach": 0,
        "region": "Louisiana"
      },
      {
        "reach": 0,
        "region": "Kentucky"
      },
      {
        "reach": 0,
        "region": "Kansas"
      },
      {
        "reach": 0,
        "region": "Idaho"
      },
      {
        "reach": 0,
        "region": "Delaware"
      },
      {
        "reach": 0,
        "region": "Connecticut"
      },
      {
        "reach": 0,
        "region": "Arkansas"
      },
      {
        "reach": 0,
        "region": "Hawaii"
      },
      {
        "reach": 0,
        "region": "Alaska"
      },
      {
        "reach": 0,
        "region": "Montana"
      },
      {
        "reach": 0,
        "region": "West Virginia"
      },
      {
        "reach": 0,
        "region": "Vermont"
      },
      {
        "reach": 0,
        "region": "Mississippi"
      },
      {
        "reach": 0,
        "region": "Wyoming"
      },
      {
        "reach": 0,
        "region": "Oklahoma"
      },
      {
        "reach": 0,
        "region": "North Dakota"
      },
      {
        "reach": 0,
        "region": "New Mexico"
      },
      {
        "reach": 0,
        "region": "New Hampshire"
      },
      {
        "reach": 0,
        "region": "Nebraska"
      },
      {
        "reach": 0,
        "region": "Rhode Island"
      },
      {
        "reach": 0,
        "region": "South Dakota"
      },
      {
        "reach": 0.01,
        "region": "Wisconsin"
      },
      {
        "reach": 0.01,
        "region": "Missouri"
      },
      {
        "reach": 0.01,
        "region": "Oregon"
      },
      {
        "reach": 0.01,
        "region": "Minnesota"
      },
      {
        "reach": 0.01,
        "region": "Maryland"
      },
      {
        "reach": 0.01,
        "region": "New Jersey"
      },
      {
        "reach": 0.01,
        "region": "Tennessee"
      },
      {
        "reach": 0.01,
        "region": "Washington, District of Columbia"
      },
      {
        "reach": 0.01,
        "region": "Indiana"
      },
      {
        "reach": 0.02,
        "region": "Michigan"
      },
      {
        "reach": 0.02,
        "region": "Iowa"
      },
      {
        "reach": 0.02,
        "region": "North Carolina"
      },
      {
        "reach": 0.02,
        "region": "Georgia"
      },
      {
        "reach": 0.02,
        "region": "Colorado"
      },
      {
        "reach": 0.02,
        "region": "Ohio"
      },
      {
        "reach": 0.02,
        "region": "Arizona"
      },
      {
        "reach": 0.02,
        "region": "Pennsylvania"
      },
      {
        "reach": 0.02,
        "region": "Virginia"
      },
      {
        "reach": 0.03,
        "region": "Washington"
      },
      {
        "reach": 0.03,
        "region": "Massachusetts"
      },
      {
        "reach": 0.04,
        "region": "Illinois"
      },
      {
        "reach": 0.04,
        "region": "Florida"
      },
      {
        "reach": 0.06,
        "region": "New York"
      },
      {
        "reach": 0.13,
        "region": "California"
      },
      {
        "reach": 0.19,
        "region": "Texas"
      }
    ],
    "singleCountry": "US",
    "spend": "$500 - $599",
    "pageSpend": {
      "currentWeek": null,
      "isPoliticalPage": true,
      "weeklyByDisclaimer": {
        "WARREN FOR PRESIDENT, INC.": 270970
      },
      "lifetimeByDisclaimer": {
        "Elizabeth for MA": 781272,
        "Warren for President": 3396973,
        "": 13584,
        "WARREN FOR PRESIDENT, INC.": 4081618,
        "the Elizabeth Warren Presidential Exploratory Committee": 219471
      },
      "hasPoliticalSpendInAnyCountry": true
    },
    "pageBlurb": "United States Senator from Massachusetts, former teacher, and candidate for President of the United States. (official campaign account)"
  },
  "bootloadable": {},
  "ixData": {},
  "bxData": {},
  "gkxData": {},
  "qexData": {},
  "lid": "6796246259692811543"
}

最后，要从R运行此python代码，请使用reticulate，然后简单地将整个python脚本作为字符串运行-请注意，如果python脚本不包含任何"字符，则使用它非常方便直接掉入R，像这样

library(reticulate)
py_run_string("import json
import requests

rest of script"

此外，您将需要安装脚本使用的两个python库。这可以通过以下方式完成：在Mac上打开终端，然后输入pip install json以安装json python库，并输入pip install requests作为请求库）

1
投票

这不是一个完整的答案，但希望它能有所帮助。

我进行了抓取/解析，但是无法理解图形数据，因为它似乎位于通过chrome开发工具中的“网络”标签访问的许多文件中的复杂位置（我发现了数据补丁，通过在网络标签中使用command + f并搜索图表中包含的单词（例如“妇女”，“未知”等））

熟悉ReactJS的人可能会更幸运！

可能有效

您可以尝试使用光学字符识别（OCR）的完全不同的方法。

即，截取屏幕截图（即remote$screenshot()），从base64转换为图像，读取它，提取相关区域（即您要获取的特定数据的位置），并使用here描述的方法将包含所需数据的区域转换为文本！ （如果有机会尝试的话，我会更新，但看起来不太可能，渴望听到您的情况）

最新问题
© www.soinside.com 2019 - 2024. All rights reserved.