我想提取网站上图表中显示的数据并自动下载。 这是标题后的第一张图表：“办公楼投资价格（总欧元 psqm）” 上：https://www.immostat.com/market-data
然而，数据位于 iframe 后面并不断更新，从而更新图表，从而更新 iframe 中的链接。所以我需要动态更新链接。使用浏览器中的开发工具，我设法在名为“htmlComp-iframe (ae915e_6e3e7e9d19bc8d5af1f3e4b96ae5c686.html) 的源代码中找到数据，并且可以通过以下链接加载：
https://www-immostat-com.filesusr.com/html/ae915e_6e3e7e9d19bc8d5af1f3e4b96ae5c686.html
我对 HTML 等的经验非常有限，但我发现该图表是作为 iframe 中的单独网页加载的。所以我希望只能在原始 HTML 中找到链接。由于此链接指向一组静态数据，并且当新数据可用时使用新链接，因此我需要找到一种动态查找数据的方法。然而，当我在 httr:GET 中找到内容中的链接并使用 content("text") 提取它时，我无法找到内容中的链接。
httr::GET(
"https://www.immostat.com/market-data",
verbose()
) %>%
content("text")
我错过了什么吗？
我会怎样
iframe
，有：
<div id="comp-iw3d16s21" class="comp-iw3d16s21 _xg6_p"></div>
所有这些动态元素均由浏览器中的 Javascript 呈现，此过程的一部分由资产管理器协调，该资产管理器将 id-s（如 comp-iw3d16s21
）转换为特定页面版本的 url-s。该响应的一部分如下所示：
"comp-iw3d16s21": {
"url": "https://www-immostat-com.filesusr.com/html/ae915e_6e3e7e9d19bc8d5af1f3e4b96ae5c686.html",
"translations": {
"title": "Embedded Content"
}
}
从技术上讲，您也可以自己提出该请求结束提取 iframe
url，尽管它包含 35 个参数，并且使其适用于下一页版本看起来是一个适当的挑战：
https://siteassets.parastorage.com/pages/pages/thunderbolt?appDefinitionIdToSiteRevision=%7B%2214bcded7-0066-7c35-14d7-466cb3f09103%22%3A%22855%22%7D&beckyExperiments=specs.thunderbolt.supportSpxInEEMappers%3Atrue%2Cspecs.thunderbolt.one_cell_grid_display_flex%3Atrue%2Cspecs.thunderbolt.MediaContainerAndPageBackgroundMapper%3Atrue%2Cspecs.thunderbolt.catharsis_theme_optimize_css%3Atrue%2Cspecs.thunderbolt.WRichTextSemanticClasses%3Atrue%2Cspecs.thunderbolt.ghostify_hidden_comps%3Atrue%2Cspecs.thunderbolt.edixIsInFirstFold%3Atrue%2Cspecs.thunderbolt.catharsis_theme%3Atrue%2Cspecs.thunderbolt.DatePickerPortal%3Atrue%2Cspecs.thunderbolt.native_css_mappers_popups%3Atrue%2Cspecs.thunderbolt.wowImageRelayout%3Atrue%2Cspecs.thunderbolt.useElementoryRelativePath%3Atrue%2Cspecs.thunderbolt.new_responsive_layout_render_all_breakpoints%3Atrue%2Cspecs.thunderbolt.mesh_css_catharsis%3Atrue%2Cspecs.thunderbolt.DDMenuMigrateCssCarmiMapper%3Atrue%2Cspecs.thunderbolt.responsiveShapeDividersPublic%3Atrue%2Cspecs.thunderbolt.compsMeasuresCss_catharsis%3Atrue%2Cspecs.thunderbolt.customElemCollapsedheight%3Atrue%2Cspecs.thunderbolt.url_hierarchy%3Atrue%2Cspecs.thunderbolt.scaleprop%3Atrue%2Cspecs.thunderbolt.interactionsOverrides%3Atrue%2Cspecs.thunderbolt.displayRefComponentsAsBlock%3Atrue%2Cspecs.thunderbolt.pinned_layout_css_catharsis%3Atrue%2CuseTranslatedUrlSlugs%3Atrue%2Cspecs.thunderbolt.responsiveLayout_optimize_css%3Atrue%2Cspecs.thunderbolt.theme_fonts_colors_catharsis%3Atrue%2Cspecs.thunderbolt.catharsis_fontFaces%3Atrue&contentType=application%2Fjson&deviceType=Desktop&dfCk=6&dfVersion=1.2684.0&disableStaticPagesUrlHierarchy=false&editorName=Unknown&experiments=bv_remove_add_chat_viewer_fixer%2Cdm_linkTargetDefaults%2Cdm_removePageDataUnderTranslations%2Cdm_runTranslationsPageUriSeoFixer&externalBaseUrl=https%3A%2F%2Fwww.immostat.com&fileId=5745cace.bundle.min&formFactor=desktop&hasTPAWorkerOnSite=false&isConsentPolicyActive=true&isHttps=true&isInSeo=false&isMultilingualEnabled=false&isPremiumDomain=true&isTrackClicksAnalyticsEnabled=false&isUrlMigrated=true&isWixCodeOnPage=false&isWixCodeOnSite=false&language=fr&languageResolutionMethod=QueryParam&metaSiteId=6d08802f-53b1-41d3-8efe-e9ff693936c5&module=thunderbolt-features&originalLanguage=fr&pageId=ae915e_8a4ec4fca009e211728288dc101e4786_1143.json&quickActionsMenuEnabled=false®istryLibrariesTopology=%5B%7B%22artifactId%22%3A%22editor-elements%22%2C%22namespace%22%3A%22wixui%22%2C%22url%22%3A%22https%3A%2F%2Fstatic.parastorage.com%2Fservices%2Feditor-elements%2F1.11361.0%22%7D%2C%7B%22artifactId%22%3A%22editor-elements%22%2C%22namespace%22%3A%22dsgnsys%22%2C%22url%22%3A%22https%3A%2F%2Fstatic.parastorage.com%2Fservices%2Feditor-elements%2F1.11361.0%22%7D%5D&remoteWidgetStructureBuilderVersion=1.238.0&siteId=64b052e8-197b-46d4-8d7f-2954e7e9a6a6&siteRevision=1144&staticHTMLComponentUrl=https%3A%2F%2Fwww-immostat-com.filesusr.com%2F&useSandboxInHTMLComp=false&viewMode=desktop
我个人倾向于无头浏览器，它可以执行网站的 JavaScript 并以最终用户看到的方式呈现该页面；在这样的环境中评估自定义 javascript 也非常方便。这里个人偏好是 {chromote}
，但它与 RSelenium / Selenium 比较相似。我确信这个过程可以优化，但这里是这样的：
chromote加载页面，网站的 JavaScript 将被执行，并且所有可通过网络浏览器元素检查器访问的元素都可以访问；
iframe，使用带有预定义文本的标题作为锚点，提取
iframeurl；
{rvest}解析内容，通过CSS选择器和XPath提取元素很方便；
chromote的JS运行时评估修改后的JS（我们已经启动并运行它，所以我们不妨使用它）
library(rvest)
library(stringr)
library(chromote)
# create new Chrome session, wait until page is loaded + few more moments
b <- ChromoteSession$new()
{
b$Page$navigate("https://www.immostat.com/market-data")
b$Page$loadEventFired()
Sys.sleep(.5)
}
# evaluate JavaScript in Chromote:
# xpath to find iframe in relation to the h2 element with specific text content,
# "Office Investment Prices (gross € psqm)",
# extract src atribute from the first matched iframe
iframe_src <- b$Runtime$evaluate('
var xpath = \'//h2[text()="Office Investment Prices (gross € psqm)"]/../following-sibling::div/wix-iframe/div/iframe\';
document.evaluate(
xpath,
document, null, XPathResult.UNORDERED_NODE_ITERATOR_TYPE, null )
.iterateNext()
.getAttribute("src")')$result$value
iframe_src
#> [1] "https://www-immostat-com.filesusr.com/html/ae915e_6e3e7e9d19bc8d5af1f3e4b96ae5c686.html"
如果从 javascript 提取数据系列已经启动并运行，您可以在此停止并忽略其余部分。
# with rvest extract script element from iframe page source, we only need element containing "var dataIDF",
# split it by linefeed,
# get a line from jsvsacript with line chart data,
# modify resulting javascript so it would return google.visualization.arrayToDataTable() argument as
# JSON string, i.e.
# var dataIDF = google.visualization.arrayToDataTable([['Quarter','Greater Paris Region'],['Q1 2006',4490],...)
# becomes:
# JSON.stringify([['Quarter','Greater Paris Region'],['Q1 2006',4490],...)
array_stringify_js <- read_html(iframe_src) |>
html_element(xpath = "//script[contains(., 'var dataIDF')]") |>
html_text() |>
str_split("\n") |>
unlist() |>
str_subset("var dataIDF") |>
str_replace("var.*arrayToDataTable", "JSON.stringify")
str_trunc(array_stringify_js, 80)
#> [1] "\tJSON.stringify([['Quarter','Greater Paris Region'],['Q1 2006',4490],['Q2 200..."
# use existing JS runtime to evalute that frankenscript we just created,
# parse resulting JSON with jsonlite, result is a list
chart_data_lst <- b$Runtime$evaluate(array_stringify_js)$result$value |>
jsonlite::parse_json()
str(chart_data_lst[1:3])
#> List of 3
#> $ :List of 2
#> ..$ : chr "Quarter"
#> ..$ : chr "Greater Paris Region"
#> $ :List of 2
#> ..$ : chr "Q1 2006"
#> ..$ : int 4490
#> $ :List of 2
#> ..$ : chr "Q2 2006"
#> ..$ : int 4631
# returned list structure could use some tweaking,
# we want list of named lists and items with same names should share same types,
# this structure can be converted to data.frame with proper column types,
# without having to deal with numeric values turned into strings;
# use the first list item for names for all others, drop the first element,
# bind list of named lists to data.frame
chart_data <- lapply(chart_data_lst[-1], setNames, chart_data_lst[[1]]) |>
do.call(rbind, args = _) |>
as.data.frame()
结果：
head(chart_data)
#> Quarter Greater Paris Region
#> 1 Q1 2006 4490
#> 2 Q2 2006 4631
#> 3 Q3 2006 4803
#> 4 Q4 2006 4837
#> 5 Q1 2007 4953
#> 6 Q2 2007 5064