如何使用Selenium和BeautifulSoup刮取div中的div和iframe的内容？

Question

我正在抓一个格式如下的页面：

<div id="container>
   <script>Script that cause iframe contents to load correctly</script>
   <iframe>Contents of iFrame</iframe>
   <script>More scripts</script>
</div>

我可以轻松地抓取页面，但这不会刮掉iframe内容，所以我切换框架：

driver.switch_to.frame(iframeElement)

这允许我获取iframe内容。这引出了我现在的问题，即我如何获取容器div，并在已删除的div中插入已删除的iframe的内容。页面的设置方式，iframe之前有动态脚本，允许iframe的内容工作，这就是我需要在ifped内嵌入iframe内容的原因。

以下相关Python：

driver.get(url)
iframeElement = driver.find_element_by_tag_name('iframe')
driver.switch_to.frame(iframeElement)
time.sleep(3) #Wait for the contents to generate
# driver.switch_to_default_content() #Commented out, but I know to use this to exit out of the iframe

html = driver.page_source
soup=BeautifulSoup(html, features="lxml")
print(soup)
print(soup.find("div", {"id": "Container"})) #Let's see the HTML of the container
soupStr=str(soup)
Con = str(soup.find("div", {"id": "Container"})) #Create a variable with JUST the container HTML

with open('iframeWithinDiv.html', 'w', encoding='utf-8') as f_out: #Save the file
    f_out.write(soupStr)```

Answer 1

您可以使用execute_script和一些jquery将它附加到以下div（您可以使用纯JS）:)

html = driver.page_source
soup=BeautifulSoup(html, features="lxml")
print(soup)
print(soup.find("div", {"id": "Container"})) #Let's see the HTML of the container
soupStr=str(soup)
Con = str(soup.find("div", {"id": "Container"}))1

#### Append your variable to the given string within wrap ###

driver.execute_script("$('#container').val('newhtmlcontent')")