无法使用requests_html从url获取全文

Question

我正在尝试从此网址解析页面：

https://www.mathworks.com/help/radar/referencelist.html?type=block

我需要从“Radar Toolbox — Blocks”标题下的块列表中获取所有链接，即在

<div id="reflist_content">

内。

我像这样使用

requests_html

：

from requests_html import HTMLSession

session = HTMLSession()

url = 'https://www.mathworks.com/help/radar/referencelist.html?type=block'
r = session.get(url)

r.html.arender()

results = r.html.find('div')
res_str = ''

for item in results:
    #print(item)
    #print(item.text)
    res_str += str(item) + '\n'
    res_str += item.text + '\n\n'

结果中

reflist_content

的文字为空。

我在结果中找不到任何需要的内容。我尝试通过不同的 html 标签或关键字进行搜索，但似乎带有块的表格根本没有呈现。 我做错了什么？

Answer 1

您看到的数据是从外部 URL 加载的。要加载它，您可以使用以下示例：

import requests
from bs4 import BeautifulSoup

url = "https://www.mathworks.com/help/radar/referencelist_block_cat.xml"

soup = BeautifulSoup(requests.get(url).content, "xml")

for category in soup.select("cat:has(ref)"):
    print(category.title.text)
    print()
    for r in category.select("ref"):
        print(
            f'{r.text[:40]:<40} {"https://www.mathworks.com/help/radar/" + r["target"]}'
        )
    print()

打印：

Radar Toolbox

Backscatter signals from bicyclist (Sinc https://www.mathworks.com/help/radar/ref/backscatterbicyclistblock.html
Backscatter signals from pedestrian (Sin https://www.mathworks.com/help/radar/ref/backscatterpedestrianblock.html
Barrage jammer interference source (Sinc https://www.mathworks.com/help/radar/ref/barragejammer.html
Constant gamma clutter simulation (Since https://www.mathworks.com/help/radar/ref/constantgammaclutter.html
Constant gamma clutter simulation using  https://www.mathworks.com/help/radar/ref/gpuconstantgammaclutter.html
Generate radar sensor detections and tra https://www.mathworks.com/help/radar/ref/radardatagenerator.html
Combine detection reports from different https://www.mathworks.com/help/radar/ref/detectionconcatenation.html
Two-ray channel environment (Since R2021 https://www.mathworks.com/help/radar/ref/tworaychannel.html
Wideband two-ray channel environment (Si https://www.mathworks.com/help/radar/ref/widebandtworaychannel.html
Library of pulse waveforms (Since R2021a https://www.mathworks.com/help/radar/ref/pulsewaveformlibrary.html
Library of pulse compression specificati https://www.mathworks.com/help/radar/ref/pulsecompressionlibrary.html
Cluster detections (Since R2021a)        https://www.mathworks.com/help/radar/ref/dbscanclusterer.html

Data Synthesis

Backscatter signals from bicyclist (Sinc https://www.mathworks.com/help/radar/ref/backscatterbicyclistblock.html
Backscatter signals from pedestrian (Sin https://www.mathworks.com/help/radar/ref/backscatterpedestrianblock.html
Barrage jammer interference source (Sinc https://www.mathworks.com/help/radar/ref/barragejammer.html
Constant gamma clutter simulation (Since https://www.mathworks.com/help/radar/ref/constantgammaclutter.html
Constant gamma clutter simulation using  https://www.mathworks.com/help/radar/ref/gpuconstantgammaclutter.html
Generate radar sensor detections and tra https://www.mathworks.com/help/radar/ref/radardatagenerator.html
Combine detection reports from different https://www.mathworks.com/help/radar/ref/detectionconcatenation.html
Two-ray channel environment (Since R2021 https://www.mathworks.com/help/radar/ref/tworaychannel.html
Wideband two-ray channel environment (Si https://www.mathworks.com/help/radar/ref/widebandtworaychannel.html
Library of pulse waveforms (Since R2021a https://www.mathworks.com/help/radar/ref/pulsewaveformlibrary.html

Signal and Data Processing

Library of pulse compression specificati https://www.mathworks.com/help/radar/ref/pulsecompressionlibrary.html
Cluster detections (Since R2021a)        https://www.mathworks.com/help/radar/ref/dbscanclusterer.html

Detection, Range, Angle, and Doppler Estimation

Library of pulse compression specificati https://www.mathworks.com/help/radar/ref/pulsecompressionlibrary.html

Clustering

Cluster detections (Since R2021a)        https://www.mathworks.com/help/radar/ref/dbscanclusterer.html