无法让脚本生成模仿 Chrome 开发工具后的 XHR 请求的 JSON 响应

Question

当我在Chrome浏览器中打开这个链接时，将光标悬停在位于左上角的

Stocks

上，然后从类别

Trading Liquidity

中选择

Most Active

，我可以看到所需的数据显示在那一页。

我尝试使用下面的脚本来复制我可以在 Chrome 开发工具中看到的 XHR 查询，但结果是 status_code 400。

import requests

url = 'https://www.barchart.com/proxies/core-api/v1/quotes/get'
link = 'https://www.barchart.com/stocks/most-active/daily-volume-leaders'

headers = {
    'Accept': 'application/json',
    'Accept-Encoding': 'gzip, deflate, br',
    'Accept-Language': 'en-US,en;q=0.5',
    'Host': 'www.barchart.com',
    'Referer': 'https://www.barchart.com/stocks/most-active/trading-liquidity',
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:121.0) Gecko/20100101 Firefox/121.0',
}

params = {
    'lists': 'stocks.us.trading_liquidity.advances.overall',
    'orderDir': 'desc',
    'fields': 'symbol,symbolName,lastPrice,priceChange,percentChange,averageVolume100d,sharesOutstanding,tradingRatio,tradeTime,symbolCode,symbolType,hasOptions',
    'orderBy': 'tradingRatio',
    'meta': 'field.shortName,field.type,field.description,lists.lastUpdate',
    'hasOptions': 'true',
    'page': '1',
    'limit': '100',
    'raw': '1',
}

with requests.Session() as s:
    s.headers.update(headers)
    res = s.get(link)
    s.headers['X-Xsrf-Token'] = res.cookies['XSRF-TOKEN']
    resp = s.get(url)
    print(resp.status_code)
    print(resp.json())

如何让脚本生成 JSON 响应？

Answer 1

好问题，我需要做一些挖掘。

cookie 未正确编码，因此您需要

unquote

它们：

from urllib.parse import unquote

然后：

s.headers['X-XSRF-Token'] = unquote(res.cookies['XSRF-TOKEN'])

---

使用您的代码：

import requests
from urllib.parse import unquote

url = 'https://www.barchart.com/proxies/core-api/v1/quotes/get'
link = 'https://www.barchart.com/stocks/most-active/daily-volume-leaders'

headers = {
    'Accept': 'application/json',
    'Accept-Encoding': 'gzip, deflate, br',
    'Accept-Language': 'en-US,en;q=0.5',
    'Host': 'www.barchart.com',
    'Referer': 'https://www.barchart.com/stocks/most-active/trading-liquidity',
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:121.0) Gecko/20100101 Firefox/121.0',
}

params = {
    'lists': 'stocks.us.trading_liquidity.advances.overall',
    'orderDir': 'desc',
    'fields': 'symbol,symbolName,lastPrice,priceChange,percentChange,averageVolume100d,sharesOutstanding,tradingRatio,tradeTime,symbolCode,symbolType,hasOptions',
    'orderBy': 'tradingRatio',
    'meta': 'field.shortName,field.type,field.description,lists.lastUpdate',
    'hasOptions': 'true',
    'page': '1',
    'limit': '100',
    'raw': '1',
}

with requests.Session() as s:
    s.headers.update(headers)
    res = s.get(link)
    s.headers['X-XSRF-Token'] = unquote(res.cookies['XSRF-TOKEN'])
    resp = s.get(url, params=params)
    print(resp.status_code)
    print(resp.json())