Web Scraping Yahoo Finance 现在在运行良好几个月后发送 404

Question

我一直在 node.js 中为我的财务数据/趋势跟踪应用程序抓取雅虎财经https://TrendEdge.app

一段时间以来一切正常，现在看来他们已经了解我一直在做的事情了。来自本地 node.js 和实时站点的 404 响应。

以前，我在 axios 请求中没有使用任何标头（愚蠢，是的）..但现在我仍然收到 404 错误响应，即使在访问 Chrome 浏览器中的同一页面时包含网络选项卡中使用的所有相同标头.

访问任何其他页面，例如 NASDAQ.com，使用相同的标题都没有问题。

这是我的 axios 请求的样子......

const URL = `https://finance.yahoo.com/quote/SMPL/options?p=SMPL`

const headers={
    "accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7", 
    'accept-encoding':"gzip, deflate, br",
    'cache-control':'max-age=0',
    'accept-language':"en-US,en;q=0.9",
    'content-type':'application/x-www-form-urlencoded; charset=UTF-8',
    'origin':'https://finance.yahoo.com',
    'referer':'https://finance.yahoo.com/quote/SMPL/options?p=SMPL',
    'sec-ch-ua':'"Chromium";v="112", "Google Chrome";v="112", "Not:A-Brand";v="99"',
    'sec-ch-ua-mobile':'?0',
    'sec-ch-ua-platform':'Windows',
    'sec-fetch-dest':'document',
    'sec-fetch-mode':'navigate',
    'sec-fetch-site':'none',
    'sec-fetch-user':'?1',
    'user-agent':'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/112.0.0.0 Safari/537.36'
  }

const rawData = await axios.get(URL, headers).catch(function (error) {
    if (error.response) {
      console.log(' --------- RESPONSE --------------- ')
      // The request was made and the server responded with a status code
      // that falls out of the range of 2xx
      console.log(error.response.data);
      console.log(error.response.status);
      console.log(error.response.headers);
    } else if (error.request) {
      console.log(' --------- REQUEST --------------- ')

      // The request was made but no response was received
      // `error.request` is an instance of XMLHttpRequest in the browser 
      // and an instance of http.ClientRequest in node.js
      console.log(error.request);
    } else {
      console.log(' --------- MESSAGE --------------- ')

      // Something happened in setting up the request that triggered an Error
      console.log('Error', error.message);
    }
}).then(res=>res)

有什么建议吗？

TY提前