如何使用python绕过cloudflare

问题描述 投票：0回答：4

我无法抓取此网站https://www.mentalhealthforum.net/，我收到 403 状态代码，尽管我已经尝试了互联网上的所有可用解决方案。 Cloudflare 有 h-captcha 保护，因此绕过它比较复杂

这是我的代码

def scrape(self):
    baseurl = 'https://www.mentalhealthforum.net/'
    scraper = cloudscraper.create_scraper(delay=10,
                                        browser={
                                                'browser': 'chrome',
                                                'platform': 'android',
                                                'desktop': False
                                                },
                                        debug=True, 
                                        captcha={'provider': '2captcha',
                                                 'api_key': api_key})
    response = scraper.get(baseurl)
    return response.status_code

print(scrape())

输出：

< GET / HTTP/1.1
< Host: www.mentalhealthforum.net
< User-Agent: Mozilla/5.0 (Linux; Android 4.3; SM-G710 Build/JLS36C) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.111 Mobile Safari/537.36
< Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8
< Accept-Language: en-US,en;q=0.9
< Accept-Encoding: gzip, deflate
<

> HTTP/1.1 403 Forbidden
> Date: Thu, 04 Aug 2022 04:44:23 GMT
> Content-Type: text/html; charset=UTF-8
> Transfer-Encoding: chunked
> Connection: close
> CF-Chl-Bypass: 1
> Permissions-Policy: accelerometer=(),autoplay=(),camera=(),clipboard-read=(),clipboard-write=(),fullscreen=(),geolocation=(),gyroscope=(),hid=(),interest-cohort=(),magnetometer=(),microphone=(),payment=(),publickey-credentials-get=(),screen-wake-lock=(),serial=(),sync-xhr=(),usb=()
> Cache-Control: private, max-age=0, no-store, no-cache, must-revalidate, post-check=0, pre-check=0
> Expires: Thu, 01 Jan 1970 00:00:01 GMT
> X-Frame-Options: SAMEORIGIN
> Expect-CT: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"
> Vary: Accept-Encoding
> Strict-Transport-Security: max-age=15552000; preload
> Server: cloudflare
> CF-RAY: 7354a33748c83384-DEL
> Content-Encoding: gzip
> alt-svc: h3=":443"; ma=86400, h3-29=":443"; ma=86400
>
<!DOCTYPE html>
<!--[if lt IE 7]> <html class="no-js ie6 oldie" lang="en-US"> <![endif]-->
<!--[if IE 7]>    <html class="no-js ie7 oldie" lang="en-US"> <![endif]-->
<!--[if IE 8]>    <html class="no-js ie8 oldie" lang="en-US"> <![endif]-->
<!--[if gt IE 8]><!--> <html class="no-js" lang="en-US"> <!--<![endif]-->
<head>

<title>Attention Required! | Cloudflare</title>

<meta charset="UTF-8" />
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
<meta http-equiv="X-UA-Compatible" content="IE=Edge" />
<meta name="robots" content="noindex, nofollow" />
<meta name="viewport" content="width=device-width,initial-scale=1" />
<link rel="stylesheet" id="cf_styles-css" href="/cdn-cgi/styles/cf.errors.css" />
<!--[if lt IE 9]><link rel="stylesheet" id='cf_styles-ie-css' href="/cdn-cgi/styles/cf.errors.ie.css" /><![endif]-->
<style>body{margin:0;padding:0}</style>


<!--[if gte IE 10]><!-->
<script>
  if (!navigator.cookieEnabled) {
    window.addEventListener('DOMContentLoaded', function () {
      var cookieEl = document.getElementById('cookie-alert');
      cookieEl.style.display = 'block';
    })
  }
</script>
<!--<![endif]-->


    <script>
    //<![CDATA[
    (function(){
      window._cf_chl_opt={
        cvId: "2",
        cType: "interactive",
        cNounce: "78912",
        cRay: "7354a33748c83384",
        cHash: "9d961b5f9b4ebe8",
        cUPMDTk: "\/?__cf_chl_tk=PkVh2nHuDkM8GSKSBdN6bF6yQ4tTFPfmUCVY6Zc6tQA-1659588263-0-gaNycGzNB_0",
        cFPWv: "b",
        cTTimeMs: "1000",
        cLt: "n",
        cRq: {
          ru: "aHR0cHM6Ly93d3cubWVudGFsaGVhbHRoZm9ydW0ubmV0Lw==",
          ra: "TW96aWxsYS81LjAgKExpbnV4OyBBbmRyb2lkIDQuMzsgU00tRzcxMCBCdWlsZC9KTFMzNkMpIEFwcGxlV2ViS2l0LzUzNy4zNiAoS0hUTUwsIGxpa2UgR2Vja28pIENocm9tZS82My4wLjMyMzkuMTExIE1vYmlsZSBTYWZhcmkvNTM3LjM2",
          rm: "R0VU",
          d: "cwDRskjJag43bMKn7QRhwyi8kyHqreuRwnGo+sqgbfN4uUqgwuI5Uv1VkkzWsGgvouW5wanxEIPAqrWZ7vK+KBXMwthn82Mzg2/gQpF36BPJJpPvfBg+vEE72VRJczxt02ALraAJiHgJW16MZfyPgjypbMsaCMt3lnB/3EWgzwkaeOtwJFzc7Wg5WN5RyuJNtXjZBYmU0LZVK9WYSYnyNQlZ0Mf5t7S+Y+ZTr8P5Z97W0VD12aSiHnFdXNUmAOWSOEOAxMi4a2F2U3O/kbEYsef1ouYIxKT9Nnmsw3mW2qbdnhOC24wIODeYC6DvHr5jZxRyFik3AdxHrtcKBRfLVLkvaiX6fkTdTlLMJ94p4hb8OYgh3r7qoAXyDX9gKf0pNGwF8BN6oFVMxauL+L9/Q+tXbSs5zWN3GZFe7XYcKQLMHXnrcw5s+WfCYwEkUzL0qoCg4B+JnQxF18GTXsXhhLmvDF00q71Fp3EzyBxZX54UELtPdu+IJMfo5uwb+Z62wDqWVYOQ9KDfUn9sJLl9xCFiN/gQNoyG9dgXrf9OmaxkQfEczBKa2lfAUu8a2CloY8qkGVHk55mg8SPrS2T09g==",
          t: "MTY1OTU4ODI2My41NzMwMDA=",
          m: "KejW80FyaOkUxmM47SXOqGP/cB+YHYlrIsoq37bq8Zs=",
          i1: "yDPbyhvL8j2VIY5Bqln6sg==",
          i2: "wetyBqKqvU25YTvZJoE21g==",
          zh: "OxIRYgLHg5p2pcbMMkuwgcVYeS4WO2VJlFLKmTgWwgg=",
          uh: "4vBxYA3Nh/bTpvXjoeGamwkVevjGpPRbEVqG2Joz1JM=",
          hh: "uGdsbGXsZlcoz7a5joNqzoj1ka1E1MNME2WxnV/IMIU=",
        }
      };
    }());
    //]]>
    </script>

<style>
  #cf-wrapper #spinner {width:69px; margin:  auto;}
  #cf-wrapper #cf-please-wait{text-align:center}
  .attribution {margin-top: 32px;}
  .bubbles { background-color: #f58220; width:20px; height: 20px; margin:2px; border-radius:100%; display:inline-block; }
  #cf-wrapper #challenge-form { padding-top:25px; padding-bottom:25px; }
  #cf-hcaptcha-container { text-align:center;}
  #cf-hcaptcha-container iframe { display: inline-block;}
  @keyframes fader     { 0% {opacity: 0.2;} 50% {opacity: 1.0;} 100% {opacity: 0.2;} }
  #cf-wrapper #cf-bubbles { width:69px; }
  @-webkit-keyframes fader { 0% {opacity: 0.2;} 50% {opacity: 1.0;} 100% {opacity: 0.2;} }
  #cf-bubbles > .bubbles { animation: fader 1.6s infinite;}
  #cf-bubbles > .bubbles:nth-child(2) { animation-delay: .2s;}
  #cf-bubbles > .bubbles:nth-child(3) { animation-delay: .4s;}
</style>
</head>
<body>
  <div id="cf-wrapper">
    <div class="cf-alert cf-alert-error cf-cookie-error" id="cookie-alert" data-translate="enable_cookies">Please enable cookies.</div>
    <div id="cf-error-details" class="cf-error-details-wrapper">
      <div class="cf-wrapper cf-header cf-error-overview">

        <h1 data-translate="challenge_headline">One more step</h1>
        <h2 class="cf-subheadline"><span data-translate="complete_sec_check">Please complete the security check to access</span> www.mentalhealthforum.net</h2>

      </div>

      <div class="cf-section cf-highlight cf-captcha-container">
        <div class="cf-wrapper">
          <div class="cf-columns two">
            <div class="cf-column">

              <div class="cf-highlight-inverse cf-form-stacked">
                <form id="challenge-form" class="challenge-form interactive-form" action="/?__cf_chl_f_tk=PkVh2nHuDkM8GSKSBdN6bF6yQ4tTFPfmUCVY6Zc6tQA-1659588263-0-gaNycGzNB_0" method="POST" enctype="application/x-www-form-urlencoded">
    <div id='cf-please-wait'>
      <div id='spinner'>
        <div id="cf-bubbles">
            <div class="bubbles"></div>
            <div class="bubbles"></div>
            <div class="bubbles"></div>
        </div>
      </div>
      <p data-translate="please_wait" id="cf-spinner-please-wait">Please stand by, while we are checking your browser...</p>
      <p data-translate="redirecting" id="cf-spinner-redirecting" style="display:none">Redirecting...</p>
      </div>
  <input type="hidden" name="md" value="oABL_BcWHmbxrANJi_DEsW4NSdZTow94pQNG.PPLozI-1659588263-0-AU8nZDIX4CuayrBCQWeR-rw7pdmEvwF9aAPyDTGsjUhKgb7lYEb4LbGEoGFg0yNqGuYBaPZJx_mvExqV9d-Hv4RYJZPBlsg53WXtldbxe2eNaTPPq7GiiSebI_x96CnQnwE36UsOwNvSV1WXGxqPrSf4YBqy59R5AU8TyGB-jnDZ549Kttr5AfJZMgLYXxuvxVfZ3H-d_V4-s22D-mdcyMwfvYf_SBd5ZD78UI23FYCHRX5pq2CbRfe1ntQ3gQSdD-6-JNASzzuXrHmbi1NrWLgx4bogZkUzkbjkSu2HmqIGq9-yWch8I12m2Osd1dYBE2VO13Xbmd73RHLaG1TPpKOBQf7izCaW4OUsdeEFHR0BfN7Z3b2b2JhfVHxT9RvUTr8xry2xF6OJ1MlG8GuHAImRAitEZJJKvue28KIWNR-VqS6BdaoFD-1C570aq_yegVKtV--50af-s_VBazSMECgS7usN9s4wA_piaOVH4TE2zK3kJZh7qeZlPa5GE6n5SEIhx6K7vg-uBXMMgzddGptCWz8zP9Kj8XLupIg_1Oi9_R3564djA04BCq98_9FHfW4DrYAyOodluZ8XHyNTQrYeTgP-N--GxNSEtyEHgw4tnw872nrFXNTtZkuPIoo6RA" />
  <input type="hidden" name="r" value="2RFjKAqAJ1wEDp7ctGkSzsKBpY.5nrLffwgE0dPn5So-1659588263-0-ARP6jAuzOyVp//1QkUdxc88mwa4hXj7KBBTY3yZEIYCNUeU/EWTep9CqlrAXm9nncXKVQNpiRHV5an6wE13xfCU0q7InPSRt170NvoHvcCiz5KsUgDEOxbgn3yr5n6B97g6TPJspvIF0p/+GIKd8yHpRIkIDNhlq/wszxqhvZ8GEyzBrZV97rKecUdzUhQ3wY6xPcJqY1cUB9vii/wac9GZ3GcaFs0oLKPWf6sZ6MN3q6inek4ahRfogGCTeNtWDp82pLgWMIofs6CdKZRA4NBdPnQvpB+OCAmd5ueuUVjfpNfrbCeqxN5TNbtThtbv9g22zA1SxYWP/CGtji2CuqLaaxjqsNUxoBQeLL9ERYy9qdhSvsTkmdwNpohoRo9nPsXAC09jGQ6GFSMkMEFL7OIkKIn9RZ7ttj+/OfMEH8kU8kGDANLze92S4EQLLUrSmiWoatPiOpxNfRgYq2DAY6HhgC87GMvfecjLlywtzmp9poC9OAnzdLVWGShxngN3KFj5BcIpIBswuQ1aC3n9UXjuhc+0rEbF2a+BOKjopzuH+njzJNjusseuxP7kC3+GZcb8Wc3OZNOd1CQhIWvnU5Y+Pn1E4myJLxFVIKqGn7KVhBY54oWULfMg9vF2jPFaojY1XDPKUbMGJed+VFQfVePwIc3+wikeeemM/I/2JjrxzxsbV8AP1YEoRpLhZObWa0p7wMmhT2gicvPPRaTkOgW3P4LKIFK1S7/gp48ENoxnpgOYdLbH6zsawjEMisg9K0WmGo9WzFlLhW6f1RzcRtM/kIs4n3aYdUzRnpHdRaAkjPyLVhEPG3gVAVLpIx3PmdMRgE2AylrikXjzG8L5CJcUKdE0KVOKYVWk68gEb13KAx801yQvQ4oOXkm/iH8hKljzUcGCCKWE0gocwxUtd3DleTo+vYdHdps08aHJrZ5Mqh+QsaCB8aIZVhPnJbLe7TMyFE61sgkuCaPUs+gF9AzNPB/gfqWsmjGXE7xDqKNNo4yGU6iKRfooKmMJVJ8lxwjF8XtlXkHcMmni6dT2AnbRVCGfRzt1ETTpjZkLAzcWXxg+5IrO/Xt4Nll6qMBz6ZIwPt8K9uh6QBKe76WMQd5szwqXnGXROF+mOg70Ro+H5roF610FiCEv7oEKdZ//AOSWRSfcYGznkrHjbhChx5pnjaQNJn72sAIGpe0w5XxHKdVssfpniEul0xeUihigo4JLBMLS93wmhQtuYHkxvgjCGz2QPI+iW08BKgda+s7277/vDLLTZnBWNvRWKbSYL6t7ZUb+y6zDlxnMrl8MBc3TKJcDdKBWc1DACottU8M9FFAohTYTKklygxYhbkS3Xrzs2wIQfPUPzwMKsTLdOuB+L/vPc8jWIIYQs8ca11+pge+WcN8ZQ/mRZLns1bno3bczan31UKKGs2/BFgvq4gd8HSEfchAzvKyL2nlihsNrGaehdFx2vavvi1XmQPs2oPZEXJx0GaSfuxKxlGetWDBoRt1Auwna7UXdt2Rxzx/HFeiUxNaNhDkIT02kvqewSTLt2ZoJifnQPADmx/88ek+PoZQZelHeFjZN1y57U3i38jfpmYmm5Yw8uXIb5Z/iXcK/UCab0+/wfZAhcuu421vkoakSuzI+bmMbj6IqLSD/zkaFDL8wSGCtLnsZ0rMNMScj2/f9jMescJYeF/2VAq/1vlH/93yuR4KJXA4MWfk6s49lOaxjv8Vh6WU8rRTh4YrimPtKi/BUPgG+0BC95nmwiPcjrtrO8C52ITotpWCmLq6tIBICpct8XE3kCERA0kOy038JHrCXMvBhA3MucRyrjfa5J8gViq7sePQc/P4bGqUsrOrAFm0O1Bp6zzn/Zp7xQGTbU5a+ZFjsdZHUv6ax3zys4vmxVTMI0OPnoxRx7q9pcqGI0SgM5HPPIK80GLchJ2DBPxVNNtTdayu9GUWwCZOb8tXmg2pTjHDR24Oft46ccM46p8Y/ekkJfRC2vCw==">
  <noscript id="cf-captcha-bookmark" class="cf-captcha-info">
  <h1 data-translate="turn_on_js" style="color:#bd2426;">Please turn JavaScript on and reload the page.</h1>
  </noscript>
    <div id="no-cookie-warning" class="cookie-warning" data-translate="turn_on_cookies" style="display:none">
      <p data-translate="turn_on_cookies" style="color:#bd2426;">Please enable Cookies and reload the page.</p>
    </div>
  <script>
  //<![CDATA[
    var a = function() {try{return !!window.addEventListener} catch(e) {return !1} },
      b = function(b, c) {a() ? document.addEventListener("DOMContentLoaded", b, c) : document.attachEvent("onreadystatechange", b)};
      b(function(){
        var cookiesEnabled=(navigator.cookieEnabled)? true : false;
        if(!cookiesEnabled){
          var q = document.getElementById('no-cookie-warning');q.style.display = 'block';
        }
      });
  //]]>
  </script>
  <div id="trk_captcha_js" style="background-image:url('/cdn-cgi/images/trace/captcha/nojs/transparent.gif?ray=7354a33748c83384')"></div>
</form>
  <script>
    //<![CDATA[
    (function(){
        var isIE = /(MSIE|Trident\/|Edge\/)/i.test(window.navigator.userAgent);
        var trkjs = isIE ? new Image() : document.createElement('img');
        trkjs.setAttribute("src", "/cdn-cgi/images/trace/captcha/js/transparent.gif?ray=7354a33748c83384");
        trkjs.id = "trk_captcha_js";
        trkjs.setAttribute("alt", "");
        document.body.appendChild(trkjs);
        var cpo=document.createElement('script');
        cpo.type='text/javascript';
        cpo.src = '/cdn-cgi/challenge-platform/h/b/orchestrate/captcha/v1?ray=7354a33748c83384';

        window._cf_chl_opt.cOgUHash = location.hash === '' && location.href.indexOf('#') !== -1 ? '#' : location.hash;
        window._cf_chl_opt.cOgUQuery = location.search === '' && location.href.slice(0, -window._cf_chl_opt.cOgUHash.length).indexOf('?') !== -1 ? '?' : location.search;
        if (window._cf_chl_opt.cUPMDTk && window.history && window.history.replaceState) {
          var ogU = location.pathname + window._cf_chl_opt.cOgUQuery + window._cf_chl_opt.cOgUHash;
          history.replaceState(null, null, "\/?__cf_chl_rt_tk=PkVh2nHuDkM8GSKSBdN6bF6yQ4tTFPfmUCVY6Zc6tQA-1659588263-0-gaNycGzNB_0" + window._cf_chl_opt.cOgUHash);
          cpo.onload = function() {
            history.replaceState(null, null, ogU);
          };
        }

        document.getElementsByTagName('head')[0].appendChild(cpo);
    }());
    //]]>
    </script>


              </div>
            </div>

            <div class="cf-column">
              <div class="cf-screenshot-container">

                <span class="cf-no-screenshot"></span>

              </div>
            </div>
          </div>
        </div>
      </div>

      <div class="cf-section cf-wrapper">
        <div class="cf-columns two">
          <div class="cf-column">
            <h2 data-translate="why_captcha_headline">Why do I have to complete a CAPTCHA?</h2>

            <p data-translate="why_captcha_detail">Completing the CAPTCHA proves you are a human and gives you temporary access to the web property.</p>
          </div>

          <div class="cf-column">
            <h2 data-translate="resolve_captcha_headline">What can I do to prevent this in the future?</h2>


            <p data-translate="resolve_captcha_antivirus">If you are on a personal connection, like at home, you can run an anti-virus scan on your device to make sure it is not infected with malware.</p>

            <p data-translate="resolve_captcha_network">If you are at an office or shared network, you can ask the network administrator to run a scan across the network looking for misconfigured or infected devices.</p>


              <p data-translate="resolve_captcha_privacy_pass">Another way to prevent getting this page in the future is to use Privacy Pass. Check out the browser extension in the <a rel="noopener noreferrer" href="https://chrome.google.com/webstore/detail/privacy-pass/ajhmfdgkijocedmfjonnpjfojldioehi">Chrome Web Store</a>.</p>


          </div>
        </div>
      </div>


      <div class="cf-error-footer cf-wrapper w-240 lg:w-full py-10 sm:py-4 sm:px-8 mx-auto text-center sm:text-left border-solid border-0 border-t border-gray-300">
  <p class="text-13">
    <span class="cf-footer-item sm:block sm:mb-1">Cloudflare Ray ID: <strong class="font-semibold">7354a33748c83384</strong></span>
    <span class="cf-footer-separator sm:hidden">&bull;</span>
    <span id="cf-footer-item-ip" class="cf-footer-item hidden sm:block sm:mb-1">
      Your IP:
      <button type="button" id="cf-footer-ip-reveal" class="cf-footer-ip-reveal-btn">Click to reveal</button>
      <span class="hidden" id="cf-footer-ip">49.36.219.70</span>
      <span class="cf-footer-separator sm:hidden">&bull;</span>
    </span>
    <span class="cf-footer-item sm:block sm:mb-1"><span>Performance &amp; security by</span> <a rel="noopener noreferrer" href="https://www.cloudflare.com/5xx-error-landing" id="brand_link" target="_blank">Cloudflare</a></span>

  </p>
  <script>(function(){function d(){var b=a.getElementById("cf-footer-item-ip"),c=a.getElementById("cf-footer-ip-reveal");b&&"classList"in b&&(b.classList.remove("hidden"),c.addEventListener("click",function(){c.classList.add("hidden");a.getElementById("cf-footer-ip").classList.remove("hidden")}))}var a=document;document.addEventListener&&a.addEventListener("DOMContentLoaded",d)})();</script>
</div><!-- /.error-footer -->


    </div>
  </div>

  <script>
  window._cf_translation = {};


</script>


</body>
</html>

403

有谁知道如何解决这个问题吗？

python web-scraping beautifulsoup cloudflare anti-bot
4个回答
5
投票

整个 Cloudflare 很难被绕过，因为它在不断更新。但是，您可以尝试解决它，这需要良好的逆向工程技能。如果您尝试抓取的网站不需要非常快。我建议使用基于浏览器的解决方案。像playwright或selenium之类的东西。如果您确实想“解决”cloudflare，则必须对逆向工程、TLS、密码等进行一些安静的背景研究。

0
投票

我自己也发现了这篇文章，寻找类似的解决方案。实际上，我在阅读这篇文章后找到了答案，并且对答案不满意（当然他们没有解决我遇到的问题）。

当您拥有 v2 协议时，Zenrows 似乎是一个很好的解决方案。这是具有层级的服务之一，但如果您只需要偶尔抓取，那么您可以请求的免费 API 密钥将解决这个问题。

我在这个网站上进行了测试，但没有收到 403 错误：

import requests 

response = requests.get("https://api.zenrows.com/v1/?apikey=YOUR_API_KEY&url=https%3A%2F%2Fwww.mentalhealthforum.net%2F") 

print(response.text)

这里是我生成免费 API 密钥的地方：https://app.zenrows.com/builder

让我知道这是如何工作的。

0
投票

有一个 StackOverflow 标签，https://stackoverflow.com/questions/tagged/undetected-chromedriver（已经有超过 250 个问题），专用于 Python 存储库 (https://github.com/ultrafunkamsterdam/undetected- chromedriver），其唯一目的是绕过 Cloudflare。在那里，您会找到您想要的答案。

除此之外，还有其他Python框架，例如https://github.com/seleniumbase/SeleniumBase，也具有绕过验证码的能力。

在 

pip install seleniumbase
之后，尝试使用 
python
运行此脚本，这将打开受 Cloudflare 保护的 URL：

from seleniumbase import Driver

driver = Driver(uc=True)
driver.get("https://www.g2.com/categories/video")
driver.sleep(3)
driver.quit()

对于较慢的互联网连接，您可能需要调整一根线路来逃避检测：

from seleniumbase import Driver

driver = Driver(uc=True)
driver.uc_open_with_reconnect("https://www.g2.com/categories/video", 4)
driver.sleep(3)
driver.quit()
-5
投票

我在Python中使用selenium，每当该网站似乎需要验证码来验证你是否是人类时，我只是关闭并重新打开浏览器，并且我能够访问该网站并进行抓取。 希望这对某人有帮助。

最新问题
© www.soinside.com 2019 - 2024. All rights reserved.