重定向上的httpclient抛出异常

问题描述 投票:1回答:1

我正在尝试使用HttpClient下载一个网页,这是我的代码:

private async Task<string> _doRequest(string url)
{
  string result = string.Empty;

  var client = HttpClient;
  using(var request = new HttpRequestMessage()
  {
    RequestUri = new Uri(url),
    Method = HttpMethod.Get
  }){
    using (HttpResponseMessage response = client.SendAsync(request).Result)
      if (response.Headers.Location == null)
      {
        using (HttpContent content = response.Content)
        {
          result = await content.ReadAsStringAsync();
        }
      }
      else
      {
        result = await _doRequest(response.Headers.Location.ToString());
      }
  };

  return result;
}

HttpClient是一个初始化为的静态变量,如下所示:

  var handler = new HttpClientHandler();
  handler.AutomaticDecompression = DecompressionMethods.Deflate | DecompressionMethods.GZip;
  handler.AllowAutoRedirect = false;
  HttpClient = new HttpClient(handler);
  HttpClient.DefaultRequestHeaders.Add("User-Agent", @"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.84 Safari/537.36");

当我尝试使用url = "https://www.gls-italy.com/?option=com_gls&view=track_e_trace&mode=search&numero_spedizione=TE170187747&tipo_codice=nazionale"执行代码时

我得到以下内容:enter image description here

这是让我尝试卷曲的原因:enter image description here

在这里,我迷失了。对我来说,它看起来像一个有效的302位置,但由于某些原因HttpClient相信,否则只是抛出异常。

最初要清楚我依赖于AllowAutoRedirect默认值并推动HttpClient进行重定向,它没有用,我得到了同样的异常,导致我尝试自己管理它。但没有成功。

谁知道发生了什么?如何使它工作?

提前致谢。

c# redirect web-scraping dotnet-httpclient
1个回答
0
投票

班级对象:

HttpClientHandler Http_Handler = new HttpClientHandler();
HttpClient Http_Client = new HttpClient();
CookieContainer HttpClCookieJar = new CookieContainer();

HttpClient设置存根:

private void HttpClient_Setup()
{
   Http_Handler.AutomaticDecompression = DecompressionMethods.GZip | DecompressionMethods.Deflate;
   Http_Handler.AllowAutoRedirect = false;
   Http_Handler.CookieContainer = HttpClCookieJar;
   Http_Handler.UseCookies = true;
   Http_Handler.UseDefaultCredentials = true;
   Http_Client.Timeout = new TimeSpan(30000);
   Http_Client = new HttpClient(Http_Handler);
   Http_Client.DefaultRequestHeaders.Add("User-Agent", @"Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:56.0) Gecko/20100101 Firefox/56.0");
   Http_Client.DefaultRequestHeaders.Add("Accept-Language", "it-IT,it;q=0.8,en-US;q=0.5,en;q=0.3");
   Http_Client.DefaultRequestHeaders.Add("Accept", "*/*");
   Http_Client.DefaultRequestHeaders.Add("Cache-Control", "no-cache");
}

异步HttpClient请求:

public async Task<string> HttpClient_Request(string RequestURL)
{
   string _responseHtml = string.Empty;
   ServicePointManager.SecurityProtocol = SecurityProtocolType.Ssl3 | 
                                          SecurityProtocolType.Tls11 | 
                                          SecurityProtocolType.Tls12;
   try
   {
      using (HttpRequestMessage _requestMsg = new HttpRequestMessage())
      {
         _requestMsg.Method = HttpMethod.Get;
         _requestMsg.RequestUri = new Uri(RequestURL);

         using (HttpResponseMessage _response = await Http_Client.SendAsync(_requestMsg))
         {
            using (HttpContent _content = _response.Content)
            {
               _responseHtml = await _content.ReadAsStringAsync();
            };
         };
      };
   }
   catch (HttpRequestException eW)
   {
      Console.WriteLine("Message: {}  Source: {1}", eW.Message, eW.Source);
   }
   catch (Exception eX)
   {
      Console.WriteLine("Message: {}  Source: {1}", eX.Message, eX.Source);
   }
   return _responseHtml;
}
© www.soinside.com 2019 - 2024. All rights reserved.