下载网页的HTML返回“\ u0003T0”

问题描述 投票:0回答:3

我试图让这个页面的HTML

https://ec.europa.eu/esco/portal/skill?uri=http%3A%2F%2Fdata.europa.eu%2Fesco%2Fskill%2F00735755-adc6-4ea0-b034-b8caff339c9f&conceptLanguage=en&full=true

但由于某些原因,我收到的输出是这样的:

\0\0\0\0\0\0\u0003�T���0\u0010�#�\u000f�\aNM�.+�b�\"v�\u0010�\u0015+��\u001b����[�\u000e���\u001e�\v���

下面的代码:

using (WebClient client = new WebClient())
{
    client.Headers.Add("Host", "ec.europa.eu");
    client.Headers.Add("User-Agent", "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv,65.0) Gecko/20100101 Firefox/65.0");
    client.Headers.Add("Accept", "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8");
    client.Headers.Add("Accept-Language", "pl,en-US;q=0.7,en;q=0.3");
    client.Headers.Add("Accept-Encoding", "gzip, deflate, br");
    client.Headers.Add("DNT", "1");
    client.Headers.Add("Cookie", "JSESSIONID=-(...); escoLanguage=en");

    var output = client.DownloadString(new Uri("https://ec.europa.eu/esco/portal/skill?uri=http%3A%2F%2Fdata.europa.eu%2Fesco%2Fskill%2F00735755-adc6-4ea0-b034-b8caff339c9f&conceptLanguage=en&full=true"));
}

任何人有一个想法是什么引起的?

我也试图与HTML敏捷包:

var url = urls.First();
var web = new HtmlWeb();
var doc = web.Load(url);

doc.Textnull

c# html web-scraping
3个回答
2
投票

头“接受编码:gzip”可以送你用gzip压缩的原始数据。您必须手动解压缩输出流。例如,

curl -H "Accept-Encoding: gzip" "$url" --output - | gzip -d
if you are using a Linux shell.

更好的方法是只删除此头。


1
投票
 using (WebClient client = new WebClient())
        {
            client.Encoding = Encoding.UTF8;
            client.Headers.Add("Host", "ec.europa.eu");
            client.Headers.Add("User-Agent", "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv,65.0) Gecko/20100101 Firefox/65.0");
            client.Headers.Add("Accept", "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8");
            client.Headers.Add("Accept-Language", "pl,en-US;q=0.7,en;q=0.3");
            client.Headers.Add("Accept-Encoding", "gzip, deflate, br");
            client.Headers.Add("DNT", "1");
            client.Headers.Add("Cookie", "JSESSIONID=-(...); escoLanguage=en");
            var downloadStr = client.DownloadData(new Uri("https://ec.europa.eu/esco/portal/skill?uri=http%3A%2F%2Fdata.europa.eu%2Fesco%2Fskill%2F00735755-adc6-4ea0-b034-b8caff339c9f&conceptLanguage=en&full=true"));

            MemoryStream stream = new MemoryStream();
            using (GZipStream g = new GZipStream(new MemoryStream(downloadStr), CompressionMode.Decompress))
            {

                g.CopyTo(stream);


            }

            var output=  Encoding.UTF8.GetString(stream.ToArray());
        }

由于输出被压缩,它看起来像这样使用gzip未压缩。


0
投票

删除:client.Headers.Add("Accept-Encoding", "gzip, deflate, br");是为Web客户端解决方案

© www.soinside.com 2019 - 2024. All rights reserved.