C# HttpClient 请求类似浏览器
声明：本页面是StackOverFlow热门问题的中英对照翻译，遵循CC BY-SA 4.0协议，如果您需要使用它，必须同样遵循CC BY-SA许可，注明原文地址和作者信息，同时你必须将它归于原作者（不是我）：StackOverFlow
原文地址: http://stackoverflow.com/questions/15026953/
Warning: these are provided under cc-by-sa 4.0 license. You are free to use/share it, But you must attribute it to the original authors (not me):
StackOverFlow
HttpClient Request like browser
提问by Norbert Pisz
When I calling site www.livescore.com by HttpClient class I always getting error "500". Probably server blocked request from HttpClients.
当我通过 HttpClient 类调用网站 www.livescore.com 时，我总是收到错误“500”。可能是服务器阻止了来自 HttpClients 的请求。
1)There is any other method to get html from webpage?
1）有没有其他方法可以从网页中获取html？
2)How I can set the headers to get html content?
2）如何设置标题以获取 html 内容？
When I set headers like in browser I always get stange encoded content.
当我像在浏览器中一样设置标题时，我总是得到奇怪的编码内容。
http_client.DefaultRequestHeaders.TryAddWithoutValidation("Accept", "text/html,application/xhtml+xml,application/xml");
http_client.DefaultRequestHeaders.TryAddWithoutValidation("Accept-Encoding", "gzip, deflate");
http_client.DefaultRequestHeaders.TryAddWithoutValidation("User-Agent", "Mozilla/5.0 (Windows NT 6.2; WOW64; rv:19.0) Gecko/20100101 Firefox/19.0");
http_client.DefaultRequestHeaders.TryAddWithoutValidation("Accept-Charset", "ISO-8859-1");
3) How I can slove this problem? Any suggestions?
3）我如何解决这个问题？有什么建议？
I using Windows 8 Metro Style App in C# and HttpClientClass
我在 C# 和 HttpClientClass 中使用 Windows 8 Metro Style App
采纳答案by Jesse C. Slicer
Here you go - note you have to decompress the gzip encoded-result you get back as permleroy:
在这里 - 请注意，您必须按照mleroy解压缩您返回的 gzip 编码结果：
private static readonly HttpClient _HttpClient = new HttpClient();
private static async Task<string> GetResponse(string url)
{
using (var request = new HttpRequestMessage(HttpMethod.Get, new Uri(url)))
{
request.Headers.TryAddWithoutValidation("Accept", "text/html,application/xhtml+xml,application/xml");
request.Headers.TryAddWithoutValidation("Accept-Encoding", "gzip, deflate");
request.Headers.TryAddWithoutValidation("User-Agent", "Mozilla/5.0 (Windows NT 6.2; WOW64; rv:19.0) Gecko/20100101 Firefox/19.0");
request.Headers.TryAddWithoutValidation("Accept-Charset", "ISO-8859-1");
using (var response = await _HttpClient.SendAsync(request).ConfigureAwait(false))
{
response.EnsureSuccessStatusCode();
using (var responseStream = await response.Content.ReadAsStreamAsync().ConfigureAwait(false))
using (var decompressedStream = new GZipStream(responseStream, CompressionMode.Decompress))
using (var streamReader = new StreamReader(decompressedStream))
{
return await streamReader.ReadToEndAsync().ConfigureAwait(false);
}
}
}
}
call such like:
像这样调用：
var response = await GetResponse("http://www.livescore.com/").ConfigureAwait(false); // or var response = GetResponse("http://www.livescore.com/").Result;
回答by markoo
I think you can be pretty certain that they have done everything to stop developers from screen-scraping.
我认为您可以非常肯定，他们已尽一切努力阻止开发人员进行屏幕抓取。
If I try from a standard C# project using this code :
如果我使用此代码从标准 C# 项目中尝试：
var request = WebRequest.Create("http://www.livescore.com ");
var response = request.GetResponse();
I get this response:
我得到这个回应：
The remote server returned an error: (403) Forbidden.
回答by siger
Several things to take note of.
有几件事要注意。
That site requires you to provide a user agent, or it returns a 500 HTTP error.
A GET request to livescore.com responds with a 302 to livescore.us. You need to handle the redirection or directly request livescore.us
- You need to decompress a gzip-compressed response
该站点要求您提供用户代理，否则会返回 500 HTTP 错误。
对 livecore.com 的 GET 请求以 302 响应到 livecore.us。您需要处理重定向或直接请求 livecore.us
- 您需要解压缩 gzip 压缩的响应
This code works using the .NET 4 Client Profile, I'll let you figure out if it fits a Windows Store app.
此代码使用 .NET 4 Client Profile 工作，我会让您确定它是否适合 Windows Store 应用程序。
var request = (HttpWebRequest)HttpWebRequest.Create("http://www.livescore.com");
request.AllowAutoRedirect = true;
request.UserAgent = "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.17 (KHTML, like Gecko) Chrome/24.0.1312.57 Safari/537.17";
string content;
using (var response = (HttpWebResponse)request.GetResponse())
using (var decompressedStream = new GZipStream(response.GetResponseStream(), CompressionMode.Decompress))
using (var streamReader = new StreamReader(decompressedStream))
{
content = streamReader.ReadToEnd();
}
回答by user3285954
Could try this as well to add compression support:
也可以试试这个来添加压缩支持：
var compressclient = new HttpClient(new HttpClientHandler()
{
AutomaticDecompression = DecompressionMethods.Deflate | DecompressionMethods.GZip
});
This adds the headers too.
这也添加了标题。
According to the same thread support is now in Windows Store framework: http://social.msdn.microsoft.com/Forums/windowsapps/en-US/429bb65c-5f6b-42e0-840b-1f1ea3626a42/httpclient-data-compression-and-caching?prof=required
根据相同的线程支持现在在 Windows Store 框架中：http: //social.msdn.microsoft.com/Forums/windowsapps/en-US/429bb65c-5f6b-42e0-840b-1f1ea3626a42/httpclient-data-compression-and -caching?prof=required
