C# httpwebrequest 和 javascript
声明：本页面是StackOverFlow热门问题的中英对照翻译，遵循CC BY-SA 4.0协议，如果您需要使用它，必须同样遵循CC BY-SA许可，注明原文地址和作者信息，同时你必须将它归于原作者（不是我）：StackOverFlow
原文地址: http://stackoverflow.com/questions/516027/
Warning: these are provided under cc-by-sa 4.0 license. You are free to use/share it, But you must attribute it to the original authors (not me):
StackOverFlow
C# httpwebrequest and javascript
提问by Amit Raz
I am using C# HttpWebRequest to get some data of a webpage. The problem is that some of the data is updated using javascript/ajax after the page is loaded and I am not getting it in the response string. Is there a way to have the webrequest wait untill all the scripts in the page have finished executing?
我正在使用 C# HttpWebRequest 来获取网页的一些数据。问题是在页面加载后使用 javascript/ajax 更新了一些数据，但我没有在响应字符串中获取它。有没有办法让 webrequest 等到页面中的所有脚本都执行完毕？
Thanks
谢谢
Amit
阿米特
采纳答案by splattne
If I correctly interpret your question, there is no simple solution for your problem.
如果我正确解释了您的问题，那么您的问题没有简单的解决方案。
You are scraping the HTML from a server and since your C# code is not a real web browser, it doesn't execute client scripts.
您正在从服务器抓取 HTML，并且由于您的 C# 代码不是真正的 Web 浏览器，因此它不会执行客户端脚本。
This way you can't access information which the HTML you fetch doesn't contain.
这样您就无法访问您获取的 HTML 不包含的信息。
Edit:I don't know how complex these AJAX calls from the original web site are, but you could use Firebug or Fiddler for IE to see how the requests are made in order to call these AJAX calls in your C# application too. So you could add the pieces of information you'll need. But it's only a theoretical solution.
编辑：我不知道来自原始网站的这些 AJAX 调用有多复杂，但是您可以使用 Firebug 或 Fiddler for IE 查看请求是如何发出的，以便在您的 C# 应用程序中调用这些 AJAX 调用。所以你可以添加你需要的信息。但这只是一个理论上的解决方案。
回答by Misko
When you open a web page in a web browser, it is the browser that executes the javascript and downloads additional resources used by the page (images, scripts, etc). HttpWebRequest by itself will not do any of this, it will only download the html for the page you requested. It will never execute any of the javascript/ajax code on it's own.
当您在 Web 浏览器中打开网页时，浏览器会执行 javascript 并下载该页面使用的其他资源（图像、脚本等）。HttpWebRequest 本身不会执行任何操作，它只会下载您请求的页面的 html。它永远不会自行执行任何 javascript/ajax 代码。
回答by roryf
HttpWebRequest does not emulate a web browser, it just downloads the resource you point it at. This means it will not execute or even download JavaScript files.
HttpWebRequest 不模拟 Web 浏览器，它只是下载您指向它的资源。这意味着它不会执行甚至下载 JavaScript 文件。
You would have to use something like FireBug to get the URL for the data being pulled in via JavaScript, and point your HttpWebRequest at that.
您必须使用 FireBug 之类的东西来获取通过 JavaScript 拉入的数据的 URL，并将您的 HttpWebRequest 指向该 URL。
回答by missaghi
Just an idea but there is a way to have .net load a webpage as if it were in a browser: using System.Windows.Forms
只是一个想法，但有一种方法可以让 .net 像在浏览器中一样加载网页：使用 System.Windows.Forms
you could Load the webpage into a WebBrowser control
您可以将网页加载到 WebBrowser 控件中
WebBrowser wb = new WebBrowser();
wb.ScrollBarsEnabled = false;
wb.ScriptErrorsSuppressed = true;
wb.Navigate(url);
while (wb.ReadyState != WebBrowserReadyState.Complete) { Application.DoEvents(); }
wb.Document.DomDocument.ToString()
This will probably give you the pre ajax DOM but maybe there is a way to let it run the ajax first.
这可能会为您提供 pre ajax DOM，但也许有一种方法可以让它先运行 ajax。
回答by Chris Almond
Use HttpWebRequest to download the page, programatically search the source code for the relevant ajax information and then use a new HttpWebRequest to pull that data down.
使用 HttpWebRequest 下载页面，以编程方式搜索相关 ajax 信息的源代码，然后使用新的 HttpWebRequest 拉取该数据。
回答by Nabin Aryal
Use HttpWebRequestto download the page. Search the source code for the relevant AJAX information and then use a new HttpWebRequestto pull that data down.
使用HttpWebRequest下载页面。搜索相关 AJAX 信息的源代码，然后使用 newHttpWebRequest拉取该数据。
回答by Nabin Aryal
You could use of the PhantomJs. I had this Issue, but don't found solution for my problem. In my opinion, best solution is This.
您可以使用PhantomJs。我有这个问题，但没有找到解决我的问题的方法。在我看来，最好的解决方案是This。
My solution is look like this:
我的解决方案是这样的：
var page = require('webpage').create();
page.open("https://sample.com", function(){
page.evaluate(function(){
var i = 0,
oJson = jsonData,
sKey;
localStorage.clear();
for (; sKey = Object.keys(oJson)[i]; i++) {
localStorage.setItem(sKey,oJson[sKey])
}
});
page.open("https://sample.com", function(){
setTimeout(function(){
page.render("screenshoot.png")
// Where you want to save it
console.log(page.content); //page source
// You can access its content using jQuery
var fbcomments = page.evaluate(function(){
return $("body").contents().find(".content")
})
phantom.exit();
},10000)
});
});
