javascript 有没有java脚本网络爬虫框架
声明：本页面是StackOverFlow热门问题的中英对照翻译，遵循CC BY-SA 4.0协议，如果您需要使用它，必须同样遵循CC BY-SA许可，注明原文地址和作者信息，同时你必须将它归于原作者（不是我）：StackOverFlow
原文地址: http://stackoverflow.com/questions/5555930/
Warning: these are provided under cc-by-sa 4.0 license. You are free to use/share it, But you must attribute it to the original authors (not me):
StackOverFlow
is there any java script web crawler framework
提问by saleh Hosseinkahni
Is there any JavaScript web crawler framework?
有没有 JavaScript 网络爬虫框架？
回答by Shakakai
There's a new framework that was just release for Node.js called spider. It uses jQuery under the hood to crawl/index a website's HTML pages. The API and configuration are really nice especially if you already know jQuery.
有一个刚刚为 Node.js 发布的新框架叫做spider。它在后台使用 jQuery 来抓取/索引网站的 HTML 页面。API 和配置非常好，特别是如果您已经了解 jQuery。
From the test suite, here's an example of crawling the New York Times website:
在测试套件中，以下是抓取纽约时报网站的示例：
var spider = require('../main');
spider()
.route('www.nytimes.com', '/pages/dining/index.html', function (window, $) {
$('a').spider();
})
.route('travel.nytimes.com', '*', function (window, $) {
$('a').spider();
if (this.fromCache) return;
var article = { title: $('nyt_headline').text(), articleBody: '', photos: [] }
article.body = ''
$('div.articleBody').each(function () {
article.body += this.outerHTML;
})
$('div#abColumn img').each(function () {
var p = $(this).attr('src');
if (p.indexOf('ADS') === -1) {
article.photos.push(p);
}
})
console.log(article);
})
.route('dinersjournal.blogs.nytimes.com', '*', function (window, $) {
var article = {title: $('h1.entry-title').text()}
console.log($('div.entry-content').html())
})
.get('http://www.nytimes.com/pages/dining/index.html')
.log('info')
;
回答by zindel
Try the PhantomJS. Not exactly a crawler, but could be easily used for that purpose. It has the fully-functional WebKit engine built-in, with an ability to save screenshots etc. Works as the simple command-line JS interpreter.
试试PhantomJS。不完全是一个爬虫，但可以很容易地用于这个目的。它内置了功能齐全的 WebKit 引擎，具有保存屏幕截图等功能。作为简单的命令行 JS 解释器工作。
