数百万行的 JavaScript 数据网格

JavaScript data grid for millions of rows

提问by Rudiger

I need to present a large number of rows of data (ie. millions of rows) to the user in a grid using JavaScript.

我需要使用 JavaScript 在网格中向用户呈现大量数据行（即数百万行）。

The user shouldn't see pages or view only finite amounts of data at a time.

用户不应一次查看页面或仅查看有限数量的数据。

Rather, it should appear that all of the data are available.

相反，似乎所有数据都可用。

Instead of downloading the data all at once, small chunks are downloaded as the user comes to them (ie. by scrolling through the grid).

不是一次性下载所有数据，而是在用户访问时下载小块（即通过滚动网格）。

The rows will not be edited through this front end, so read-only grids are acceptable.

行不会通过这个前端编辑，所以只读网格是可以接受的。

What data grids, written in JavaScript, exist for this kind of seamless paging?

对于这种无缝分页，存在哪些用 JavaScript 编写的数据网格？

采纳答案by Tin

(Disclaimer: I am the author of SlickGrid)

（免责声明：我是 SlickGrid 的作者）

UPDATEThis has now been implemented in SlickGrid.

更新现在已经在SlickGrid 中实现了。

Please see http://github.com/mleibman/SlickGrid/issues#issue/22for an ongoing discussion on making SlickGrid work with larger numbers of rows.

请参阅http://github.com/mleibman/SlickGrid/issues#issue/22以获取有关使 SlickGrid 处理更多行的持续讨论。

The problem is that SlickGrid does not virtualize the scrollbar itself - the scrollable area's height is set to the total height of all the rows. The rows are still being added and removed as the user is scrolling, but the scrolling itself is done by the browser. That allows it to be very fast yet smooth (onscroll events are notoriously slow). The caveat is that there are bugs/limits in the browsers' CSS engines that limit the potential height of an element. For IE, that happens to be 0x123456 or 1193046 pixels. For other browsers it is higher.

问题是 SlickGrid 没有虚拟化滚动条本身——可滚动区域的高度设置为所有行的总高度。当用户滚动时，行仍在添加和删除，但滚动本身是由浏览器完成的。这使得它非常快而流畅（众所周知，滚动事件非常慢）。需要注意的是，浏览器的 CSS 引擎中存在限制元素潜在高度的错误/限制。对于 IE，这恰好是 0x123456 或 1193046 像素。对于其他浏览器，它更高。

There is an experimental workaround in the "largenum-fix" branch that raises that limit significantly by populating the scrollable area with "pages" set to 1M pixels height and then using relative positioning within those pages. Since the height limit in the CSS engine seems to be different and significantly lower than in the actual layout engine, this gives us a much higher upper limit.

“largenum-fix”分支中有一个实验性的解决方法，通过将“页面”设置为 1M 像素高度填充可滚动区域，然后在这些页面中使用相对定位，显着提高了该限制。由于 CSS 引擎中的高度限制似乎与实际布局引擎中的不同，并且明显低于实际布局引擎中的高度限制，因此这为我们提供了更高的上限。

I am still looking for a way to get to unlimited number of rows without giving up the performance edge that SlickGrid currently holds over other implementations.

我仍在寻找一种方法来获得无限数量的行，而不会放弃 SlickGrid 目前相对于其他实现的性能优势。

Rudiger, can you elaborate on how you solved this?

Rudiger，你能详细说明你是如何解决这个问题的吗？

回答by Andras Vass

https://github.com/mleibman/SlickGrid/wiki

https://github.com/mleibman/SlickGrid/wiki

"SlickGrid utilizes virtual rendering to enable you to easily work with hundreds of thousands of items without any drop in performance. In fact, there is no difference in performance between working with a grid with 10 rows versus a 100'000 rows."

“ SlickGrid 利用虚拟渲染使您能够轻松处理数十万个项目，而不会降低性能。事实上，处理 10 行与 100'000 行的网格之间的性能没有区别。”

Some highlights:

一些亮点：

• Adaptive virtual scrolling (handle hundreds of thousands of rows)
• Extremely fast rendering speed
• Background post-rendering for richer cells
• Configurable & customizable
• Full keyboard navigation
• Column resize/reorder/show/hide
• Column autosizing & force-fit
• Pluggable cell formatters & editors
• Support for editing and creating new rows." by mleibman

• 自适应虚拟滚动（处理数十万行）
• 极快的渲染速度
• 更丰富的单元格的背景后渲染
• 可配置和可定制
• 全键盘导航
• 列调整大小/重新排序/显示/隐藏
• 色谱柱自动调整大小和力配合
• 可插拔单元格格式器和编辑器
• 支持编辑和创建新行。” 由mleibman

It's free (MIT license). It uses jQuery.

它是免费的（MIT 许可证）。它使用jQuery。

回答by scripto

The best Grids in my opinion are below:

我认为最好的网格如下：

• Flexigrid:http://flexigrid.info/
• jQuery Grid:http://www.trirand.com/blog/
• jqGridView:http://plugins.jquery.com/project/jqGridView
• jqxGrid:http://www.jqwidgets.com/
• Ingrid:http://reconstrukt.com/ingrid/
• SlickGridhttp://github.com/mleibman/SlickGrid
• DataTableshttp://www.datatables.net/index
• ShieldUIhttp://demos.shieldui.com/web/grid-virtualization/performance-1mil-rows

• 弹性网格：http ://flexigrid.info/
• jQuery 网格：http : //www.trirand.com/blog/
• jqGridView：http ://plugins.jquery.com/project/jqGridView
• jqxGrid：http ://www.jqwidgets.com/
• 英格丽德：http : //reconstrukt.com/ingrid/
• SlickGrid http://github.com/mleibman/SlickGrid
• 数据表http://www.datatables.net/index
• ShieldUI http://demos.shieldui.com/web/grid-virtualization/performance-1mil-rows

My best 3 options are jqGrid, jqxGrid and DataTables. They can work with thousands of rows and support virtualization.

我最好的 3 个选项是 jqGrid、jqxGrid 和 DataTables。它们可以处理数千行并支持虚拟化。

回答by Daniel 'Dang' Griffith

I don't mean to start a flame war, but assuming your researchers are human, you don't know them as well as you think. Just because they havepetabytes of data doesn't make them capable of viewing even millions of records in any meaningful way. They might say they wantto see millions of records, but that's just silly. Have your smartest researchers do some basic math: Assume they spend 1 second viewing each record. At that rate, it will take 1000000 seconds, which works out to more than six weeks (of 40 hour work-weeks with no breaks for food or lavatory).

我不是要开始一场火焰War，但假设你的研究人员是人类，你并不像你想象的那样了解他们。仅仅因为他们拥有数PB 的数据并不能使他们能够以任何有意义的方式查看数百万条记录。他们可能会说他们想要查看数百万条记录，但这太愚蠢了。让最聪明的研究人员做一些基本的数学计算：假设他们花 1 秒钟查看每条记录。按照这个速度，这将需要 1000000 秒，这将超过六周（每周工作 40 小时，没有休息时间吃饭或上厕所）。

Do they (or you) seriously think one person (the one looking at the grid) can muster that kind of concentration? Are they really getting much done in that 1 second, or are they (more likely) filtering out the stuff the don'twant? I suspect that after viewing a "reasonably-sized" subset, they could describe a filter to you that would automatically filter out those records.

他们（或您）是否真的认为一个人（看着网格的人）可以集中注意力？他们真的在那一秒内完成了很多工作，还是他们（更有可能）过滤掉了不想要的东西？我怀疑在查看“合理大小”的子集后，他们可以向您描述一个过滤器，该过滤器会自动过滤掉这些记录。

As paxdiablo and Sleeper Smith and Lasse V Karlsen also implied, you (and they) have not thought through the requirements. On the up side, now that you've found SlickGrid, I'm sure the need for those filters became immediately obvious.

正如paxdiablo、Sleeper Smith 和Lasse V Karlsen 也暗示的那样，您（和他们）还没有仔细考虑这些要求。从好的方面来说，既然您已经找到了 SlickGrid，我相信对这些过滤器的需求立即变得显而易见。

回答by Lasse V. Karlsen

I can say with pretty good certainty that you seriously do not need to show millions of rows of data to the user.

我可以非常肯定地说，您真的不需要向用户显示数百万行数据。

There is no user in the world that will be able to comprehend or manage that data set so even if you technically manage to pull it off, you won't solve any known problem for that user.

世界上没有用户能够理解或管理该数据集，因此即使您在技术上设法实现了它，您也无法为该用户解决任何已知问题。

Instead I would focus on whythe user wants to see the data. The user does not want to see the data just to see the data, there is usually a question being asked. If you focus on answering those questions instead, then you would be much closer to something that solves an actual problem.

相反，我将集中在为什么用户想要看到的数据。用户不想看数据只是为了看数据，通常会有一个问题被问到。如果您专注于回答这些问题，那么您将更接近解决实际问题的方法。

回答by richardtallent

I recommend the Ext JS Grid with the Buffered View feature.

我推荐带有缓冲视图功能的 Ext JS Grid。

http://www.extjs.com/deploy/dev/examples/grid/buffer.html

http://www.extjs.com/deploy/dev/examples/grid/buffer.html

回答by vitmalina

(Disclaimer: I am the author of w2ui)

（免责声明：我是 w2ui 的作者）

I have recently written an article on how to implement JavaScript grid with 1 million records (http://w2ui.com/web/blog/7/JavaScript-Grid-with-One-Million-Records). I discovered that ultimately there are 3 restrictions that prevent from taking it highter:

我最近写了一篇关于如何实现具有 100 万条记录的 JavaScript 网格的文章（http://w2ui.com/web/blog/7/JavaScript-Grid-with-One-Million-Records）。我发现最终有 3 个限制可以防止它变得更高：

1. Height of the div has a limit (can be overcome by virtual scrolling)
2. Operations such as sort and search start being slow after 1 million records or so
3. RAM is limited because data is stored in JavaScript array

1. div 的高度有限制（可以通过虚拟滚动来克服）
2. 大约 100 万条记录后，排序和搜索等操作开始变慢
3. RAM 有限，因为数据存储在 JavaScript 数组中

I have tested the grid with 1 million records (except IE) and it performs well. See article for demos and examples.

我已经用 100 万条记录（IE 除外）测试了网格，它表现良好。有关演示和示例，请参阅文章。

回答by peller

dojox.grid.DataGridoffers a JS abstraction for data so you can hook it up to various backends with provided dojo.data stores or write your own. You'll obviously need one that supports random access for this many records. DataGrid also provides full accessibility.

dojox.grid.DataGrid为数据提供了 JS 抽象，因此您可以使用提供的 dojo.data 存储将其连接到各种后端或编写您自己的。您显然需要一个支持对这么多记录进行随机访问的记录。DataGrid 还提供完全可访问性。

Edit so here's a link to Matthew Russell's articlethat should provide the example you need, viewing millions of records with dojox.grid. Note that it uses the old version of the grid, but the concepts are the same, there were just some incompatible API improvements.

编辑这里是Matthew Russell 文章的链接，该文章应该提供您需要的示例，使用 dojox.grid 查看数百万条记录。请注意，它使用旧版本的网格，但概念是相同的，只是有一些不兼容的 API 改进。

Oh, and it's totally free open source.

哦，它是完全免费的开源。

回答by Anurag

Here are a couple of optimizations you can apply you speed up things. Just thinking out loud.

这里有一些优化，你可以应用你加快速度。只是大声思考。

Since the number of rows can be in the millions, you will want a caching system just for the JSON data from the server. I can't imagine anybody wanting to download all X million items, but if they did, it would be a problem. This little teston Chrome for an array on 20M+ integers crashes on my machine constantly.

由于行数可能以数百万计，您将需要一个仅用于来自服务器的 JSON 数据的缓存系统。我无法想象有人想要下载所有 X 百万个项目，但如果他们这样做了，那将是一个问题。在 Chrome 上对 20M+ 整数数组进行的这个小测试在我的机器上不断崩溃。

var data = [];
for(var i = 0; i < 20000000; i++) {
    data.push(i);
}
console.log(data.length);?

You could use LRUor some other caching algorithm and have an upper bound on how much data you're willing to cache.

您可以使用LRU或其他一些缓存算法，并对您愿意缓存的数据量设置上限。

For the table cells themselves, I think constructing/destroying DOM nodes can be expensive. Instead, you could just pre-define X number of cells, and whenever the user scrolls to a new position, inject the JSON data into these cells. The scrollbar would virtually have no direct relationship to how much space (height) is required to represent the entire dataset. You could arbitrarily set the table container's height, say 5000px, and map that to the total number of rows. For example, if the containers height is 5000px and there are a total of 10M rows, then the starting row ≈ (scroll.top/5000) * 10Mwhere scroll.toprepresents the scroll distance from the top of the container. Small demo here.

对于表格单元本身，我认为构建/销毁 DOM 节点可能很昂贵。相反，您可以只预定义 X 个单元格，并且每当用户滚动到新位置时，将 JSON 数据注入这些单元格。滚动条实际上​​与表示整个数据集所需的空间（高度）没有直接关系。您可以任意设置表格容器的高度，例如 5000 像素，并将其映射到总行数。例如，如果容器高度为 5000px 并且总共有 10M 行，那么starting row ≈ (scroll.top/5000) * 10Mwherescroll.top表示距容器顶部的滚动距离。小演示在这里。

To detect when to request more data, ideally an object should act as a mediator that listens to scroll events. This object keeps track of how fast the user is scrolling, and when it looks like the user is slowing down or has completely stopped, makes a data request for the corresponding rows. Retrieving data in this fashion means your data is going to be fragmented, so the cache should be designed with that in mind.

要检测何时请求更多数据，理想情况下，对象应充当侦听滚动事件的中介。这个对象会跟踪用户滚动的速度，当看起来用户正在减速或完全停止时，会为相应的行发出数据请求。以这种方式检索数据意味着您的数据将会碎片化，因此在设计缓存时应考虑到这一点。

Also the browser limits on maximum outgoing connections can play an important part. A user may scroll to a certain position which will fire an AJAX request, but before that finishes the user can scroll to some other portion. If the server is not responsive enough the requests would get queued up and the application will look unresponsive. You could use a request manager through which all requests are routed, and it can cancel pending requests to make space.

此外，浏览器对最大传出连接的限制也很重要。用户可以滚动到某个位置，这将触发 AJAX 请求，但在完成之前，用户可以滚动到其他部分。如果服务器没有足够的响应，请求将被排队，应用程序将看起来没有响应。您可以使用请求管理器，通过它路由所有请求，它可以取消挂起的请求以腾出空间。

回答by Amr Elgarhy

I used jQuery Grid Plugin, it was nice.

我使用了jQuery Grid Plugin，它很好。

Demos

演示