抛出了“System.OutOfMemoryException”类型的异常。C# 使用 IDataReader 时
声明：本页面是StackOverFlow热门问题的中英对照翻译，遵循CC BY-SA 4.0协议，如果您需要使用它，必须同样遵循CC BY-SA许可，注明原文地址和作者信息，同时你必须将它归于原作者（不是我）：StackOverFlow
原文地址: http://stackoverflow.com/questions/14020671/
Warning: these are provided under cc-by-sa 4.0 license. You are free to use/share it, But you must attribute it to the original authors (not me):
StackOverFlow
Exception of type 'System.OutOfMemoryException' was thrown. C# when using IDataReader
提问by DA_Prog
I have an application in which I have to get a large amount of data from DB. Since it failed to get all of those rows (it's close to 2,000,000 rows...), I cut it in breaks, and I run each time the sql query and get only 200,000 rows each time.
我有一个应用程序，我必须从数据库中获取大量数据。由于它未能获得所有这些行（接近 2,000,000 行......），我将其中断，每次 sql 查询时我都运行，每次只获得 200,000 行。
I use DataTable to which I enter all of the data (meaning - all 2,000,000 rows should be there).
我使用 DataTable 输入所有数据（意思是 - 所有 2,000,000 行都应该在那里）。
The first few runs are fine. Then it fails with the OutOfMemoryException.
前几次运行很好。然后它因 OutOfMemoryException 而失败。
My code works as following:
我的代码工作如下：
private static void RunQueryAndAddToDT(string sql, string lastRowID, SqlConnection conn, DataTable dt, int prevRowCount)
{
if (string.IsNullOrEmpty(sql))
{
sql = generateSqlQuery(lastRowID);
}
if (conn.State == ConnectionState.Closed)
{
conn.Open();
}
using (IDbCommand cmd2 = conn.CreateCommand())
{
cmd2.CommandType = CommandType.Text;
cmd2.CommandText = sql;
cmd2.CommandTimeout = 0;
using (IDataReader reader = cmd2.ExecuteReader())
{
while (reader.Read())
{
DataRow row = dt.NewRow();
row["RowID"] = reader["RowID"].ToString();
row["MyCol"] = reader["MyCol"].ToString();
... //In one of these rows it returns the exception.
dt.Rows.Add(row);
}
}
}
if (conn != null)
{
conn.Close();
}
if (dt.Rows.Count > prevRowCount)
{
lastRowID = dt.Rows[dt.Rows.Count - 1]["RowID"].ToString();
sql = string.Empty;
RunQueryAndAddToDT(sql, lastRowID, conn, dt, dt.Rows.Count);
}
}
It seems to me as if the reader keeps collecting rows, and that's why It throws an exception only in the third or second round.
在我看来，读者似乎一直在收集行，这就是它仅在第三轮或第二轮抛出异常的原因。
Shouldn't the Using clean the memory as its done? What may solve my problem?
Using 不应该像它那样清理内存吗？什么可以解决我的问题？
Note: I should explain - I have no other choice but get all of those rows to the datatable, Since I do some manipulation on them later, and the order of the rows is important, and I can't split it because sometimes I have to take the data of some rows and set it into one row and so on and so on, so I can't give it up.
注意：我应该解释一下 - 我别无选择，只能将所有这些行放到数据表中，因为我稍后会对它们进行一些操作，并且行的顺序很重要，我无法拆分它，因为有时我有取一些行的数据，把它设置成一行等等，所以我不能放弃。
Thanks.
谢谢。
回答by Andrew
I think simply you run out of memory because your DataTable gets so large from all the rows you keep adding to it.
我认为只是你的内存不足，因为你的 DataTable 从你不断添加的所有行中变得如此之大。
You may want to try a different pattern in this case.
在这种情况下，您可能想尝试不同的模式。
Instead of buffering your rows in a list (or DataTable), can you simply yield the rows as they are available for use when they arrive?
不是在列表（或数据表）中缓冲您的行，您是否可以简单地生成行，因为它们在到达时可供使用？
回答by Paul Ruane
You are storing a copy of the data to dt. You are simply storing so much that the machine is running out of memory. So you have few options:
您正在将数据副本存储到dt. 您只是存储了太多，以至于机器内存不足。所以你有几个选择：
- Increase the available memory.
- Reduce the amount of data you are retrieving.
- 增加可用内存。
- 减少您正在检索的数据量。
To increase the available memory, you can add physical memory to the machine. Note that a .NET process on a 32bit machine will not be able to access more than 2GB of memory though (3GB if you enable the 3GB switch in boot.ini) so you may need to switch to 64bit (machine and process) if you wish to address more memory than that.
要增加可用内存，您可以向机器添加物理内存。请注意，32 位机器上的 .NET 进程将无法访问超过 2GB 的内存（如果您启用 3GB 开关，则为 3GB boot.ini），因此如果您希望解决，您可能需要切换到 64 位（机器和进程）比这更多的内存。
Retrieving less data is probably the way to go. Depending upon what you are trying to achieve, you may be able to perform the task on subsets of the data (perhaps even on individual rows). If you are performing some kind of aggregation (e.g. a producing a summary or report from the data) you may be able to employ Map-Reduce.
检索更少的数据可能是要走的路。根据您要实现的目标，您可能能够对数据的子集（甚至可能对单个行）执行任务。如果您正在执行某种聚合（例如从数据生成摘要或报告），您可以使用Map-Reduce。
回答by Shift Technology
Check that you are building a 64-bit process, and not a 32-bit one, which is the default compilation mode of Visual Studio. To do this, right click on your project, Properties -> Build -> platform target : x64. As any 32-bit process, Visual Studio applications compiled in 32-bit have a virtual memory limit of 2GB.
检查您正在构建 64 位进程，而不是 32 位进程，这是 Visual Studio 的默认编译模式。为此，请右键单击您的项目，属性 -> 构建 -> 平台目标：x64。与任何 32 位进程一样，以 32 位编译的 Visual Studio 应用程序的虚拟内存限制为 2GB。
64-bit processes do not have this limitation, as they use 64-bit pointers, so their theoretical maximum address space is 16 exabytes (2^64). In reality, Windows x64 limits the virtual memory of processes to 8TB. The solution to the memory limit problem is then to compile in 64-bit.
64 位进程没有这个限制，因为它们使用 64 位指针，所以它们的理论最大地址空间是 16 艾字节 (2^64)。实际上，Windows x64 将进程的虚拟内存限制为 8TB。内存限制问题的解决方案是在 64 位中编译。
However, object's size in Visual Studio is still limited to 2GB, by default. You will be able to create several arrays whose combined size will be greater than 2GB, but you cannot by default create arrays bigger than 2GB. Hopefully, if you still want to create arrays bigger than 2GB, you can do it by adding the following code to you app.config file:
但是，默认情况下，Visual Studio 中对象的大小仍限制为 2GB。您将能够创建多个组合大小大于 2GB 的数组，但默认情况下不能创建大于 2GB 的数组。希望如果您仍然想创建大于 2GB 的数组，您可以通过将以下代码添加到您的 app.config 文件来实现：
<configuration>
<runtime>
<gcAllowVeryLargeObjects enabled="true" />
</runtime>
</configuration>
回答by Carlos A Merighe
Since you are using a DataTable, let me share a random problem that I was having using one. Check your Build properties. I had a problem with a DataTable throwing an out of memory exception randomly. As it turned out, the project's Build Platform target was set to Prefer 32-bit. Once I unselected that option, the random out of memory exception went away.
由于您使用的是DataTable，让我分享一个我使用一个时遇到的随机问题。检查您的构建属性。我有一个 DataTable 随机抛出内存不足异常的问题。事实证明，该项目的构建平台目标设置为Prefer 32-bit. 一旦我取消选择该选项，随机内存不足异常就消失了。
