C# 抛出了“System.OutOfMemoryException”类型的异常
声明：本页面是StackOverFlow热门问题的中英对照翻译，遵循CC BY-SA 4.0协议，如果您需要使用它，必须同样遵循CC BY-SA许可，注明原文地址和作者信息，同时你必须将它归于原作者（不是我）：StackOverFlow
原文地址: http://stackoverflow.com/questions/14606006/
Warning: these are provided under cc-by-sa 4.0 license. You are free to use/share it, But you must attribute it to the original authors (not me):
StackOverFlow
Exception of type 'System.OutOfMemoryException' was thrown
提问by
Basically I use Entity Framework to query a huge database. I want to return a string list then log it to a text file.
基本上我使用实体框架来查询一个巨大的数据库。我想返回一个字符串列表，然后将其记录到文本文件中。
List<string> logFilePathFileName = new List<string>();
var query = from c in DBContext.MyTable where condition = something select c;
foreach (var result in query)
{
filePath = result.FilePath;
fileName = result.FileName;
string temp = filePath + "." + fileName;
logFilePathFileName.Add(temp);
if(logFilePathFileName.Count %1000 ==0)
Console.WriteLine(temp+"."+logFilePathFileName.Count);
}
However I got an exception when logFilePathFileName.Count=397000.
The exception is:
但是，当logFilePathFileName.Count=397000. 例外是：
Exception of type 'System.OutOfMemoryException' was thrown.
A first chance exception of type 'System.OutOfMemoryException' occurred in System.Data.Entity.dll
抛出了“System.OutOfMemoryException”类型的异常。
System.Data.Entity.dll 中发生类型为“System.OutOfMemoryException”的第一次机会异常
UPDATE:
更新：
What I want to use a different query say: select top 1000 then add to the list, but I don't know after 1000 then what?
我想使用不同的查询说：选择前 1000 名然后添加到列表中，但我不知道 1000 名之后呢？
采纳答案by Tigran
Most probabbly it's not about a RAMas is, so increasing your RAMor even compiling and running your code in 64bit machine will not have a positive effect, in this case.
很可能它不是RAM原样，因此在这种情况下，增加RAM甚至64在位机中编译和运行代码不会产生积极影响。
I think it's related to a fact that .NETcollections are limited to maximum 2GBRAM space (no difference either 32or 64bit).
我认为这与.NET集合仅限于最大2GBRAM 空间（没有区别32或64位）的事实有关。
To resolve this, split your list to much smaller chunks and most probabblyyour problem will gone.
为了解决这个问题，分裂您的列表，以更小的块和最probabbly您的问题将会消失。
Just one possible solution:
只有一种可能的解决方案：
foreach (var result in query)
{
....
if(logFilePathFileName.Count %1000 ==0) {
Console.WriteLine(temp+"."+logFilePathFileName.Count);
//WRITE SOMEWHERE YOU NEED
logFilePathFileName = new List<string>(); //RESET LIST !|
}
}
EDIT
编辑
If you want fragment a query, you can use Skip(...)and Take(...)
如果您想要片段查询，您可以使用Skip(...)和Take(...)
Just an explanatory example:
只是一个解释性的例子：
var fisrt1000 = query.Skip(0).Take(1000);
var second1000 = query.Skip(1000).Take(1000);
... and so on..
... 等等..
Naturally put it in your iteration and parametrize it based on bounds of data you know or need.
自然地将它放在您的迭代中，并根据您知道或需要的数据范围对其进行参数化。
回答by Roy Dictus
Why are you collecting the data in a List<string>if all you need to do is write it to a text file?
List<string>如果您需要做的只是将数据写入文本文件，为什么要在 a 中收集数据？
You might as well just:
你也可以：
- Open the text file;
- Iterate over the records, appending each string to the text file (without storing the strings in memory);
- Flush and close the text file.
- 打开文本文件；
- 迭代记录，将每个字符串附加到文本文件中（不将字符串存储在内存中）；
- 刷新并关闭文本文件。
You will need far less memory than now, because you won't be keeping all those strings unnecessarily in memory.
您将需要比现在少得多的内存，因为您不会将所有这些字符串不必要地保留在内存中。
回答by Warr
You shouldn't read all records from database to list. It required a lot of memory. You an combine reading records and writing them to file. For example read 1000 records from db to list and save(append) them to text file, clear used memory (list.Clear()) and continue with new records.
您不应该将所有记录从数据库读取到列表。它需要大量内存。您可以结合阅读记录并将它们写入文件。例如，从 db 中读取 1000 条记录到 list 并将它们保存（追加）到文本文件中，清除已用内存（list.Clear()）并继续处理新记录。
回答by Bob Flannigon
You probably need to set some vmargs for memory! Also... look into writing it straight to your file and not holding it in a List
您可能需要为内存设置一些 vmargs！另外......考虑将它直接写入您的文件而不是将其保存在列表中
回答by Abbas
From several other topics on StackOverflow I read that the Entity Framework is not designed to handle bulk data like that. The EF will cache/track all data in the context and will cause the exception in cases of huge bulks of data. Options are to use SQL directly or split up your records in smaller sets.
从 StackOverflow 上的其他几个主题中，我了解到实体框架不是为处理这样的批量数据而设计的。EF 将缓存/跟踪上下文中的所有数据，并在大量数据的情况下导致异常。选项是直接使用 SQL 或将您的记录拆分为较小的集合。
回答by Jeroen van Veghel
What Roy Dictus says sounds the best way. Also you can try to add a limit to your query. So your database result won't be so large.
Roy Dictus 所说的听起来是最好的方式。您也可以尝试为查询添加限制。所以你的数据库结果不会那么大。
For info on: Limiting query size with entity framework
有关信息： 使用实体框架限制查询大小
回答by Ahmad Hassanat
I used to use the gc arraylist in VS c++ similar to the gc List that you used, to works fin with small and intermediate data sets, but when using Big Dat, same problem 'System.OutOfMemoryException' was thrown. As the size of these gcs cannot exceed 2 GB and therefore become inefficient with Big data, I built my own linked list, which gives the same functionality, dynamic increase and get by index, basically, it is a normal linked list class, with a dynamic array inside to provide getting data by index, it duplicates the space, but you may delete the linked list after updating the array is you do not need it keeping only the dynamic array, this would solve the problem. see the code:
我曾经在 VS c++ 中使用类似于您使用的 gc List 的 gc arraylist，用于处理小型和中间数据集，但是在使用 Big Data 时，抛出了同样的问题“System.OutOfMemoryException”。由于这些gcs的大小不能超过2GB，因此处理大数据时效率低下，我自己构建了一个链表，功能相同，动态增加，按索引获取，基本上是一个普通的链表类，有一个动态数组里面提供通过索引获取数据，它复制了空间，但是你可以在更新数组后删除链表，因为你不需要它只保留动态数组，这样就可以解决问题。看代码：
struct LinkedNode
{
long data;
LinkedNode* next;
};
class LinkedList
{
public:
LinkedList();
~LinkedList();
LinkedNode* head;
long Count;
long * Data;
void add(long data);
void update();
//long get(long index);
};
LinkedList::LinkedList(){
this->Count = 0;
this->head = NULL;
}
LinkedList::~LinkedList(){
LinkedNode * temp;
while(head){
temp= this->head ;
head = head->next;
delete temp;
}
if (Data)
delete [] Data; Data=NULL;
}
void LinkedList::add (long data){
LinkedNode * node = new LinkedNode();
node->data = data;
node->next = this->head;
this->head = node;
this->Count++;}
void LinkedList::update(){
this->Data= new long[this->Count];
long i = 0;
LinkedNode * node =this->head;
while(node){
this->Data[i]=node->data;
node = node->next;
i++;
}
}
If you use this, please refer to my work https://www.liebertpub.com/doi/10.1089/big.2018.0064
如果你使用这个，请参考我的作品https://www.liebertpub.com/doi/10.1089/big.2018.0064
