wpf 在 C# 中读取和写入非常大的文本文件

Reading and writing very large text files in C#

提问by Cass

I have a very large file, almost 2GB in size. I am trying to write a process to read the file in and write it out without the first row. I pretty much have been only able to read and write one line at a time which takes forever. I can open it, remove the first row and save it faster in TextPad, though that is still very slow.

我有一个非常大的文件，几乎 2GB。我正在尝试编写一个进程来读入文件并在没有第一行的情况下将其写出。我几乎一次只能读写一行，这需要很长时间。我可以打开它，删除第一行并在 TextPad 中更快地保存它，尽管这仍然很慢。

I use this code to get the number of records in the file:

我使用此代码获取文件中的记录数：

private long getNumRows(string strFileName)
{
    long lngNumRows = 0;
    string strMsg;

    try
    {
        lngNumRows = 0;
        using (var strReader = File.OpenText(@strFileName))
        {
            while (strReader.ReadLine() != null)
            {
                lngNumRows++;
            }

            strReader.Close();
            strReader.Dispose();
        }
    }
    catch (Exception excExcept)
    {
        strMsg = "The File could not be read: ";
        strMsg += excExcept.Message;
        System.Windows.MessageBox.Show(strMsg);
        //Console.WriteLine("Thee was an error reading the file: ");
        //Console.WriteLine(excExcept.Message);

        //Console.ReadLine();
    }

    return lngNumRows;
}

This only takes seconds to run. When I add the following code it takes forever to run. Am I doing something wrong? Why does the write add so much time? Any ideas on how I can make this faster?

这只需几秒钟即可运行。当我添加以下代码时，它需要永远运行。难道我做错了什么？为什么写入会增加这么多时间？关于如何加快速度的任何想法？

private void ProcessTextFiles(string strFileName)
{
    string strDataLine;
    string strFullOutputFileName;
    string strSubFileName;
    int intPos;
    long lngTotalRows = 0;
    long lngCurrNumRows = 0;
    long lngModNumber = 0;
    double dblProgress = 0;
    double dblProgressPct = 0;
    string strPrgFileName = "";
    string strOutName = "";
    string strMsg;
    long lngFileNumRows;

    try
    {
       using (StreamReader srStreamRdr = new StreamReader(strFileName))
        {
            while ((strDataLine = srStreamRdr.ReadLine()) != null)
            {
                lngCurrNumRows++;

                if (lngCurrNumRows > 1)
                {
                    WriteDataRow(strDataLine, strFullOutputFileName);
                }
            }

            srStreamRdr.Dispose();
        }
    }
    catch (Exception excExcept)
    {
        strMsg = "The File could not be read: ";
        strMsg += excExcept.Message;
        System.Windows.MessageBox.Show(strMsg);
        //Console.WriteLine("The File could not be read:");
        //Console.WriteLine(excExcept.Message);
    }
}

public void WriteDataRow(string strDataRow, string strFullFileName)
{
    //using (StreamWriter file = new StreamWriter(@strFullFileName, true, Encoding.GetEncoding("iso-8859-1")))
    using (StreamWriter file = new StreamWriter(@strFullFileName, true, System.Text.Encoding.UTF8))
    {
        file.WriteLine(strDataRow);
        file.Close();
    }
}

回答by Steve

Not sure how much this will improve the performance, but surely, opening and closing the output file for every line that you want to write is not a good idea.

不确定这会在多大程度上提高性能，但可以肯定的是，为您要写入的每一行打开和关闭输出文件并不是一个好主意。

Instead open both files just one time and then write the line directly

而是只打开两个文件一次，然后直接写一行

using (StreamWriter file = new StreamWriter(@strFullFileName, true, System.Text.Encoding.UTF8))
using (StreamReader srStreamRdr = new StreamReader(strFileName))
{
    while ((strDataLine = srStreamRdr.ReadLine()) != null)
    {
        lngCurrNumRows++;

        if (lngCurrNumRows > 1)
           file.WriteLine(strDataRow);
    }
}

You could also remove the check on lngCurrNumRowsimply making an empty read before entering the while loop

您还lngCurrNumRow可以在进入 while 循环之前删除对简单地进行空读取的检查

strDataLine = srStreamRdr.ReadLine();
if(strDataLine != null)
{
    while ((strDataLine = srStreamRdr.ReadLine()) != null)
    {
           file.WriteLine(strDataRow);
    }
}

回答by BugFinder

Depending on the memory of your machine. You could try the following (my big file was "D:\savegrp.log" I had a 2gb file knocking about) This used about 6gb memory when I tried it

取决于您机器的内存。您可以尝试以下操作（我的大文件是“D:\savegrp.log”我有一个 2gb 的文件在敲打）当我尝试它时使用了大约 6gb 的内存

int counter = File.ReadAllLines(@"D:\savegrp.log").Length;
Console.WriteLine(counter);

It does depends on the memory available..

它确实取决于可用的内存..

File.WriteAllLines(@"D:\savegrp2.log",File.ReadAllLines(@"D:\savegrp.log").Skip(1));
Console.WriteLine("file saved");