bash 如何比较两个巨大的文本文件(每个超过 50GB)?

声明:本页面是StackOverFlow热门问题的中英对照翻译,遵循CC BY-SA 4.0协议,如果您需要使用它,必须同样遵循CC BY-SA许可,注明原文地址和作者信息,同时你必须将它归于原作者(不是我):StackOverFlow 原文地址: http://stackoverflow.com/questions/36580942/
Warning: these are provided under cc-by-sa 4.0 license. You are free to use/share it, But you must attribute it to the original authors (not me): StackOverFlow

提示:将鼠标放在中文语句上可以显示对应的英文。显示中英文
时间:2020-09-18 14:29:19  来源:igfitidea点击:

How to compare two huge text files (more than 50gb each)?

bashunixtext-filesdifffile-comparison

提问by sara

I have 2 huge text files and want to find the difference between them? What is the fast command/ utility/ or script to do this job?

我有 2 个巨大的文本文件,想找出它们之间的区别吗?完成这项工作的快速命令/实用程序/或脚本是什么?

I try using unix diffbut, it failed for huge files. It show me "Permission Denied"

我尝试使用unix diff,但是对于大文件却失败了。它告诉我"Permission Denied"

Also, I try unix bdiff(as I read that is good for huge files) but, I did not complete test it because it show me "bdiff: command not found"

另外,我尝试过unix bdiff(因为我读到这对大文件很有用)但是,我没有完成测试,因为它显示了我"bdiff: command not found"

Also, I try windows FC(file compare) but, I did not know how to show the output in a new text file. The result output in the cmd and I can't figure it all.

另外,我尝试了windows FC(文件比较)但是,我不知道如何在新的文本文件中显示输出。cmd中的结果输出,我无法全部弄清楚。

Any suggestion will help me a lot. Please help me in this matter.

任何建议都会对我有很大帮助。请帮助我解决这个问题。

Thanks a lot.

非常感谢。

回答by SLePort

You can try diffwith the --speed-large-filesoption :

您可以尝试diff使用以下--speed-large-files选项:

diff --speed-large-files file1 file2

cmpis another alternative (compare files byte by byte) :

cmp是另一种选择(逐字节比较文件):

cmp file1 file2

回答by Munir

If fcworks for you, you can send the output of fcto text file with:

如果fc适合您,您可以将输出发送fc到文本文件:

fc file1 file2 >output.txt