Linux 如何限制复制使用的缓存，以便其他缓存仍有可用内存？
声明：本页面是StackOverFlow热门问题的中英对照翻译，遵循CC BY-SA 4.0协议，如果您需要使用它，必须同样遵循CC BY-SA许可，注明原文地址和作者信息，同时你必须将它归于原作者（不是我）：StackOverFlow
原文地址: http://stackoverflow.com/questions/10105203/
Warning: these are provided under cc-by-sa 4.0 license. You are free to use/share it, But you must attribute it to the original authors (not me):
StackOverFlow
How can I limit the cache used by copying so there is still memory available for other cache?
提问by Peter
Basic situation:
基本情况：
I am copying some NTFS disks in openSuSE. Each one is 2TB. When I do this, the system runs slow.
我正在 openSuSE 中复制一些 NTFS 磁盘。每个是2TB。当我这样做时，系统运行缓慢。
My guesses:
我的猜测：
I believe it is likely due to caching. Linux decides to discard useful cache (eg. kde4 bloat, virtual machine disks, LibreOffice binaries, Thunderbird binaries, etc.) and instead fill all available memory (24 GB total) with stuff from the copying disks, which will be read only once, then written and never used again. So then any time I use these apps (or kde4), the disk needs to be read again, and reading the bloat off the disk again makes things freeze/hiccup.
我相信这可能是由于缓存。Linux 决定丢弃有用的缓存（例如 kde4 膨胀、虚拟机磁盘、LibreOffice 二进制文件、Thunderbird 二进制文件等），而是用复制磁盘中的内容填充所有可用内存（总共 24 GB），这些内容只会被读取一次，然后写下来，再也没有使用过。因此，每当我使用这些应用程序（或 kde4）时，都需要再次读取磁盘，再次读取磁盘上的膨胀会使事情冻结/打嗝。
Due to the cache being gone and the fact that these bloated applications need lots of cache, this makes the system horribly slow.
由于缓存消失了，而且这些臃肿的应用程序需要大量缓存，这使得系统非常缓慢。
Since it is USB,the disk and disk controller are not the bottleneck, so using ionice does not make it faster.
既然是USB，磁盘和磁盘控制器不是瓶颈，所以使用ionice并不会让它更快。
I believe it is the cache rather than just the motherboard going too slow, because if I stop everything copying, it still runs choppy for a while until it recaches everything. And if I restart the copying, it takes a minute before it is choppy again. But also, I can limit it to around 40 MB/s, and it runs faster again (not because it has the right things cached, but because the motherboard busses have lots of extra bandwidth for the system disks). I can fully accept a performance loss from my motherboard's IO capability being completely consumed (which is 100% used, meaning 0% wasted power which makes me happy), but I can't accept that this caching mechanism performs so terribly in this specific use case.
我相信这是缓存而不仅仅是主板速度太慢，因为如果我停止所有复制，它仍然会运行一段时间，直到它重新缓存所有内容。如果我重新开始复制，它需要一分钟才能再次断断续续。而且，我可以将其限制在 40 MB/s 左右，并且它再次运行得更快（不是因为它缓存了正确的东西，而是因为主板总线为系统磁盘提供了大量额外带宽）。我完全可以接受完全消耗我的主板 IO 能力带来的性能损失（100% 使用，这意味着 0% 浪费了电力，这让我很高兴），但我不能接受这种缓存机制在这种特定用途中表现得如此糟糕案件。
# free
total used free shared buffers cached
Mem: 24731556 24531876 199680 0 8834056 12998916
-/+ buffers/cache: 2698904 22032652
Swap: 4194300 24764 4169536
I also tried the same thing on Ubuntu, which causes a total system hang instead. ;)
我也在 Ubuntu 上尝试了同样的事情，这会导致整个系统挂起。;)
And to clarify, I am not asking how to leave memory free for the "system", but for "cache". I know that cache memory is automatically given back to the system when needed, but my problem is that it is not reserved for caching of specific things.
澄清一下，我不是在问如何为“系统”腾出内存，而是为“缓存”腾出内存。我知道缓存会在需要时自动返回给系统，但我的问题是它不是为缓存特定事物而保留的。
Question:
题：
Is there some way to tell these copy operations to limit memory usage so some important things remain cached, and therefore any slowdowns are a result of normal disk usage and not rereading the same commonly used files? For example, is there a setting of max memory per process/user/file system allowed to be used as cache/buffers?
有没有办法告诉这些复制操作来限制内存使用，以便一些重要的东西保持缓存，因此任何减速都是正常磁盘使用而不是重新读取相同常用文件的结果？例如，是否允许将每个进程/用户/文件系统的最大内存设置用作缓存/缓冲区？
采纳答案by nealmcb
The nocache command is the general answer to this problem! See https://github.com/Feh/nocacheor find it in Debian and Ubuntu 13.10 (saucy).
nocache 命令是这个问题的通用答案！请参阅https://github.com/Feh/nocache或在 Debian 和 Ubuntu 13.10（俏皮）中找到它。
Thanks, Peter, for alerting us to the --drop-cache" option in rsync. But that was rejected upstream (Bug 9560 – drop-cache option), in favor of a more general solution for this: the new "nocache" command based on the rsync work with fadvise.
谢谢，彼得，提醒我们注意 rsync 中的 --drop-cache" 选项。但是上游拒绝了（错误 9560 – drop-cache 选项），支持更通用的解决方案：新的“nocache”命令基于与 fadvise 的 rsync 工作。
You just prepend "nocache" to any command you want. It also has nice utilities for describing and modifying the cache status of files. E.g. here are the effects with and without nocache:
您只需在您想要的任何命令前添加“nocache”。它还具有用于描述和修改文件缓存状态的不错的实用程序。例如，这里是使用和不使用 nocache 的效果：
$ ./cachestats ~/file.mp3
pages in cache: 154/1945 (7.9%) [filesize=7776.2K, pagesize=4K]
$ ./nocache cp ~/file.mp3 /tmp
$ ./cachestats ~/file.mp3
pages in cache: 154/1945 (7.9%) [filesize=7776.2K, pagesize=4K]\
$ cp ~/file.mp3 /tmp
$ ./cachestats ~/file.mp3
pages in cache: 1945/1945 (100.0%) [filesize=7776.2K, pagesize=4K]
So hopefully that will work for other backup programs (rsnapshot, duplicity, rdiff-backup, amanda, s3sync, s3ql, tar, etc) and other commands that you don't want trashing your cache.
所以希望这适用于其他备份程序（rsnapshot、duplicity、rdiff-backup、amanda、s3sync、s3ql、tar 等）和其他您不想破坏缓存的命令。
回答by Kristof Provost
It's not possible if you're using plain old cp, but if you're willing to re-implement or patch it yourself setting posix_fadvise(fd, 0, 0, POSIX_FADV_NOREUSE)on both input and output file will probably help.
如果您使用的是普通 old cp，这是不可能的，但是如果您愿意重新实现或自己修补它posix_fadvise(fd, 0, 0, POSIX_FADV_NOREUSE)，则在输入和输出文件上的设置可能会有所帮助。
posix_fadvise()tells the kernel about your intended access pattern. In this case, you'd only use the data once so there's no point in caching it.
The Linux kernel honours these flags, so shouldn't be caching the data any more.
posix_fadvise()告诉内核你想要的访问模式。在这种情况下，您只会使用数据一次，因此缓存它没有意义。Linux 内核尊重这些标志，因此不应再缓存数据。
回答by KurzedMetal
try using ddinstead of cp.
尝试使用dd代替cp.
Or mountthe filesystem with the syncflag.
或者mount带有sync标志的文件系统。
I'm not completely sure if these methods bypass the swap, but it may be worth giving a try.
我不完全确定这些方法是否绕过交换，但可能值得一试。
Just my 2c.
只是我的2c。
回答by KurzedMetal
Ok, now that i know that your using rsyncand i could dig a bit more:
好的，现在我知道你的使用rsync，我可以再挖掘一点：
It seems that rsync is ineffective when used with tons of files at the same time, there's an entry in their FAQ, it's not a linux/cache problem, it's an rsync problem eating too much RAM.
这似乎与吨文件的使用时Rsync是无效的同时，有一个条目中的他们的常见问题，它不是一个Linux /缓存问题，这是一个rsync的问题，吃太多RAM。
Googling around someone recommended to split the syncing in multiple rsyncinvocations
Hope it helps.
希望能帮助到你。
回答by Turbo J
I am copying some NTFS disks [...] the system runs slow. [...] Since it is USB [...]
我正在复制一些 NTFS 磁盘 [...] 系统运行缓慢。[...] 因为它是 USB [...]
The slowdown is a known memory management issue.
速度减慢是一个已知的内存管理问题。
Use a newer Linux Kernel. The older ones have a problem with USB data and "Transparent Huge Pages". See this LWN article. Very recently this issue was addressed, see "Memory Management" in LinuxChanges.
使用较新的 Linux 内核。较旧的有 USB 数据和“透明大页面”问题。请参阅这篇LWN 文章。最近这个问题得到了解决，请参阅LinuxChanges 中的“内存管理” 。
回答by Peter
Kristof Provost was very close, but in my situation, I didn't want to use dd or write my own software, so the solution was to use the "--drop-cache" option in rsync.
Kristof Provost 非常接近，但在我的情况下，我不想使用 dd 或编写自己的软件，因此解决方案是使用 rsync 中的“--drop-cache”选项。
I have used this many times since creating this question, and it seems to fix the problem completely. One exception was when I am using rsync to copy from a FreeBSD machine, which doesn't support "--drop-cache". So I wrote a wrapper to replace the /usr/local/bin/rsync command, and remove that option, and now it works copying from there too.
自从创建这个问题以来，我已经多次使用它，它似乎完全解决了这个问题。一个例外是当我使用 rsync 从不支持“--drop-cache”的 FreeBSD 机器复制时。所以我写了一个包装器来替换 /usr/local/bin/rsync 命令，并删除该选项，现在它也可以从那里复制。
It still uses huge amount of memory for buffers, and seems to keep almost no cache, but works smoothly anyway.
它仍然使用大量内存用于缓冲区，并且似乎几乎没有缓存，但无论如何都可以顺利运行。
$ free
total used free shared buffers cached
Mem: 24731544 24531576 199968 0 15349680 850624
-/+ buffers/cache: 8331272 16400272
Swap: 4194300 602648 3591652
回答by MPW
The kernel can not know, that you won't use the cached data from copying agains. This is your information advantage.
内核无法知道，您不会再次使用复制的缓存数据。这是您的信息优势。
But you could set the swapiness to 0: sudo sysctl vm.swappiness=0. This will cause linux to drop the cache before libraries etc. are written to the swap.
但是您可以将 swapiness 设置为 0：sudo sysctl vm.swappiness=0。这将导致 linux 在库等写入交换之前删除缓存。
Works nice for me too, especially very performant in combination with hugh ram (16-32 GB).
对我来说也很好用，尤其是与 hugh ram (16-32 GB) 结合时性能非常好。
