Linux 为什么 mmap() 比顺序 IO 快？

Why mmap() is faster than sequential IO?

提问by Lunar Mushrooms

Possible Duplicate:
mmap() vs. reading blocks

可能的重复：
mmap() 与读取块

I heard (read it on the internet somewhere) that mmap()is faster than sequential IO. Is this correct? If yes then why it is faster?

我听说（在互联网上某处阅读）mmap()比顺序 IO 快。这样对吗？如果是，那么为什么它更快？

• mmap()is not reading sequentially.
• mmap()has to fetch from the disk itself same as read()does
• The mapped area is not sequential - so no DMA (?).

• mmap()不是按顺序阅读。
• mmap()必须从磁盘本身获取与做read()一样
• 映射区域不是连续的 - 所以没有 DMA (?)。

So mmap()should actually be slower than read()from a file? Which of my assumptions above are wrong?

那么mmap()实际上应该比read()文件慢吗？我上面的哪个假设是错误的？

采纳答案by Tony Delroy

I heard (read it on the internet somewhere) that mmap() is faster than sequential IO. Is this correct? If yes then why it is faster?

我听说（在互联网上的某处阅读） mmap() 比顺序 IO 快。这样对吗？如果是，那么为什么它更快？

It can be - there are pros and cons, listed below. When you really have reason to care, always benchmark both.

它可以 - 有优点和缺点，如下所列。 当您确实有理由关心时，请始终对两者进行基准测试。

Quite apart from the actual IO efficiency, there are implications for the way the application code tracks when it needs to do the I/O, and does data processing/generation, that can sometimes impact performance quite dramatically.

除了实际的 IO 效率之外，应用程序代码在需要执行 I/O 以及执行数据处理/生成时跟踪的方式也有影响，这有时会显着影响性能。

1) mmap() is not reading sequentially. 2) mmap() has to fetch from the disk itself same as read() does 3) The mapped area is not sequential - so no DMA (?).

So mmap() should actually be slower than read() from a file? Which of my assumptions above are wrong?

1) mmap() 不是按顺序读取的。2) mmap() 必须像 read() 一样从磁盘本身获取数据 3) 映射区域不是连续的 - 所以没有 DMA (?)。

那么 mmap() 实际上应该比从文件中读取 () 慢吗？我上面的哪个假设是错误的？

1) is wrong... mmap()assigns a region of virtual address space corresponding to file content... whenever a page in that address space is accessed, physical RAM is found to back the virtual addresses and the corresponding disk content is faulted into that RAM. So, the order in which reads are done from the disk matches the order of access. It's a "lazy" I/O mechanism. If, for example, you needed to index into a huge hash table that was to be read from disk, then mmaping the file and starting to do access means the disk I/O is not done sequentially and may therefore result in longer elapsed time until the entire file is read into memory, but while that's happening lookups are succeeding and dependent work can be undertaken, and if parts of the file are never actually needed they're not read (allow for the granularity of disk and memory pages, and that even when using memory mapping many OSes allow you to specify some performance-enhancing / memory-efficiency tips about your planned access patterns so they can proactively read ahead or release memory more aggressively knowing you're unlikely to return to it).

1) 是错误的...mmap()分配一个与文件内容相对应的虚拟地址空间区域...每当访问该地址空间中的一个页面时，就会发现物理 RAM 支持虚拟地址，并且相应的磁盘内容被错误地放入该 RAM . 因此，从磁盘读取的顺序与访问顺序相匹配。这是一种“惰性”I/O 机制。例如，如果您需要索引一个将从磁盘读取的巨大哈希表，那么mmap读取文件并开始访问意味着磁盘 I/O 不是按顺序完成的，因此可能会导致整个文件被读入内存之前经过的时间更长，但是在这种情况下，查找会成功并且可以进行相关工作，并且如果文件的某些部分实际上从来不需要它们，它们就不会被读取（考虑到磁盘和内存页面的粒度，即使在使用内存映射时，许多操作系统也允许您指定一些有关您计划的性能增强/内存效率的提示）访问模式，以便他们可以主动提前阅读或更积极地释放内存，因为他们知道您不太可能返回它）。

2) absolutely true

2）绝对正确

3) "The mapped area is not sequential" is vague. Memory mapped regions are "contiguous" (sequential) in virtual address space. We've discussed disk I/O being sequential above. Or, are you thinking of something else? Anyway, while pages are being faulted in, they may indeed be transferred using DMA.

3）“映射区域不连续”含糊不清。内存映射区域在虚拟地址空间中是“连续的”（顺序的）。我们已经在上面讨论了磁盘 I/O 是顺序的。或者，你在想别的事情吗？无论如何，当页面出现故障时，它们确实可以使用 DMA 传输。

Further, there are other reasons why memory mapping may outperform usual I/O:

此外，内存映射可能优于普通 I/O 的原因还有其他一些：

• there's less copying:
  • often OS & library level routines pass data through one or more buffers before it reaches an application-specified buffer, the application then dynamically allocates storage, then copies from the I/O buffer to that storage so the data's usable after the file reading completes
  • memory mapping allows (but doesn't force) in-place usage (you can just record a pointer and possibly length)
    • continuing to access data in-place risks increased swapping later: the file/memory-map could be more verbose than data structures into which it could be parsed, so access patterns on data therein could have more delays to fault in more memory pages
• memory mapping can simplify the application's parsing job by letting the application treat the entire file content as accessible, rather than worrying about when to read another buffer full
• the application defers more to the OS's wisdom re number of pages that are in physical RAM at any single point in time, effectively sharing a direct-access disk cache with the application
• as well-wisher comments below, "using memory mapping you typically use less system calls"
• if multiple processes are accessing the same file, they should be able to share the physical backing pages

• 复制更少：
  • 通常操作系统和库级例程在数据到达应用程序指定的缓冲区之前通过一个或多个缓冲区传递数据，然后应用程序动态分配存储，然后从 I/O 缓冲区复制到该存储，以便数据在文件读取完成后可用
  • 内存映射允许（但不强制）就地使用（您可以只记录一个指针和可能的长度）
    • 继续就地访问数据风险增加了以后交换的风险：文件/内存映射可能比可以解析成的数据结构更冗长，因此其中数据的访问模式可能会延迟更多的内存页面出现故障
• 内存映射可以通过让应用程序将整个文件内容视为可访问来简化应用程序的解析工作，而不是担心何时读取另一个缓冲区已满
• 应用程序更多地遵循操作系统的智慧，即在任何单个时间点物理 RAM 中的页面数量，有效地与应用程序共享直接访问磁盘缓存
• 正如下面的祝福者评论，“使用内存映射，您通常使用较少的系统调用”
• 如果多个进程访问同一个文件，它们应该能够共享物理后台页面

The are also reasons why mmapmay be slower - do read Linus Torvald's post herewhich says of mmap:

这也是mmap可能会变慢的原因- 请阅读 Linus Torvald在此处的帖子，其中提到mmap：

...page table games along with the fault (and even just TLB miss) overhead is easily more than the cost of copying a page in a nice streaming manner...

...页表游戏以及错误（甚至只是 TLB 未命中）开销很容易超过以良好的流方式复制页面的成本...

And from another of his posts:

从他的另一个帖子：

• quite noticeable setup and teardown costs. And I mean noticeable. It's things like following the page tables to unmap everything cleanly. It's the book-keeping for maintaining a list of all the mappings. It's The TLB flush needed after unmapping stuff.

• page faulting is expensive. That's how the mapping gets populated, and it's quite slow.

• 相当可观的安装和拆卸成本。我的意思是引人注目。就像按照页表干净地取消映射所有内容一样。它是维护所有映射列表的簿记。这是取消映射后需要的 TLB 刷新。

• 页面错误是昂贵的。这就是映射的填充方式，而且速度很慢。

FWIW, the last time this arose for me at work, memory mapped input was 80% faster than freadet al for reading binary database records into a proprietary database, on 64 bit Linux with ~170GB files.

FWIW，上次我在工作中出现这种情况时fread，在具有 ~170GB 文件的 64 位 Linux 上，将二进制数据库记录读入专有数据库时，内存映射输入比等人快 80% 。

回答by Jens Gustedt

"Faster" in absolute terms doesn't exist. You'd have to specify constraints and circumstances.

绝对意义上的“更快”是不存在的。您必须指定约束和情况。

mmap() is not reading sequentially.

mmap() 没有按顺序读取。

what makes you think that? If you really access the mapped memory sequentially, the system will usually fetch the pages in that order.

什么让你有那个想法？如果您真的按顺序访问映射的内存，系统通常会按该顺序获取页面。

mmap() has to fetch from the disk itself same as read() does

mmap() 必须像 read() 一样从磁盘本身获取

sure, but the OS determines the time and buffer size

当然可以，但操作系统决定了时间和缓冲区大小

The mapped area is not sequential - so no DMA (?).

映射区域不是连续的 - 所以没有 DMA (?)。

see above

看上面

What mmaphelps with is that there is no extra user space buffer involved, the "read" takes place there where the OS kernel sees fit and in chunks that can be optimized. This maybe an advantage in speed, but first of all this is just an interface that is easier to use.

有mmap帮助的是，不涉及额外的用户空间缓冲区，“读取”发生在操作系统内核认为合适的地方以及可以优化的块中。这可能是速度上的优势，但首先这只是一个更易于使用的界面。

If you want to know about speed for a particular setup (hardware, OS, use pattern) you'd have to measure.

如果您想了解特定设置（硬件、操作系统、使用模式）的速度，则必须进行测量。

回答by J-16 SDiZ

1. mmap()can share between process.
2. DMA will be used whenever possible. DMA does not require contiguous memory -- many high end cards support scatter-gather DMA.
3. The memory area may be shared with kernel block cache if possible. So there is lessor copying.
4. Memory for mmapis allocated by kernel, it is always aligned.

1. mmap()进程间可以共享。
2. DMA 将尽可能使用。DMA 不需要连续内存——许多高端卡支持分散-聚集 DMA。
3. 如果可能，内存区域可以与内核块缓存共享。所以有出租人抄袭。
4. 内存mmap由内核分配，它总是对齐的。