共享内存或 mmap - Linux C/C++ IPC

Shared Memory or mmap - Linux C/C++ IPC

提问by Humble Debugger

The context is Inter-Process-Communication where one process("Server") has to send fixed-size structs to many listening processes("Clients") running on the same machine.

上下文是进程间通信，其中一个进程（“服务器”）必须向在同一台机器上运行的许多侦听进程（“客户端”）发送固定大小的结构。

I am very comfortable doing this in Socket Programming. To make the communication between the Server and the Clients faster and to reduce the number of copies, I want to try out using Shared Memory(shm) or mmaps.

我在 Socket Programming 中做这件事很舒服。为了使服务器和客户端之间的通信更快并减少副本数量，我想尝试使用共享内存（shm）或 mmaps。

The OS is RHEL 64bit.

操作系统是 RHEL 64 位。

Since I am a newbie, please suggest which should I use. I'd appreciate it if someone could point me to a book or online resource to learn the same.

由于我是新手，请建议我应该使用哪个。如果有人能指点我一本书或在线资源来学习同样的东西，我将不胜感激。

Thanks for the answers. I wanted to add that the Server ( Market Data Server ) will typically be receiving multicast data, which will cause it to be "sending" about 200,000 structs per second to the "Clients", where each struct is roughly 100 Bytes. Does shm_open/mmap implementation outperform sockets only for large blocks of data or a large volume of small structs as well ?

感谢您的回答。我想补充一点，服务器（市场数据服务器）通常会接收多播数据，这将导致它每秒向“客户端”“发送”大约 200,000 个结构，其中每个结构大约为 100 字节。shm_open/mmap 实现是否仅在大数据块或大量小结构方面优于套接字？

回答by Jens Gustedt

I'd use mmaptogether with shm_opento map shared memory into the virtual address space of the processes. This is relatively direct and clean:

我会mmap一起使用withshm_open将共享内存映射到进程的虚拟地址空间。这是比较直接和干净的：

• you identify your shared memory segment with some kind of symbolic name, something like "/myRegion"
• with shm_openyou open a file descriptor on that region
• with ftruncateyou enlarge the segment to the size you need
• with mmapyou map it into your address space

• 你用某种符号名称来标识你的共享内存段，比如 "/myRegion"
• 与shm_open您在该区域上打开文件描述符
• 与ftruncate您放大段的大小，你需要
• 与mmap您将其映射到您的地址空间

The shmatand Co interfaces have (at least historically) the disadvantage that they may have a restriction in the maximal amount of memory that you can map.

在shmat和Co接口有（至少在历史上）的缺点是它们可能具有的内存，你可以映射的最大数量的限制。

Then, all the POSIX thread synchronization tools (pthread_mutex_t, pthread_cond_t, sem_t, pthread_rwlock_t, ...) have initialization interfaces that allow you to use them in a process shared context, too. All modern Linux distributions support this.

然后，所有 POSIX 线程同步工具（pthread_mutex_t, pthread_cond_t, sem_t, pthread_rwlock_t, ...）都有初始化接口，允许您在进程共享上下文中使用它们。所有现代 Linux 发行版都支持这一点。

Whether or not this is preferable over sockets? Performance wise it could make a bit of a difference, since you don't have to copy things around. But the main point I guess would be that, once you have initialized your segment, this is conceptually a bit simpler. To access an item you'd just have to take a lock on a shared lock, read the data and then unlock the lock again.

这是否比套接字更可取？性能明智它可能会有所不同，因为您不必复制东西。但我想主要的一点是，一旦你初始化了你的段，这在概念上会更简单一些。要访问某个项目，您只需锁定共享锁，读取数据，然后再次解锁该锁。

As @R suggests, if you have multiple readers pthread_rwlock_twould probably the best lock structure to use.

正如@R 所建议的那样，如果您有多个读者，则pthread_rwlock_t可能是最好的锁结构。

回答by peoro

I once implemented an IPC library using shared memory segments; this allowed me to avoid a copy (instead of copying data from sender memory, to kernel space, and then from kernel space to receiver memory, I could directly copying from sender to receiver memory).

我曾经使用共享内存段实现了一个 IPC 库；这使我可以避免复制（而不是将数据从发送方内存复制到内核空间，然后从内核空间复制到接收方内存，我可以直接从发送方内存复制到接收方内存）。

Anyway results weren't as good as I was expecting: actually sharing a memory segment was a really expensive process, since remapping TLB entries and all the rest is quite expensive. See this mailfor more details (I'm no one of those guys, but got into such mail while developing my library).

无论如何，结果并不像我预期的那么好：实际上共享一个内存段是一个非常昂贵的过程，因为重新映射 TLB 条目和所有其余的都非常昂贵。有关更多详细信息，请参阅此邮件（我不是这些人中的一员，但在开发我的库时收到了此类邮件）。

Results were good only for really big messages (say more than a few megabytes), if you're working with little buffers, unix sockets are the most optimized thing you can find unless you are willing to write a kernel module.

结果仅适用于非常大的消息（比如超过几兆字节），如果您使用的是小缓冲区，除非您愿意编写内核模块，否则 unix 套接字是您能找到的最优化的东西。

回答by datenwolf

Apart from what's been suggested already, I'd like to offer another method: IPv6 Node/Interface Local Multicast, i.e. a multicast constrained to the loopback interface. http://www.iana.org/assignments/ipv6-multicast-addresses/ipv6-multicast-addresses.xml#ipv6-multicast-addresses-1

除了已经提出的建议之外，我想提供另一种方法：IPv6 节点/接口本地多播，即限制到环回接口的多播。 http://www.iana.org/assignments/ipv6-multicast-addresses/ipv6-multicast-addresses.xml#ipv6-multicast-addresses-1

At first this might seem quite heavyweight, but most OS implement loopback sockets in a zero-copy architecture. The page(s) mapped to the bufparameter passed to sendwill be assigned an additional mapping and marked as copy on write so that if the sending program overwrites the data therein, or deallocates the contents will be preserved.

乍一看，这可能看起来很重量级，但大多数操作系统都在零复制架构中实现了环回套接字。映射到buf传递给参数的页面send将被分配一个额外的映射并标记为写入时复制，以便如果发送程序覆盖其中的数据，或取消分配内容将被保留。

Instead of passing raw structs you should use a robust data structure. Netstrings http://cr.yp.to/proto/netstrings.txtand BSON http://bsonspec.org/come to mind.

您应该使用健壮的数据结构，而不是传递原始结构。Netstrings http://cr.yp.to/proto/netstrings.txt和 BSON http://bsonspec.org/浮现在脑海中。

回答by Diomidis Spinellis

Choosing between the POSIX shm_open/mmapinterface and the older System V shmopone won't make a big difference, because after the initialization system calls, you end up with the same situation: a memory area that is shared between various processes. If your system supports it, I'd recommend to go with shm_open/mmap, because this is a better designed interface.

在 POSIXshm_open/mmap接口和较旧的 System V 接口之间进行选择shmop不会有太大区别，因为在初始化系统调用之后，您最终会遇到相同的情况：不同进程之间共享的内存区域。如果您的系统支持它，我建议您使用shm_open/mmap，因为这是一个设计更好的界面。

You then use the shared memory area as a common blackboard where all processes can scribble their data. The difficult part is to synchronize the processes accessing this area. Here I recommend to avoid concocting your own synchronization scheme, which can be fiendishly difficult and error-prone. Instead, use the existing working socket-based implementation for synchronizing access between processes, and use the shared memory only for transferring large amounts of data between processes. Even with this scheme you'll need a central process to coordinate the allocation of buffers, so this scheme is worth it only if you have very large volumes of data to transfer. Alternatively, use a synchronization library, like Boost.Interprocess.

然后，您可以将共享内存区域用作公共黑板，所有进程都可以在其中涂抹其数据。困难的部分是同步访问该区域的进程。在这里，我建议避免编造自己的同步方案，这可能非常困难且容易出错。相反，使用现有的基于工作套接字的实现来同步进程之间的访问，并且仅将共享内存用于进程之间传输大量数据。即使使用此方案，您也需要一个中央进程来协调缓冲区的分配，因此只有当您要传输大量数据时，此方案才值得。或者，使用同步库，如Boost.Interprocess。