windows 使用多个线程访问单个文件

Accessing a single file with multiple threads

提问by Raymond Wilson

I need to access a file concurrently with multiple threads. This needs to be done concurrently, without thread serialisation for performance reasons.

我需要与多个线程同时访问一个文件。这需要并发完成，出于性能原因没有线程序列化。

The file in particular has been created with the 'temporary' file attribute that encourages windows to keep the file in the system cache. This means most of the time the file read wont go near the disk, but will read the portion of the file from the system cache.

特别是该文件是使用“临时”文件属性创建的，该属性鼓励 Windows 将文件保留在系统缓存中。这意味着大部分时间读取的文件不会靠近磁盘，而是会从系统缓存中读取文件的一部分。

Being able to concurrently access this file will significantly improve performance of certain algorithms in my code.

能够同时访问此文件将显着提高我代码中某些算法的性能。

So, there are two questions here:

所以，这里有两个问题：

1. Is it possible for windows to concurrently access the same file from different threads?
2. If so, how do you provide this ability? I've tried creating the temp file and opening the file again to provide two file handles, but the second open does not succeed.

1. windows 是否可以同时从不同线程访问同一个文件？
2. 如果是这样，你如何提供这种能力？我已经尝试创建临时文件并再次打开该文件以提供两个文件句柄，但第二次打开没有成功。

Here's the create:

这是创建：

FFileSystem := CreateFile(PChar(FFileName),
                          GENERIC_READ + GENERIC_WRITE,
                          FILE_SHARE_READ + FILE_SHARE_WRITE,
                          nil,
                          CREATE_ALWAYS,
                          FILE_ATTRIBUTE_NORMAL OR
                          FILE_FLAG_RANDOM_ACCESS OR
                          FILE_ATTRIBUTE_TEMPORARY OR
                          FILE_FLAG_DELETE_ON_CLOSE,
                          0);

Here's the second open:

这是第二次开放：

FFileSystem2 := CreateFile(PChar(FFileName),
                          GENERIC_READ,
                          FILE_SHARE_READ,
                          nil,
                          OPEN_EXISTING,
                          FILE_ATTRIBUTE_NORMAL OR
                          FILE_FLAG_RANDOM_ACCESS OR
                          FILE_ATTRIBUTE_TEMPORARY OR
                          FILE_FLAG_DELETE_ON_CLOSE,
                          0);

I've tried various combinations of the flags with no success so far. The second file open always fails, with messages to the affect that the file cannot be accessed as it is in use by another process.

到目前为止，我已经尝试了各种标志组合，但都没有成功。第二个文件打开总是失败，并显示无法访问该文件的消息，因为它正在被另一个进程使用。

Edit:

编辑：

OK, some more information (I was hoping to not get lost in the weeds here...)

好的，更多信息（我希望不要在这里迷失方向......）

The process in question is a Win32 server process running on WinXP 64. It's maintaining large spatial databases and would like to keep as much of the spatial database as possible in memory in an L1/L2 cache structure. L1 already exists. L2 exists as a 'temporary' file that stays in the windows system cache (it's somewhat of a dirty trick, but gets around win32 memory limitations somewhat). Win64 means I can have lots of memory used by the system cache so memory used to hold the L2 cache does count towards process memory.

有问题的进程是一个运行在 WinXP 64 上的 Win32 服务器进程。它维护着大型空间数据库，并希望在 L1/L2 缓存结构的内存中保留尽可能多的空间数据库。L1 已经存在。L2 作为一个“临时”文件存在于 Windows 系统缓存中（这有点肮脏，但在某种程度上解决了 win32 内存限制）。Win64 意味着系统缓存可以使用大量内存，因此用于保存 L2 缓存的内存确实计入进程内存。

Multiple (potentially many) threads want to concurrently access information contained in the L2 cache. Currently, access is serialised, which means one thread gets to read it's data while most (or the rest) of the threads are blocked pending completion of that operation.

多个（可能很多）线程想要同时访问 L2 缓存中包含的信息。目前，访问是序列化的，这意味着一个线程可以读取它的数据，而大多数（或其余）线程在该操作完成之前被阻塞。

The L2 cache file does get written to, but I'm happy to globally serialise/interleave read and write type operations as long as I can perform concurrent reads.

L2 缓存文件确实被写入，但只要我可以执行并发读取，我很乐意全局序列化/交错读写类型操作。

I'm aware there are nasty potential thread concurrency issues, and I'm aware there are dozens of ways to skin this cat in other contexts. I have this particular context, and I'm trying to determine if there is a way to permit concurrent thread read access within the file and within the same process.

我知道存在令人讨厌的潜在线程并发问题，并且我知道有许多方法可以在其他上下文中为这只猫剥皮。我有这个特定的上下文，我正在尝试确定是否有一种方法可以允许文件内和同一进程内的并发线程读取访问。

Another approach I have considered would be two split the L2 cache into multiple temporary files, where each file serialises thread access the way the current single L2 cache file does.

我考虑过的另一种方法是将 L2 缓存分成多个临时文件，其中每个文件以当前单个 L2 缓存文件的方式序列化线程访问。

And yes, this somewhat desparate approach is because 64 bit Delphi wont be with us any time soon :-(

是的，这种有点绝望的方法是因为 64 位 Delphi 不会很快出现:-(

Thanks, Raymond.

谢谢，雷蒙德。

回答by Rob Kennedy

Yes, it's possible for a program to open the same file multiple times from different threads. You'll want to avoid reading from the file at the same time you're writing to it, though. You can use TMultiReadExclusiveWriteSynchronizerto control access to the entire file. It's less serialized than, say, a critical section. For more granular control, take a look at LockFileExto control access to specific regions of the file as you need them. When writing, request an exclusive lock; when reading, a shared lock.

是的，一个程序可以从不同的线程多次打开同一个文件。但是，您需要避免在写入文件的同时读取文件。您可以使用TMultiReadExclusiveWriteSynchronizer来控制对整个文件的访问。它不像关键部分那样序列化。如需更精细的控制，请查看以LockFileEx根据需要控制对文件特定区域的访问。写入时，请求排他锁；读取时，共享锁。

As for the code you posted, specifying File_Share_Writein the initial sharing flags means that all subsequent open operations must also share the file for writing. Quoting from the documentation:

至于您发布的代码，File_Share_Write在初始共享标志中指定意味着所有后续打开操作也必须共享文件以进行写入。从文档中引用：

If this flag is not specified, but the file or device has been opened for write access or has a file mapping with write access, the function fails.

如果未指定此标志，但已打开文件或设备进行写访问或具有写访问的文件映射，则该函数失败。

Your second open request was saying that it did not want anybody else to be allowed to write to the file while that handle remained open. Since there was already another handle open that didallow writing, the second request could not be fulfilled. GetLastErrorshould have returned 32, which is Error_Sharing_Violation, exactly what the documentation says should happen.

您的第二个打开请求是说它不希望在该句柄保持打开状态时允许其他任何人写入文件。因为已经有另一个手柄打开那确实让写作，第二请求不能得到满足。GetLastError应该返回 32，这Error_Sharing_Violation正是文档所说的应该发生的。

Specifying File_Flag_Delete_On_Closemeans all subsequent open requests need to share the file for deletion. The documentation again:

指定File_Flag_Delete_On_Close意味着所有后续的打开请求都需要共享文件以进行删除。再次文档：

Subsequent open requests for the file fail, unless the FILE_SHARE_DELETEshare mode is specified.

除非FILE_SHARE_DELETE指定了共享模式，否则对该文件的后续打开请求将失败。

Then, since the second open request shares the file for deletion, all other open handles must have also shared it for deletion. The documentation:

然后，由于第二个打开请求共享要删除的文件，因此所有其他打开的句柄也必须共享它以进行删除。文档：

If there are existing open handles to a file, the call fails unless they were all opened with the FILE_SHARE_DELETEshare mode.

如果存在文件的现有打开句柄，则调用将失败，除非它们都以FILE_SHARE_DELETE共享模式打开。

The bottom line is that either everybody shares alike or nobody shares at all.

最重要的是，要么每个人都共享，要么根本没有人共享。

FFileSystem := CreateFile(PChar(FFileName),
  Generic_Read or Generic_Write
  File_Share_Read or File_Share_Write or File_Share_Delete,
  nil,
  Create_Always,
  File_Attribute_Normal or File_Flag_Random_Access
    or File_Attribute_Temporary or File_Flag_Delete_On_Close,
  0);

FFileSystem2 := CreateFile(PChar(FFileName),
  Generic_Read,
  File_Share_Read or File_Share_Write or File_Share_Delete,
  nil,
  Open_Existing,
  File_Attribute_Normal or File_Flag_Random_Access
    or File_Attribute_Temporary or File_Flag_Delete_On_Close,
  0);

In other words, all the parameters are the same except for the fifth one.

换句话说，除了第五个参数外，所有参数都相同。

These rules apply to two attempts to open on the samethread as well as attempts from different threads.

这些规则适用于在同一线程上的两次打开尝试以及来自不同线程的尝试。

回答by meklarian

Update #2

更新 #2

I wrote some test projects in C to try and figure this out- although Rob Kennedy beat me to the answer while I was away. Both conditions are possible, including cross-process, as he outlines. Here's a link if anyone else would like to see this in action.

我用 C 编写了一些测试项目来尝试解决这个问题——尽管 Rob Kennedy 在我不在的时候打败了我。正如他所概述的，这两种情况都是可能的，包括跨进程。如果其他人希望看到此操作，请点击此处的链接。

SharedFileTests.zip (VS2005 C++ Solution) @ meklarian.com

SharedFileTests.zip（VS2005 C++ 解决方案）@ meklarian.com

There are three projects:

共有三个项目：

InProcessThreadShareTest - Test a creator and client thread.
InProcessThreadShareTest.cpp Snippet @ gist.github

InProcessThreadShareTest - 测试创建者和客户端线程。
InProcessThreadShareTest.cpp Snippet @ gist.github

SharedFileHost - Create a host that runs for 1 minute and updates a file.
SharedFileClient - Create a client that runs for 30 seconds and polls a file.
SharedFileHost.cpp and SharedFileClient.cpp Snippet @ gist.github

SharedFileHost - 创建一个运行 1 分钟并更新文件的主机。
SharedFileClient - 创建一个运行 30 秒并轮询文件的客户端。
SharedFileHost.cpp 和 SharedFileClient.cpp Snippet @ gist.github

All of these projects assume the location C:\data\tmp\sharetest.txt is creatable and writable.

所有这些项目都假定位置 C:\data\tmp\sharetest.txt 是可创建和可写的。

---

---

Update

更新

Given your scenario, sounds like you need a very large chunk of memory. Instead of gaming the system cache, you can use AWE to have access to more than 4Gb of memory, although you will need to map portions at a time. This should cover your L2 scenario as you wish to ensure that physical memory is used.

鉴于您的情况，听起来您需要非常大的内存块。您可以使用 AWE 访问超过 4Gb 的内存，而不是玩弄系统缓存，尽管您需要一次映射部分。这应该涵盖您的 L2 方案，因为您希望确保使用物理内存。

Address Windowing Extensions @ MSDN

地址窗口扩展@ MSDN

Use AllocateUserPhysicalPages and VirtualAlloc to reserve memory.

使用 AllocateUserPhysicalPages 和 VirtualAlloc 来预留内存。

AllocateUserPhysicalPages Function (Windows) @ MSDN
VirtualAlloc Function (Windows) @ MSDN

AllocateUserPhysicalPages 函数 (Windows) @ MSDN
VirtualAlloc 函数 (Windows) @ MSDN

---

---

Initial

最初的

Given that you are using the flag FILE_FLAG_DELETE_ON_CLOSE, is there any reason you wouldn't consider using a memory-mapped file instead?

鉴于您正在使用标志 FILE_FLAG_DELETE_ON_CLOSE，您是否有任何理由不考虑使用内存映射文件？

Managing Memory-Mapped files in Win32 @ MSDN

在 Win32 @ MSDN 中管理内存映射文件

From what I see in your CreateFile statements, it appears that you want to share data across-thread or across-process, with regard only to having the same file present while any sessions are open. A memory mapped file allows you to use the same logical filename in all sessions. Another benefit is that you can map views and lock portions of the mapped file with safety across all sessions. If you have a strict server with N-client scenario, it should be easy to implement. If you have a case where any client may be the opening server, you may wish to consider using some other mechanism to ensure that only one client gets to initiate the serving file first (via a global mutex, perhaps).

从我在您的 CreateFile 语句中看到的内容来看，您似乎希望跨线程或跨进程共享数据，只考虑在任何会话打开时存在相同的文件。内存映射文件允许您在所有会话中使用相同的逻辑文件名。另一个好处是您可以在所有会话中安全地映射视图并锁定映射文件的部分。如果您有 N 客户端场景的严格服务器，则应该很容易实现。如果您遇到任何客户端可能是打开服务器的情况，您可能希望考虑使用其他一些机制来确保只有一个客户端首先启动服务文件（可能通过全局互斥锁）。

CreateMutex @ MSDN

CreateMutex@MSDN

If you only need one-way transmission of data, perhaps you could use named pipes instead.
(edit) This is best for 1 server to 1 client.

如果您只需要单向传输数据，也许您可​​以改用命名管道。
（编辑）这最适合 1 个服务器到 1 个客户端。

Named Pipes (Windows) @ MSDN

命名管道 (Windows) @ MSDN

回答by GJ.

You can do on that way...

你可以这样做...

First thread with read/write access must at first create file:

具有读/写访问权限的第一个线程必须首先创建文件：

FileHandle := CreateFile(
  PChar(FileName),
  GENERIC_READ or GENERIC_WRITE,
  FILE_SHARE_READ,
  nil,
  CREATE_ALWAYS,
  FILE_ATTRIBUTE_NORMAL,
  0);

Sencond thread with only read access then opens the same file:

仅具有读取访问权限的第二个线程然后打开相同的文件：

  FileHandle := CreateFile(
    PCHar(FileName),
    GENERIC_READ,
    FILE_SHARE_READ + FILE_SHARE_WRITE,
    nil,
    OPEN_EXISTING,
    FILE_ATTRIBUTE_NORMAL,
    0);

I didn't test if works with...

我没有测试是否适用于...

FILE_ATTRIBUTE_TEMPORARY,
FILE_FLAG_DELETE_ON_CLOSE

attributes...

属性...

回答by Eric H.

I need to access a file concurrently with multiple threads. This needs to be done concurrently, without thread serialisation for performance reasons.

我需要与多个线程同时访问一个文件。这需要并发完成，出于性能原因没有线程序列化。

Either you don't need to use the same file within different threads, or you do need some kind of serialization.

要么您不需要在不同线程中使用相同的文件，要么您确实需要某种序列化。

Otherwise, you're just setting yourself up for heartache down the road.

否则，你只会让自己在路上心痛。