.net System.IO.FileSystemWatcher 监视网络服务器文件夹 - 性能注意事项

System.IO.FileSystemWatcher to monitor a network-server folder - Performance considerations

提问by CAD bloke

I want to watch a folder tree on a network server for changes. The files all have a specific extension. There are about 200 folders in the tree and about 1200 files with the extension I am watching.

我想查看网络服务器上的文件夹树是否有变化。这些文件都有特定的扩展名。树中大约有 200 个文件夹，大约有 1200 个带有我正在查看的扩展名的文件。

I can't write a service to run on the server (off-limits!) so the solution has to be local to the client. Timeliness is not particularly important. I can live with a minute or more delay in notifications. I am watching for Create, Delete, Rename and Changes.

我无法编写在服务器上运行的服务（禁止！），因此解决方案必须是客户端本地的。及时性不是特别重要。我可以忍受一分钟或更长时间的通知延迟。我正在关注创建、删除、重命名和更改。

Would using the .NET System.IO.fileSystemWatcher create much of a load on the server?

使用 .NET System.IO.fileSystemWatcher 会在服务器上产生很大的负载吗？

How about 10 separate watchers to cut down the number of folders/files being watched? (down to 200 from 700 folders, 1200 from 5500 files in total) More network traffic instead of less? My thoughts are a reshuffle on the server to put the watched files under 1 tree. I may not always have this option hence the team of watchers.

10 个独立的观察者如何减少被观察的文件夹/文件的数量？（从 700 个文件夹减少到 200 个，从总共 5500 个文件减少到 1200 个）更多的网络流量而不是更少？我的想法是在服务器上重新洗牌，将观看的文件放在 1 棵树下。我可能并不总是有这个选项，因此观察者团队。

I suppose the other solution is a periodic check if the FSW creates an undue load on the server, or if it doesn't work for a whole bunch of SysAdmin type reasons.

我想另一个解决方案是定期检查 FSW 是否在服务器上产生了过度负载，或者是否由于一大堆 SysAdmin 类型的原因而不起作用。

Is there a better way to do this?

有一个更好的方法吗？

回答by mdb

From a server load point of view, using the IO.FileSystemWatcherfor remote change notifications in the scenario you describe is probably the most efficient method possible. It uses the FindFirstChangeNotificationand ReadDirectoryChangesWWin32 API functions internally, which in turn communicate with the network redirector in an optimized way (assuming standard Windows networking: if a third-party redirector is used, and it doesn't support the required functionality, things won't work at all). The .NET wrapper also uses async I/O and everything, further assuring maximum efficiency.

从服务器负载的角度来看，在您描述的场景中使用IO.FileSystemWatcher进行远程更改通知可能是最有效的方法。它在内部使用FindFirstChangeNotification和ReadDirectoryChangesWWin32 API 函数，这些函数又以优化的方式与网络重定向器通信（假设标准 Windows 网络：如果使用第三方重定向器，并且它不支持所需的功能，事情就赢了根本不起作用）。.NET 包装器还使用异步 I/O 和一切，进一步确保最大效率。

The only problem with this solution is that it's not very reliable. Other than having to deal with network connections going away temporarily (which isn't too much of a problem, since IO.FileSystemWatcher will fire an error event in this case which you can handle), the underlying mechanism has certain fundamental limitations. From the MSDN documentation for the Win32 API functions:

这个解决方案的唯一问题是它不是很可靠。除了必须处理暂时消失的网络连接（这不是什么大问题，因为在这种情况下 IO.FileSystemWatcher 会触发一个您可以处理的错误事件），底层机制有一些基本的限制。来自 Win32 API 函数的 MSDN 文档：

• ReadDirectoryChangesWfails with ERROR_INVALID_PARAMETER when the buffer length is greater than 64 KB and the application is monitoring a directory over the network. This is due to a packet size limitation with the underlying file sharing protocols

• Notifications may not be returned when calling FindFirstChangeNotificationfor a remote file system

• 当缓冲区长度大于 64 KB 并且应用程序正在监视网络上的目录时，ReadDirectoryChangesW失败并显示 ERROR_INVALID_PARAMETER。这是由于底层文件共享协议的数据包大小限制

• 为远程文件系统调用FindFirstChangeNotification时可能不会返回通知

In other words: under high load (when you would need a large buffer) or, worse, under random unspecified circumstances, you may not get the notifications you expect. This is even an issue with local file system watchers, but it's much more of a problem over the network. Another question here on SOdetails the inherent reliability problems with the API in a bit more detail.

换句话说：在高负载下（当您需要大缓冲区时），或者更糟糕的是，在随机未指定的情况下，您可能无法收到您期望的通知。这甚至是本地文件系统观察者的问题，但它在网络上的问题更多。SO 上的另一个问题更详细地详细说明了 API 的固有可靠性问题。

When using file system watchers, your application should be able to deal with these limitations. For example:

使用文件系统观察器时，您的应用程序应该能够处理这些限制。例如：

• If the files you're looking for have sequence numbers, store the last sequence number you got notified about, so you can look for 'gaps' on future notifications and process the files for which you didn't get notified;

• On receiving a notification, always do a full directory scan. This may sound really bad, but since the scan is event-driven, it's still much more efficient than dumb polling. Also, as long as you keep the total number of files in a single directory, as well as the number of directories to scan, under a thousand or so, the impact of this operation on performance should be pretty minimal anyway.

• 如果您要查找的文件有序列号，请存储您收到通知的最后一个序列号，以便您可以在以后的通知中查找“空白”并处理您没有收到通知的文件；

• 收到通知后，请始终进行完整目录扫描。这听起来可能很糟糕，但由于扫描是事件驱动的，它仍然比哑轮询有效得多。此外，只要您将单个目录中的文件总数以及要扫描的目录数保持在 1000 左右以下，无论如何，此操作对性能的影响应该非常小。

Setting up multiple listeners is something you should avoid as much as possible: if anything, this will make things even lessreliable...

设置多个监听器是你应该尽可能避免的事情：如果有的话，这会让事情变得更不可靠......

Anyway, if you absolutely haveto use file system watchers, things can work OK as long as you're aware of the limitations, and don't expect 1:1 notification for every file modified/created.

无论如何，如果您绝对必须使用文件系统观察程序，只要您知道这些限制，事情就可以正常工作，并且不要期望每个修改/创建的文件都会收到 1:1 的通知。

So, if you have other options (essentially, having the process writing the files notify you in a non-file system based way: any regular RPC method will be an improvement...), those are definitely worth looking into from a reliability point of view.

因此，如果您有其他选择（本质上，让写入文件的进程以基于非文件系统的方式通知您：任何常规的 RPC 方法都将是一种改进......），从可靠性角度来看，这些绝对值得研究看法。

回答by Nick Randell

I've used the file system watchers from C# a number of times. The first time I used them, I had problems with them stopping working, mainly due to the fact that I was processing the changes in the thread that reported the change.

我已经多次使用 C# 中的文件系统观察程序。我第一次使用它们时，我遇到了它们停止工作的问题，主要是因为我正在处理报告更改的线程中的更改。

Now however, I just push the change onto a queue and process the queue on another thread. This seems to solve the problem I originally had. For your problem, you could have multiple watchers pushing onto the same queue.

但是，现在我只是将更改推送到队列并在另一个线程上处理队列。这似乎解决了我最初遇到的问题。对于您的问题，您可以将多个观察者推入同一个队列。

However, I haven't used this with your sort of scale of problem.

但是，我还没有将它用于您的这种规模的问题。

回答by Treb

In my experience, an FSW does not create high network traffic. However, if there is a performance problem, your approach of using several watchers and breaking it down to fewer folders being watched sounds reasonable.

根据我的经验，FSW 不会产生高网络流量。但是，如果存在性能问题，您使用多个观察者并将其分解为被观察的更少文件夹的方法听起来是合理的。

I had some big problems with FSW on network drives, though: Deleting a file always threw the error event, never the deleted event. I did not find a solution, so I now avoid using FSW if there is a way around it...

不过，我在网络驱动器上使用 FSW 时遇到了一些大问题：删除文件总是引发错误事件，而不是已删除事件。我没有找到解决方案，所以我现在避免使用 FSW，如果有办法解决的话......

回答by Jim Burger

The MSDN documentation indicatesthat you can use the FileSystemWatcher component to watch for filesystem changes on a network drive.

在MSDN文档指出，你可以使用FileSystemWatcher组件来监视网络上的文件系统变化的驱动。

It also indicates that the watcher component listens for file system change notifications rather than periodically interrogating the target drive for changes.

它还表明观察器组件侦听文件系统更改通知，而不是定期询问目标驱动器的更改。

Based on that the amount of network traffic depends entirely on how much you expect the contents of that network drive to change. The FSW component will not add to the level of network traffic.

基于此，网络流量完全取决于您期望该网络驱动器的内容更改多少。FSW 组件不会增加网络流量级别。

回答by davidWazy

Watcher looks 100% reliable - just watch the buffer size on the watcher object. I've tested thousands of file-updates, none lost.

Watcher 看起来 100% 可靠 - 只需观察 watcher 对象上的缓冲区大小。我已经测试了数千个文件更新，没有一个丢失。

I recommend using a multi-threaded approach - The trigger being the file watcher. It can launch a thread for each file-change detected. The watcher can process much faster with less chance of overflow. (use Async thread)

我建议使用多线程方法 - 触发器是文件观察器。它可以为检测到的每个文件更改启动一个线程。观察者可以处理得更快，溢出的机会更少。（使用异步线程）

回答by trevorwong77

After using System.IO.FileSystemWatcher for sometime. It is not stable enough to handle events that is coming in too fast. To ensure 100% read of the files. I use simple directory methods to search through the files. After reading it, immediately copy the files to another folder. To isolate it from new files being added while you are reading the files.

使用 System.IO.FileSystemWatcher 一段时间后。它不够稳定，无法处理来得太快的事件。确保 100% 读取文件。我使用简单的目录方法来搜索文件。阅读后，立即将文件复制到另一个文件夹中。在您阅读文件时将其与添加的新文件隔离开来。

Timer is used to regularly reading the folder. By copying the already read file to archive folder, this make sure it will not be read again. The subsequent read will be always new files.

定时器用于定期读取文件夹。通过将已读取的文件复制到存档文件夹，确保不会再次读取它。后续读取将始终是新文件。

var fileNames = Directory.GetFiles(srcFolder);
foreach (string fileName in fileNames)
{
   string[] lines = File.ReadAllLines(fileName);
}

回答by core

I wouldn't think there's any sort of active state or communication between the computer with the FSW and the computer whose location is being monitored. In other words, the FSW isn't pinging the networked OS to check on the file.

我不认为有 FSW 的计算机和位置被监控的计算机之间存在任何活动状态或通信。换句话说，FSW 不会通过 ping 网络操作系统来检查文件。

One would imagine that a message or event is onlyraised/sent to the networked FSW when a change occurs.

可以想象，只有在发生更改时才会向联网的 FSW 引发/发送消息或事件。

But this is all just speculation. :)

但这一切都只是猜测。:)