在 Java 中真正强制文件同步/刷新
声明：本页面是StackOverFlow热门问题的中英对照翻译，遵循CC BY-SA 4.0协议，如果您需要使用它，必须同样遵循CC BY-SA许可，注明原文地址和作者信息，同时你必须将它归于原作者（不是我）：StackOverFlow
原文地址: http://stackoverflow.com/questions/730521/
Warning: these are provided under cc-by-sa 4.0 license. You are free to use/share it, But you must attribute it to the original authors (not me):
StackOverFlow
Really force file sync/flush in Java
提问by dmeister
How can data written to a file reallybe flushed/synced with the block device by Java.
Java如何真正将写入文件的数据与块设备刷新/同步。
I tried this code with NIO:
我用 NIO 试过这个代码：
FileOutputStream s = new FileOutputStream(filename)
Channel c = s.getChannel()
while(xyz)
c.write(buffer)
c.force(true)
s.getFD().sync()
c.close()
I supposed that c.force(true) togehter with s.getFD().sync() should be sufficient because the doc for forcestates
我认为 c.force(true) 与 s.getFD().sync() 一起应该就足够了，因为力状态的文档
Forces any updates to this channel's file to be written to the storage device that contains it. If this channel's file resides on a local storage device then when this method returns it is guaranteed that all changes made to the file since this channel was created, or since this method was last invoked, will have been written to that device. This is useful for ensuring that critical information is not lost in the event of a system crash.
强制将此频道文件的任何更新写入包含它的存储设备。如果此通道的文件驻留在本地存储设备上，则当此方法返回时，可以保证自创建此通道或自上次调用此方法以来对文件所做的所有更改都将写入该设备。这对于确保在系统崩溃时不会丢失关键信息非常有用。
The documentation to syncstates:
同步文档指出：
Force all system buffers to synchronize with the underlying device. This method returns after all modified data and attributes of this FileDescriptor have been written to the relevant device(s). In particular, if this FileDescriptor refers to a physical storage medium, such as a file in a file system, sync will not return until all in-memory modified copies of buffers associated with this FileDesecriptor have been written to the physical medium. sync is meant to be used by code that requires physical storage (such as a file) to be in a known state.
强制所有系统缓冲区与底层设备同步。此方法在此 FileDescriptor 的所有修改数据和属性已写入相关设备后返回。特别是，如果这个 FileDescriptor 指的是一个物理存储介质，比如文件系统中的一个文件，同步将不会返回，直到与这个 FileDescriptor 关联的缓冲区的所有内存中修改副本都被写入物理介质。sync 旨在供需要物理存储（例如文件）处于已知状态的代码使用。
These two calls should be sufficient. Is it? I guess they aren't.
这两个调用应该足够了。是吗？我猜他们不是。
Background: I do a small performance comparison (2 GB, sequential write) using C/Java and the Java version is twice as fast as the C version and probably faster than the hardware (120 MB/s on a single HD). I also tried to execute the command line tool sync with Runtime.getRuntime().exec("sync") but that hasn't changed the behavior.
背景：我使用 C/Java 进行了一个小的性能比较（2 GB，顺序写入），Java 版本的速度是 C 版本的两倍，并且可能比硬件更快（单个 HD 上为 120 MB/s）。我还尝试使用 Runtime.getRuntime().exec("sync") 执行命令行工具同步，但这并没有改变行为。
The C code resulting in 70 MB/s is (using the low level APIs (open,write,close) doesn't change much):
产生 70 MB/s 的 C 代码是（使用低级 API（打开、写入、关闭）没有太大变化）：
FILE* fp = fopen(filename, "w");
while(xyz) {
fwrite(buffer, 1, BLOCK_SIZE, fp);
}
fflush(fp);
fclose(fp);
sync();
Without the final call to sync; I got unrealistical values (over 1 GB aka main memory performance).
没有最后的同步调用；我得到了不切实际的值（超过 1 GB 又名主内存性能）。
Why is there such a big difference between C and Java? There are two possiblities: I doesn't sync the data correctly in Java or the C code is suboptimal for some reason.
为什么C和Java有这么大的区别？有两种可能性：我没有在 Java 中正确同步数据，或者 C 代码由于某种原因不是最理想的。
Update: I have done strace runs with "strace -cfT cmd". Here are the results:
更新：我已经使用“strace -cfT cmd”完成了 strace 运行。结果如下：
C (Low-Level API): MB/s 67.389782
C（低级 API）：MB/s 67.389782
% time seconds usecs/call calls errors syscall ------ ----------- ----------- --------- --------- ---------------- 87.21 0.200012 200012 1 fdatasync 11.05 0.025345 1 32772 write 1.74 0.004000 4000 1 sync
C (High-Level API): MB/s 61.796458
C（高级 API）：MB/s 61.796458
% time seconds usecs/call calls errors syscall ------ ----------- ----------- --------- --------- ---------------- 73.19 0.144009 144009 1 sync 26.81 0.052739 1 65539 write
Java (1.6 SUN JRE, java.io API): MB/s 128.6755466197537
Java（1.6 SUN JRE，java.io API）：MB/s 128.6755466197537
% time seconds usecs/call calls errors syscall ------ ----------- ----------- --------- --------- ---------------- 80.07 105.387609 3215 32776 write 2.58 3.390060 3201 1059 read 0.62 0.815251 815251 1 fsync
Java (1.6 SUN JRE, java.nio API): MB/s 127.45830221558376
Java（1.6 SUN JRE，java.nio API）：MB/s 127.45830221558376
5.52 0.980061 490031 2 fsync 1.60 0.284752 9 32774 write 0.00 0.000000 0 80 close
The time values seem to be system time only and are therefore pretty meaningless.
时间值似乎只是系统时间，因此毫无意义。
Update 2: I switched to another server, rebooted, and I use a fresh formatted ext3. Now I get only 4% differences between Java and C. I simply don't know what went wrong. Sometimes things are strange. I should have tried the measurement with another system before writing this question. Sorry.
更新 2：我切换到另一台服务器，重新启动，并使用新格式化的 ext3。现在我得到的 Java 和 C 之间只有 4% 的差异。我只是不知道出了什么问题。有时事情很奇怪。在写这个问题之前，我应该用另一个系统尝试测量。对不起。
Update 3: To summarize the answers:
更新 3：总结答案：
- Use c.force(true) followed by s.getFD().sync() for Java NIO and s.flush() and s.getFD().sync() for Java's stream API. For the High-Level API in C don't forget to sync. A fflush submitted the data to the OS, but doesn't bring your data to the block device.
- Use strace to analyze the syscalls done by a command
- Cross check your results before posting a question.
- 使用 c.force(true) 后跟 s.getFD().sync() 用于 Java NIO 和 s.flush() 和 s.getFD().sync() 用于 Java 的流 API。对于 C 中的高级 API，不要忘记同步。fflush 将数据提交给操作系统，但不会将您的数据带到块设备。
- 使用 strace 分析命令完成的系统调用
- 在发布问题之前交叉检查您的结果。
Update 4: Please note the following follow-up question.
更新 4：请注意以下后续问题。
采纳答案by Charlie Martin
You need to tell us more about the hardware and operating system, also the specific Java version. How are you measuring this throughput?
您需要告诉我们更多有关硬件和操作系统的信息，以及具体的 Java 版本。你如何衡量这个吞吐量？
You're correct that force/sync should force the data out to the physical media.
您是正确的，强制/同步应该将数据强制输出到物理媒体。
Here's a raw version of copy. Compiled with gcc 4.0 on an Intel Mac, should be clean.
这是副本的原始版本。在 Intel Mac 上用 gcc 4.0 编译，应该是干净的。
/* rawcopy -- pure C, system calls only, copy argv[1] to argv[2] */
/* This is a test program which simply copies from file to file using
* only system calls (section 2 of the manual.)
*
* Compile:
*
* gcc -Wall -DBUFSIZ=1024 -o rawcopy rawcopy.c
*
* If DIRTY is defined, then errors are interpreted with perror(3).
* This is ifdef'd so that the CLEAN version is free of stdio. For
* convenience I'm using BUFSIZ from stdio.h; to compile CLEAN just
* use the value from your stdio.h in place of 1024 above.
*
* Compile DIRTY:
*
* gcc -DDIRTY -Wall -o rawcopy rawcopy.c
*
*/
#include <fcntl.h>
#include <sys/types.h>
#include <sys/uio.h>
#include <stdlib.h>
#include <unistd.h>
#if defined(DIRTY)
# if defined(BUFSIZ)
# error "Don't define your own BUFSIZ when DIRTY"
# endif
# include <stdio.h>
# define PERROR perror(argv[0])
#else
# define CLEAN
# define PERROR
# if ! defined(BUFSIZ)
# error "You must define your own BUFSIZ with -DBUFSIZ=<number>"
# endif
#endif
char * buffer[BUFSIZ]; /* by definition stdio BUFSIZ should
be optimal size for read/write */
extern int errno ; /* I/O errors */
int main(int argc, char * argv[]) {
int fdi, fdo ; /* Input/output file descriptors */
ssize_t len ; /* length to read/write */
if(argc != 3){
PERROR;
exit(errno);
}
/* Open the files, returning perror errno as the exit value if fails. */
if((fdi = open(argv[1],O_RDONLY)) == -1){
PERROR;
exit(errno);
}
if((fdo = open(argv[2], O_WRONLY|O_CREAT)) == -1){
PERROR;
exit(errno);
}
/* copy BUFSIZ bytes (or total read on last block) fast as you
can. */
while((len = read(fdi, (void *) buffer, BUFSIZ)) > -1){
if(len == -1){
PERROR;
exit(errno);
}
if(write(fdo, (void*)buffer, len) == -1){
PERROR;
exit(errno);
}
}
/* close and fsync the files */
if(fsync(fdo) ==-1){
PERROR;
exit(errno);
}
if(close(fdo) == -1){
PERROR;
exit(errno);
}
if(close(fdi) == -1){
PERROR;
exit(errno);
}
/* if it survived to here, all worked. */
exit(0);
}
回答by Ingo
The C code could be suboptimal, because it uses stdio rather than raw OS write(). But then, java could be more optimal because it allocates larger buffers?
C 代码可能不是最理想的，因为它使用 stdio 而不是原始操作系统 write()。但是，java 可能更优化，因为它分配了更大的缓冲区？
Anyway, you can only trust the APIDOC. The rest is beyond your duties.
无论如何，您只能信任 APIDOC。剩下的就超出你的职责范围了。
回答by araqnid
Actually, in C you want to just call fsync()on the one file descriptor, not sync()(or the "sync" command) which signals the kernel to flushall buffers to disk system-wide.
实际上，在 C 中，您只想调用fsync()一个文件描述符，而不是sync()（或“同步”命令）向内核发送信号flush到磁盘系统范围内的所有缓冲区。
If you strace(getting Linux-specific here) the JVM you should be able to observe an fsync()or fdatasync()system call being made on your output file. That would be what I'd expect the getFD().sync()call to do. I assume c.force(true)simply flags to NIO that fsync()should be called after each write. It might simply be that the JVM you're using doesn't actually implement the sync()call?
如果您strace（在此处获得特定于 Linux 的）JVM，您应该能够观察到对输出文件进行的系统调用fsync()或fdatasync()系统调用。那将是我所期望的getFD(). sync()打电话去做。我假设c.force(true)简单地标记到 NIOfsync()应该在每次写入后调用。可能只是您使用的 JVM 并没有真正实现sync()调用？
I'm not sure why you weren't seeing any difference when calling "sync" as a command: but obviously, after the first sync invocation, subsequent ones are usually quite a lot faster. Again, I'd be inclined to break out strace(truss on Solaris) as a "what's actually happening here?" tool.
我不确定为什么在将“sync”作为命令调用时没有看到任何区别：但显然，在第一次同步调用之后，后续调用通常要快得多。再一次，我倾向于将strace（在 Solaris 上的桁架）作为“这里实际发生了什么？” 工具。
回答by Adam Fanello
(I know this is a very late reply, but I ran into this thread doing a Google search, and that's probably how you ended up here too.)
（我知道这是一个很晚的回复，但我在谷歌搜索时遇到了这个线程，这可能也是你最终来到这里的方式。）
Your calling sync() in Java on a single file descriptor, so only that buffers related to that one file get flushed out to disk.
您在 Java 中对单个文件描述符调用 sync()，因此只有与该文件相关的缓冲区才会刷新到磁盘。
In C and command-line, you're calling sync() on the entire operating system - so every file buffer gets flushed out to disk, for everything your O/S is doing.
在 C 和命令行中，您在整个操作系统上调用 sync() - 因此每个文件缓冲区都会被刷新到磁盘，用于您的 O/S 所做的一切。
To be comparable, the C call should be to syncfs(fp);
为了比较，C 调用应该是 syncfs(fp);
From the Linux man page:
从 Linux 手册页：
sync() causes all buffered modifications to file metadata and data to
be written to the underlying file systems.
syncfs() is like sync(), but synchronizes just the file system contain‐
ing file referred to by the open file descriptor fd.
回答by eckes
It is a good idea to use the synchronized I/O data integrity completion. However your C sample is using the wrong method. You use sync(), which is used to sync the whole OS.
使用同步 I/O 数据完整性完成是一个好主意。但是，您的 C 示例使用了错误的方法。您使用sync()，它用于同步整个操作系统。
If you want to write the blocks of that single file to disk, you need to use fsync(2)or fdatasync(2)in C. BTW: when you use buffered stdio in C (or a BufferedOutputStream or some Writer in Java) you need to flush both first before you sync.
如果要将单个文件的块写入磁盘，则需要在 C 中使用fsync(2)或fdatasync(2).
The fdatasync()variant is a bit more efficient if the file has not changed name or size since you sync. But it might also not persit all the meta data. If you want to write your own transactional safe database systems, you need to observe some more stuff (like fsyncing the parent directory).
fdatasync()如果文件在同步后没有更改名称或大小，则该变体会更有效一些。但它也可能不会保留所有元数据。如果你想编写自己的事务安全数据库系统，你需要观察更多的东西（比如 fsyncing 父目录）。
