在使用文件时截断文件 (Linux)

Truncating a file while it's being used (Linux)

提问by Hobo

I have a process that's writing a lot of data to stdout, which I'm redirecting to a log file. I'd like to limit the size of the file by occasionally copying the current file to a new name and truncating it.

我有一个进程将大量数据写入标准输出，我将其重定向到日志文件。我想通过偶尔将当前文件复制到新名称并截断它来限制文件的大小。

My usual techniques of truncating a file, like

我常用的截断文件的技术，比如

cp /dev/null file

don't work, presumably because the process is using it.

不工作，大概是因为进程正在使用它。

Is there some way I can truncate the file? Or delete it and somehow associate the process' stdout with a new file?

有什么方法可以截断文件吗？或者删除它并以某种方式将进程的标准输出与新文件相关联？

FWIW, it's a third party product that I can't modify to change its logging model.

FWIW，它是第三方产品，我无法修改以更改其日志记录模型。

EDITredirecting over the file seems to have the same issue as the copy above - the file returns to its previous size next time it's written to:

编辑重定向文件似乎与上面的副本存在相同的问题 - 下次写入文件时，文件将恢复到以前的大小：

ls -l sample.log ; echo > sample.log ; ls -l sample.log ; sleep 10 ; ls -l sample.log
-rw-rw-r-- 1 user group 1291999 Jun 11  2009 sample.log
-rw-rw-r-- 1 user group 1 Jun 11  2009 sample.log
-rw-rw-r-- 1 user group 1292311 Jun 11  2009 sample.log

采纳答案by Michiel Buddingh

Take a look at the utility split(1), part of GNU Coreutils.

看一看实用程序split(1)，它是 GNU Coreutils 的一部分。

回答by Robert Munteanu

Try > file.

试试> file。

---

---

Update regarding the comments: it works nicely for me:

关于评论的更新：它对我来说很好用：

robert@rm:~> echo "content" > test-file
robert@rm:~> cat test-file 
content
robert@rm:~> > test-file
robert@rm:~> cat test-file 

回答by Billy

as the file is being used, if you try to nullify it or something like that, sometimes it might "confuse" the app that's writing into the log file and it might not log anything after that.

当文件正在被使用时，如果您尝试将其取消或类似的东西，有时它可能会“混淆”写入日志文件的应用程序，之后它可能不会记录任何内容。

What I'd try ot do is to set up a kind of proxy/filter for that log, instead of redirecting to file, redirect to a process or something that would get input and write to a rolling file.

我会尝试做的是为该日志设置一种代理/过滤器，而不是重定向到文件，重定向到进程或可以获取输入并写入滚动文件的内容。

Maybe it can be done by script otherwise you could write a simple app for that ( java or something else ). The impact on app performance should be quite small, but you'll have to run some tests.

也许它可以通过脚本完成，否则您可以为此编写一个简单的应用程序（java 或其他东西）。对应用程序性能的影响应该很小，但您必须运行一些测试。

Btw, your app, is it a stand-alone, web app, ... ? Maybe there are other options to be investigated.

顺便说一句，你的应用程序是一个独立的网络应用程序，......？也许还有其他选择需要研究。

Edit: there's also an Append Redirection Operator >>that I've personally never used, but it might not lock the file.

编辑：还有一个Append Redirection Operator >>我个人从未使用过，但它可能不会锁定文件。

回答by Vex

Did you check the behavior of any signals like SIGHUP to the third party product, to see if it will start logging a fresh file? You would move the old file to a permanent name, first.

您是否检查了第三方产品的任何信号（如 SIGHUP）的行为，以查看它是否会开始记录新文件？首先，您将旧文件移动到一个永久名称。

kill -HUP [process-id]

kill -HUP [进程 ID]

And then it would start writing out again.

然后它会再次开始写出来。

Alternatively (as Billy suggested) maybe redirecting the output from the application to a logging program like multilog or the one that is commonly used with Apache, known as cronolog. Then you'll have more fine grained control of where everything goes before it is written to that initial file descriptor (file), which is really all it is.

或者（如 Billy 建议的那样）可能将应用程序的输出重定向到一个日志记录程序，如 multilog 或通常与 Apache 一起使用的程序，称为 cronolog。然后，在将所有内容写入初始文件描述符（文件）之前，您将对其进行更精细的控制，这实际上就是全部内容。

回答by Arthur Ulfeldt

In Linux (actually all unicies) files are created when they are opened and deleted when nothing holds a reference to them. In this case the program that opened it and the directory it was opened 'in' hold references to the file. When the cp program wants to write to the file it gets a reference to it from the directory, writes a length of zero into the metadata stored in the directory (this is a slight simplification) and gives up the handle. Then the original program, still holding the original file handle, writes some more data to the file and saves what it thinks the length should be.

在 Linux（实际上是所有 unicies）中，文件在打开时创建，在没有任何引用时删除。在这种情况下，打开它的程序和打开它的目录包含对文件的引用。当 cp 程序想要写入文件时，它会从目录中获取对该文件的引用，将长度为零的内容写入存储在目录中的元数据（这是一个轻微的简化）并放弃句柄。然后，仍然持有原始文件句柄的原始程序将更多数据写入文件并保存它认为应该是的长度。

even if you where to delete the file from the directory the program would continue to write data to it (and use up disc space) even though no other program would have any way of referencing it.

即使您从目录中删除文件，程序也会继续向其中写入数据（并占用磁盘空间），即使没有其他程序可以引用它。

in short once the program has a reference (handle) to a file nothing you do is going to change that.

简而言之，一旦程序具有对文件的引用（句柄），您所做的任何事情都不会改变它。

there are in theory ways of modifying the programs behavior by setting LD_LIBRARY_PATH to include a program that intercepts all the file access system calls. I recall seeing something like this somewhere though cant recall the name.

理论上，通过设置 LD_LIBRARY_PATH 来包含一个拦截所有文件访问系统调用的程序来修改程序行为。我记得在某处看到过类似的东西，虽然记不起名字。

回答by Jonathan Leffler

The interesting thing about those regrown files is that the first 128 KB or so will be all zeroes after you truncate the file by copying /dev/nullover it. This happens because the file is truncated to zero length, but the file descriptor in the application still points immediately after its last write. When it writes again, the file system treats the start of the file as all zero bytes - without actually writing the zeroes to disk.

这些重新生成的文件的有趣之处在于，在您通过复制/dev/null来截断文件后，前 128 KB 左右将全部为零。发生这种情况是因为文件被截断为零长度，但应用程序中的文件描述符在其最后一次写入后仍然立即指向。当它再次写入时，文件系统将文件的开头视为全零字节 - 而不实际将零写入磁盘。

Ideally, you should ask the vendor of the application to open the log file with the O_APPENDflag. This means that after you truncate the file, the next write will implicitly seek to the end of the file (meaning back to offset zero) and then write the new information.

理想情况下，您应该要求应用程序的供应商打开带有O_APPEND标志的日志文件。这意味着在您截断文件后，下一次写入将隐式地寻找到文​​件的末尾（意味着回到偏移量零），然后写入新信息。

---

---

This code rigs standard output so it is in O_APPENDmode and then invokes the command given by its arguments (rather like niceruns a command after adjusting its nice-level, or nohupruns a command after fixing things so it ignores SIGHUP).

此代码绑定标准输出，使其处于O_APPEND模式下，然后调用由其参数给出的命令（更像nice是在调整其 nice-levelnohup后运行命令，或在修复问题后运行命令以便忽略 SIGHUP）。

#include <stdlib.h>
#include <unistd.h>
#include <fcntl.h>
#include <stdio.h>
#include <stdarg.h>
#include <string.h>
#include <errno.h>

static char *arg0 = "<unknown>";

static void error(const char *fmt, ...)
{
    va_list args;
    int errnum = errno;
    fprintf(stderr, "%s: ", arg0);
    va_start(args, fmt);
    vfprintf(stderr, fmt, args);
    va_end(args);
    if (errnum != 0)
        fprintf(stderr, " (%d: %s)", errnum, strerror(errnum));
    putc('\n', stderr);
    fflush(0);
    exit(1);
}

int main(int argc, char **argv)
{
    int attr;
    arg0 = argv[0];

    if (argc < 2)
        error("Usage: %s cmd [arg ...]", arg0);
    if ((attr = fcntl(1, F_GETFL, &attr)) < 0)
        error("fcntl(F_GETFL) failed");
    attr |= O_APPEND;
    if (fcntl(1, F_SETFL, attr) != 0)
        error("fcntl(F_SETFL) failed");
    execvp(argv[1], &argv[1]);
    error("failed to exec %s", argv[1]);
    return(1);
}

My testing of it was somewhat casual, but just barely enough to persuade me that it worked.

我对它的测试有点随意，但几乎不足以说服我它有效。

---

---

Simpler alternative

更简单的选择

Billynotes in his answerthat '>>' is the append operator - and indeed, on Solaris 10, bash (version 3.00.16(1)) does use the O_APPENDflag - thereby making the code above unnecessary, as shown ('Black JL:' is my prompt on this machine):

比利在他的回答中指出，“ >>”是附加运算符——实际上，在 Solaris 10 上，bash（版本 3.00.16(1)）确实使用了该O_APPEND标志——从而使上面的代码变得不必要，如图所示（'Black JL:'是我在这台机器上的提示）：

Black JL: truss -o bash.truss bash -c "echo Hi >> x3.29"
Black JL: grep open bash.truss
open("/var/ld/ld.config", O_RDONLY)             Err#2 ENOENT
open("/usr/lib/libcurses.so.1", O_RDONLY)       = 3
open("/usr/lib/libsocket.so.1", O_RDONLY)       = 3
open("/usr/lib/libnsl.so.1", O_RDONLY)          = 3
open("/usr/lib/libdl.so.1", O_RDONLY)           = 3
open("/usr/lib/libc.so.1", O_RDONLY)            = 3
open("/platform/SUNW,Ultra-4/lib/libc_psr.so.1", O_RDONLY) = 3
open64("/dev/tty", O_RDWR|O_NONBLOCK)           = 3
stat64("/usr/openssl/v0.9.8e/bin/bash", 0xFFBFF2A8) Err#2 ENOENT
open64("x3.29", O_WRONLY|O_APPEND|O_CREAT, 0666) = 3
Black JL:

Use append redirection rather than the wrapper ('cantrip') code above. This just goes to show that when you use one particular technique for other (valid) purposes, adapting it to yet another is not necessarily the simplest mechanism - even though it works.

使用附加重定向而不是上面的包装器 (' cantrip') 代码。这只是表明，当您将一种特定技术用于其他（有效）目的时，将其应用于另一种不一定是最简单的机制 - 即使它有效。

回答by Arthur Ulfeldt

instead of redirecting it to a file you could pipe it to a program that automatically rotates the file by closing it, moving it and opening a new one every time it gets too big.

与其将它重定向到一个文件，您还可以将它传送到一个程序，该程序通过关闭文件、移动文件并在每次文件太大时打开一个新文件来自动旋转文件。

回答by Peter Eisentraut

As of coreutils 7.0, there is a truncatecommand.

从 coreutils 7.0 开始，有一个truncate命令。

回答by Larry Irwin

I downloaded and compiled the latest coreutilsso I could have truncateavailable.

我下载并编译了最新的，coreutils所以我可以truncate使用。

Ran ./configureand make, but did not run make install.

然./configure而make，但没跑make install。

All the compiled utilities appear in the "src" folder.

所有编译的实用程序都出现在“src”文件夹中。

I ran

我跑了

[path]/src/truncate -s 1024000 textfileineedtotruncate.log

[path]/src/truncate -s 1024000 textfileineedtotruncate.log

on a 1.7 GB log file.

在 1.7 GB 的日志文件上。

It did not change the size listed when using ls -l, but it did free up all the disk space - which is what I really needed to do before /varfilled up and killed the process.

它在使用时没有更改列出的大小ls -l，但它确实释放了所有磁盘空间 - 这是我在/var填满并终止进程之前真正需要做的事情。

Thanks for the tip on "truncate"!

感谢有关“截断”的提示！

回答by Yogesh

@Hobo use freopen(), it reuses stream to either open the file specified by filename or to change its access mode. If a new filename is specified, the function first attempts to close any file already associated with stream (third parameter) and disassociates it. Then, independently of whether that stream was successfully closed or not, freopen opens the file specified by filename and associates it with the stream just as fopen would do using the specified mode.

@Hobo 使用freopen()，它重用流来打开文件名指定的文件或更改其访问模式。如果指定了新文件名，则该函数首先尝试关闭已与流（第三个参数）关联的任何文件并将其解除关联。然后，不管该流是否成功关闭，freopen 打开由 filename 指定的文件并将其与流相关联，就像 fopen 使用指定模式所做的那样。

if a thirdparty binary is generating logs we need to write a wrapper which will rotate the logs, and thirdparty will run in proxyrun thread as below.

如果第三方二进制文件正在生成日志，我们需要编写一个包装器来轮换日志，第三方将在 proxyrun 线程中运行，如下所示。

#include <pthread.h>
#include <stdio.h>
#include <unistd.h>
#include <unistd.h>
#include <string.h>

using namespace std;

extern "C" void * proxyrun(void * pArg){
   static int lsiLineNum = 0;
   while(1) 
   {
     printf("\nLOGGER: %d",++lsiLineNum);
     fflush(stdout);
   }
  return NULL;
}


int main(int argc, char **argv)
{
  pthread_t lThdId;
  if(0 != pthread_create(&lThdId, NULL, proxyrun, NULL))
  {
    return 1;
  }

  char lpcFileName[256] = {0,};

  static int x = 0;

  while(1)
  {
    printf("\n<<<MAIN SLEEP>>>");
    fflush(stdout);
    sprintf(lpcFileName, "/home/yogesh/C++TestPrograms/std.txt%d",++x);
    freopen(lpcFileName,"w",stdout);
    sleep(10);
  }

  return 0;
}