bash 即时输出重定向，在程序仍在运行时查看文件重定向输出

on-the-fly output redirection, seeing the file redirection output while the program is still running

提问by grokkaine

If I use a command like this one:
./program >> a.txt &
, and the program is a long running one then I can only see the output once the program ended. That means I have no way of knowing if the computation is going well until it actually stops computing. I want to be able to read the redirected output on file while the program is running.

如果我使用这样的命令：
./program >> a.txt &
，并且程序是一个长时间运行的程序，那么我只能在程序结束后看到输出。这意味着在它真正停止计算之前，我无法知道计算是否进行得很好。我希望能够在程序运行时读取文件上的重定向输出。

This is similar to opening a file, appending to it, then closing it back after every writing. If the file is only closed at the end of the program then no data can be read on it until the program ends. The only redirection I know is similar to closing the file at the end of the program.

这类似于打开一个文件，附加到它，然后在每次写入后将其关闭。如果文件仅在程序结束时关闭，则在程序结束之前无法读取其上的数据。我知道的唯一重定向类似于在程序结束时关闭文件。

You can test it with this little python script. The language doesn't matter. Any program that writes to standard output has the same problem.

你可以用这个小的 python 脚本来测试它。语言无所谓。任何写入标准输出的程序都有同样的问题。

l = range(0,100000)
for i in l:
  if i%1000==0:
    print i
  for j in l:
    s = i + j

One can run this with:
./python program.py >> a.txt &
Then cat a.txt .. you will only get results once the script is done computing.

可以使用以下
命令运行：./python program.py >> a.txt &
然后 cat a.txt .. 只有在脚本完成计算后才能获得结果。

采纳答案by thkala

From the stdoutmanual page:

从stdout手册页：

The stream stderr is unbuffered. The stream stdout is line-buffered when it points to a terminal. Partial lines will not appear until fflush(3) or exit(3) is called, or a new‐line is printed.

流 stderr 是无缓冲的。当流 stdout指向终端时，它是行缓冲的 。在调用 fflush(3) 或 exit(3) 或打印新行之前，不会出现部分行。

Bottom line: Unless the output is a terminal, your program will have its standard output in fully buffered mode by default. This essentially means that it will output data in large-ish blocks, rather than line-by-line, let alone character-by-character.

底线：除非输出是终端，否则默认情况下您的程序的标准输出将处于全缓冲模式。这实质上意味着它将以大块的形式输出数据，而不是逐行输出，更不用说逐字符输出了。

Ways to work around this:

解决这个问题的方法：

• Fix your program: If you need real-time output, you need to fix your program. In C you can use fflush(stdout)after each output statement, or setvbuf()to change the buffering mode of the standard output. For Python there is sys.stdout.flush()of even some of the suggestions here.

• Use a utility that can record from a PTY, rather than outright stdout redirections. GNU Screencan do this for you:

  screen -d -m -L python test.py
  

  would be a start. This will log the output of your program to a file called screenlog.0(or similar) in your current directory with a default delay of 10 seconds, and you can use screento connect to the session where your command is running to provide input or terminate it. The delay and the name of the logfile can be changed in a configuration file or manually once you connect to the background session.

• 修复您的程序：如果您需要实时输出，则需要修复您的程序。在 C 中，您可以fflush(stdout)在每个输出语句之后使用，或者setvbuf()更改标准输出的缓冲模式。对于Python有sys.stdout.flush()，甚至一些建议在这里。

• 使用可以从 PTY 进行记录的实用程序，而不是直接的 stdout 重定向。GNU Screen可以为您做到这一点：

  screen -d -m -L python test.py
  

  将是一个开始。这会将程序的输出记录到screenlog.0当前目录中名为（或类似）的文件中，默认延迟为 10 秒，您可以使用screen它连接到运行命令的会话以提供输入或终止它。延迟和日志文件的名称可以在配置文件中更改，也可以在您连接到后台会话后手动更改。

EDIT:

编辑：

On most Linux system there is a third workaround: You can use the LD_PRELOADvariable and a preloaded library to override select functions of the C library and use them to set the stdoutbuffering mode when those functions are called by your program. This method may work, but it has a number of disadvantages:

在大多数 Linux 系统上，有第三种解决方法：您可以使用LD_PRELOAD变量和预加载的库来覆盖 C 库的选择函数，并stdout在程序调用这些函数时使用它们来设置缓冲模式。这种方法可能有效，但它有许多缺点：

• It won't work at all on static executables

• It's fragile and rather ugly.

• It won't work at all with SUID executables - the dynamic loader will refuse to read the LD_PRELOADvariable when loading such executables for security reasons.

• It's fragile and rather ugly.

• It requires that you find and override a library function that is called by your program afterit initially sets the stdoutbuffering mode and preferably beforeany output. getenv()is a good choice for many programs, but not all. You may have to override common I/O functions such as printf()or fwrite()- if push comes to shove you may just have to override all functions that control the buffering mode and introduce a special condition for stdout.

• It's fragile and rather ugly.

• It's hard to ensure that there are no unwelcome side-effects. To do this right you'd have to ensure that only stdoutis affected and that your overrides will not crash the rest of the program if e.g. stdoutis closed.

• Did I mention that it's fragile and rather ugly?

• 它在静态可执行文件上根本不起作用

• 它很脆弱，而且相当丑陋。

• 它根本不适用于 SUID 可执行文件 -LD_PRELOAD出于安全原因，动态加载程序在加载此类可执行文件时将拒绝读取变量。

• 它很脆弱，而且相当丑陋。

• 它需要你找到并重写的库函数是由你的程序调用后，它最初设置stdout缓冲模式，最好之前任何输出。getenv()是许多程序的不错选择，但不是全部。您可能必须覆盖常见的 I/O 功能，例如printf()或fwrite()- 如果推送紧急，您可能只需要覆盖控制缓冲模式的所有功能并为stdout.

• 它很脆弱，而且相当丑陋。

• 很难确保没有不受欢迎的副作用。要做到这一点，您必须确保仅stdout受影响并且您的覆盖不会使程序的其余部分崩溃，例如stdout关闭。

• 我有没有提到它很脆弱而且相当丑陋？

That said, the process it relatively simple. You put in a C file, e.g. linebufferedstdout.cthe replacement functions:

也就是说，过程相对简单。您放入一个 C 文件，例如linebufferedstdout.c替换函数：

#define _GNU_SOURCE
#include <stdlib.h>
#include <stdio.h>
#include <dlfcn.h>


char *getenv(const char *s) {
    static char *(*getenv_real)(const char *s) = NULL;

    if (getenv_real == NULL) {
        getenv_real = dlsym(RTLD_NEXT, "getenv");

        setlinebuf(stdout);
    }

    return getenv_real(s);
}

Then you compile that file as a shared object:

然后将该文件编译为共享对象：

gcc -O2 -o linebufferedstdout.so -fpic -shared linebufferedstdout.c -ldl -lc

Then you set the LD_PRELOADvariable to load it along with your program:

然后设置LD_PRELOAD变量以与程序一起加载它：

$ LD_PRELOAD=./linebufferedstdout.so python test.py | tee -a test.out 
0
1000
2000
3000
4000

If you are lucky, your problem will be solved with no unfortunateside-effects.

如果幸运的话，您的问题将得到解决，不会有任何不幸的副作用。

You can set the LD_PRELOADlibrary in the shell, if necessary, or even specify that library system-wide (definitely NOTrecommended) in /etc/ld.so.preload.

您可以设置LD_PRELOAD，库外壳如果必要的话，甚至可以指定库系统范围（绝对不要在推荐）/etc/ld.so.preload。

回答by Konerak

Have you considered piping to tee?

你有没有考虑过管道到发球台？

./program | tee a.txt

However, even tee won't work if "program" doesn't write anything to stdout until it is done. So, the effectiveness depends a lot on how your program behaves.

但是，如果“程序”在完成之前不向标准输出写入任何内容，则即使 tee 也将不起作用。因此，有效性在很大程度上取决于您的程序的行为方式。

回答by user2238884

If you're trying to modify the behavior of an existing program try stdbuf (part of coreutils starting with version 7.5 apparently).

如果您尝试修改现有程序的行为，请尝试 stdbuf（显然是从 7.5 版开始的 coreutils 的一部分）。

This buffers stdout up to a line:

这将标准输出缓冲到一行：

stdbuf -oL command > output

stdbuf -oL command > output

This disables stdout buffering altogether:

这完全禁用了标准输出缓冲：

stdbuf -o0 command > output

stdbuf -o0 command > output

回答by Andrea Spadaccini

If the program writes to a file, you can read it while it is being written using tail -f a.txt.

如果程序写入文件，您可以在使用tail -f a.txt.

回答by John Vincent

Your problem is that most programs check to see if the output is a terminal or not. If the output is a terminal then output is buffered one line at a time (so each line is output as it is generated) but if the output is not a terminal then the output is buffered in larger chunks (4096 bytes at a time is typical) This behaviour is normal behaviour in the C library (when using printf for example) and also in the C++ library (when using cout for example), so any program written in C or C++ will do this.

您的问题是大多数程序都会检查输出是否为终端。如果输出是终端，则输出一次缓冲一行（因此每行在生成时输出）但如果输出不是终端，则输出以更大的块（一次 4096 字节是典型的）进行缓冲) 这种行为在 C 库（例如使用 printf 时）和 C++ 库（例如使用 cout 时）中是正常行为，因此任何用 C 或 C++ 编写的程序都会这样做。

Most other scripting languages (like perl, python, etc.) are written in C or C++ and so they have exactly the same buffering behaviour.

大多数其他脚本语言（如 perl、python 等）都是用 C 或 C++ 编写的，因此它们具有完全相同的缓冲行为。

The answer above (using LD_PRELOAD) can be made to work on perl or python scripts, since the interpreters are themselves written in C.

上面的答案（使用 LD_PRELOAD）可以用于 perl 或 python 脚本，因为解释器本身是用 C 编写的。

回答by wheeler

The unbuffercommand from the expectpackage does exactly what you are looking for.

包中的unbuffer命令expect完全符合您的要求。

$ sudo apt-get install expect
$ unbuffer python program.py | cat -
<watch output immediately show up here>