Linux UNIX `time` 命令对于基准测试是否足够准确？

Is the UNIX `time` command accurate enough for benchmarks?

提问by chrisdotcode

Let's say I wanted to benchmark two programs: foo.py and bar.py.

假设我想对两个程序进行基准测试：foo.py 和 bar.py。

Are a couple thousand runs and the respective averages of time python foo.pyand time python bar.pyadequate enough for profiling and comparing their speed?


Edit:Additionally, if the execution of each program was sub-second (assume it wasn't for the above), would timestill be okay to use?

有一对夫妇一千运行和各自的平均值time python foo.py和time python bar.py充足的足够分析和比较他们的速度？


编辑：此外，如果每个程序的执行都是亚秒级的（假设不是上述情况），那么time仍然可以使用吗？

采纳答案by Maxim Egorushkin

timeproduces good enough times for benchmarks that run over one second otherwise the time it took exec()ing a process may be large compared to its run-time.

time为运行超过一秒的基准测试产生足够好的时间，否则exec()与运行时间相比，进程花费的时间可能会很大。

However, when benchmarking you should watch out for context switching. That is, another process may be using CPU thus contending for CPU with your benchmark and increasing its run time. To avoid contention with other processes you should run a benchmark like this:

但是，在进行基准测试时，您应该注意上下文切换。也就是说，另一个进程可能正在使用 CPU，从而与您的基准测试竞争 CPU 并增加其运行时间。为避免与其他进程发生争用，您应该运行如下基准测试：

sudo chrt -f 99 /usr/bin/time --verbose <benchmark>

Or

或者

sudo chrt -f 99 perf stat -ddd <benchmark>

sudo chrt -f 99runs your benchmark in FIFO real-time class with priority 99, which makes your process the top priority process and avoids context switching (you can change your /etc/security/limits.confso that it doesn't require a privileged process to use real-time priorities).

sudo chrt -f 99在优先级为 99 的 FIFO 实时类中运行您的基准测试，这使您的进程成为最高优先级进程并避免上下文切换（您可以更改您的进程，/etc/security/limits.conf以便它不需要特权进程来使用实时优先级）。

It also makes timereport all the available stats, including the number of context switches your benchmark incurred, which should normally be 0, otherwise you may like to rerun the benchmark.

它还time报告所有可用的统计信息，包括您的基准测试发生的上下文切换次数，通常应为 0，否则您可能想重新运行基准测试。

perf stat -dddis even more informative than /usr/bin/timeand displays such information as instructions-per-cycle, branch and cache misses, etc.

perf stat -ddd比/usr/bin/time并显示诸如每周期指令、分支和缓存未命中等信息更多信息。

And it is better to disable the CPU frequency scaling and boost, so that the CPU frequency stays constant during the benchmark to get consistent results.

并且最好禁用 CPU 频率缩放和提升，以便在基准测试期间 CPU 频率保持不变以获得一致的结果。

回答by schtever

Yes. The time command gives both elapsed time as well as consumed CPU. The latter is probably what you should focus on, unless you're doing a lot of I/O. If elapsed time is important, make sure the system doesn't have other significant activity while running your test.

是的。time 命令给出了经过的时间和消耗的 CPU。后者可能是您应该关注的重点，除非您进行大量 I/O。如果经过的时间很重要，请确保系统在运行测试时没有其他重要活动。

回答by Basile Starynkevitch

Yes, timeis accurate enough. And you'll need to run only a dozen of times your programs (provided the run lasts more than a second, or a significant fraction of a second - ie more than 200 milliseconds at least). Of course, the file system would be hot (i.e. small files would already be cached in RAM) for most runs (except the first), so take that into account.

是的，time足够准确。并且您只需要运行十几次您的程序（假设运行持续时间超过一秒，或一秒的很大一部分——即至少超过 200 毫秒）。当然，对于大多数运行（除了第一次），文件系统会很热（即小文件已经缓存在 RAM 中），所以要考虑到这一点。

^{the reason you want to have the time-d run to last a few tenths of seconds at least is the accuracy and granularity of the time measurement. Don't expect less than hundredth of second of accuracy. (you need some special kernel option to have it one millisecond)}

^{您希望time-d 运行至少持续十分之几秒的原因是时间测量的准确性和粒度。不要期望精度低于百分之一秒。（你需要一些特殊的内核选项才能让它一毫秒）}

From inside the application, you could use clock, clock_gettime, gettimeofday, getrusage, times(they surely have a Python equivalent).

在应用程序内部，您可以使用clock、clock_gettime、gettimeofday、 getrusage、times（它们肯定有 Python 等效项）。

Don't forget to read the time(7)man page.

不要忘记阅读time(7)手册页。

回答by milianw

Nowadays, imo, there is no reason to use timefor benchmarking purposes. Use perf statinstead. It gives you much more useful information and can repeat the benchmarking process any given number of time and do statistics on the results, i.e. calculate variance and mean value. This is much more reliable and just as simple to use as time:

如今，imo 没有理由time用于基准测试目的。使用perf stat来代替。它为您提供了更多有用的信息，并且可以在任意给定次数内重复基准测试过程并对结果进行统计，即计算方差和平均值。这更可靠，使用起来也很简单time：

perf stat -r 10 -d <your app and arguments>

The -r 10will run your app 10 times and do statistics over it. -doutputs some more data, such as cache misses.

在-r 10将运行你的应用程序10次，做了它的统计数据。-d输出更多数据，例如缓存未命中。

So while timemight be reliable enough for long-running applications, it definitely is not as reliable as perf stat. Use that instead.

因此，虽然time对于长时间运行的应用程序来说可能足够可靠，但它绝对不如perf stat. 改用那个。

Addendum:If you really want to keep using time, at least don't use the bash-builtin command, but the real-deal in verbose mode:

附录：如果你真的想继续使用time，至少不要使用 bash-builtin 命令，而是在详细模式下的 real-deal ：

/usr/bin/time -v <some command with arguments>

The output is then e.g.:

然后输出例如：

    Command being timed: "ls"
    User time (seconds): 0.00
    System time (seconds): 0.00
    Percent of CPU this job got: 0%
    Elapsed (wall clock) time (h:mm:ss or m:ss): 0:00.00
    Average shared text size (kbytes): 0
    Average unshared data size (kbytes): 0
    Average stack size (kbytes): 0
    Average total size (kbytes): 0
    Maximum resident set size (kbytes): 1968
    Average resident set size (kbytes): 0
    Major (requiring I/O) page faults: 0
    Minor (reclaiming a frame) page faults: 93
    Voluntary context switches: 1
    Involuntary context switches: 2
    Swaps: 0
    File system inputs: 8
    File system outputs: 0
    Socket messages sent: 0
    Socket messages received: 0
    Signals delivered: 0
    Page size (bytes): 4096
    Exit status: 0

Especially note how this is capable of measuring the peak RSS, which is often enough if you want to compare the effect of a patch on the peak memory consumption. I.e. use that value to compare before/after and if there is a significant decrease in the RSS peak, then you did something right.

特别注意它如何能够测量峰值 RSS，如果您想比较补丁对峰值内存消耗的影响，这通常就足够了。即使用该值来比较之前/之后，如果 RSS 峰值显着下降，那么您做对了。