java 如何从 JUnit 测试内部运行 JMH？

How to run JMH from inside JUnit tests?

提问by Aleksandr Dubinsky

How can I run JMH benchmarks inside my existing project using JUnit tests? The official documentation recommends making a separate project, using Maven shade plugin, and launching JMH inside the mainmethod. Is this necessary and why is it recommended?

如何使用 JUnit 测试在现有项目中运行 JMH 基准测试？官方文档推荐单独制作一个项目，使用Maven shade插件，在main方法里面启动JMH 。这是必要的，为什么推荐？

回答by Aleksandr Dubinsky

I've been running JMH inside my existing Maven project using JUnit with no apparent ill effects. I cannot answer why the authors recommend doing things differently. I have not observed a difference in results. JMH launches a separate JVM to run benchmarks to isolate them. Here is what I do:

我一直在使用 JUnit 在我现有的 Maven 项目中运行 JMH，没有明显的不良影响。我无法回答为什么作者建议做不同的事情。我没有观察到结果的差异。JMH 启动了一个单独的 JVM 来运行基准测试来隔离它们。这是我所做的：

• Add the JMH dependencies to your POM:

  <dependency>
    <groupId>org.openjdk.jmh</groupId>
    <artifactId>jmh-core</artifactId>
    <version>1.21</version>
    <scope>test</scope>
  </dependency>
  <dependency>
    <groupId>org.openjdk.jmh</groupId>
    <artifactId>jmh-generator-annprocess</artifactId>
    <version>1.21</version>
    <scope>test</scope>
  </dependency>
  

  Note that I've placed them in scope test.

  In Eclipse, you may need to configure the annotation processor manually. NetBeans handles this automatically.

• Create your JUnit and JMH class. I've chosen to combine both into a single class, but that is up to you. Notice that OptionsBuilder.includeis what actually determines which benchmarks will be run from your JUnit test!

  import java.util.ArrayList;
  import java.util.List;
  import java.util.Random;
  import java.util.concurrent.TimeUnit;
  import org.junit.Test;
  import org.openjdk.jmh.annotations.*;
  import org.openjdk.jmh.infra.Blackhole;
  import org.openjdk.jmh.runner.Runner;
  import org.openjdk.jmh.runner.options.*;
  
  
  public class TestBenchmark 
  {
  
        @Test public void 
      launchBenchmark() throws Exception {
  
              Options opt = new OptionsBuilder()
                  // Specify which benchmarks to run. 
                  // You can be more specific if you'd like to run only one benchmark per test.
                  .include(this.getClass().getName() + ".*")
                  // Set the following options as needed
                  .mode (Mode.AverageTime)
                  .timeUnit(TimeUnit.MICROSECONDS)
                  .warmupTime(TimeValue.seconds(1))
                  .warmupIterations(2)
                  .measurementTime(TimeValue.seconds(1))
                  .measurementIterations(2)
                  .threads(2)
                  .forks(1)
                  .shouldFailOnError(true)
                  .shouldDoGC(true)
                  //.jvmArgs("-XX:+UnlockDiagnosticVMOptions", "-XX:+PrintInlining")
                  //.addProfiler(WinPerfAsmProfiler.class)
                  .build();
  
              new Runner(opt).run();
          }
  
      // The JMH samples are the best documentation for how to use it
      // http://hg.openjdk.java.net/code-tools/jmh/file/tip/jmh-samples/src/main/java/org/openjdk/jmh/samples/
      @State (Scope.Thread)
      public static class BenchmarkState
      {
          List<Integer> list;
  
            @Setup (Level.Trial) public void
          initialize() {
  
                  Random rand = new Random();
  
                  list = new ArrayList<>();
                  for (int i = 0; i < 1000; i++)
                      list.add (rand.nextInt());
              }
      }
  
        @Benchmark public void 
      benchmark1 (BenchmarkState state, Blackhole bh) {
  
              List<Integer> list = state.list;
  
              for (int i = 0; i < 1000; i++)
                  bh.consume (list.get (i));
          }
  }
  

• JMH's annotation processor seems to not work well with compile-on-save in NetBeans. You may need to do a full Clean and Buildwhenever you modify the benchmarks. (Any suggestions appreciated!)

• Run your launchBenchmarktest and watch the results!

  -------------------------------------------------------
   T E S T S
  -------------------------------------------------------
  Running com.Foo
  # JMH version: 1.21
  # VM version: JDK 1.8.0_172, Java HotSpot(TM) 64-Bit Server VM, 25.172-b11
  # VM invoker: /usr/lib/jvm/java-8-jdk/jre/bin/java
  # VM options: <none>
  # Warmup: 2 iterations, 1 s each
  # Measurement: 2 iterations, 1 s each
  # Timeout: 10 min per iteration
  # Threads: 2 threads, will synchronize iterations
  # Benchmark mode: Average time, time/op
  # Benchmark: com.Foo.benchmark1
  
  # Run progress: 0.00% complete, ETA 00:00:04
  # Fork: 1 of 1
  # Warmup Iteration   1: 4.258 us/op
  # Warmup Iteration   2: 4.359 us/op
  Iteration   1: 4.121 us/op
  Iteration   2: 4.029 us/op
  
  
  Result "benchmark1":
    4.075 us/op
  
  
  # Run complete. Total time: 00:00:06
  
  REMEMBER: The numbers below are just data. To gain reusable insights, you need to follow up on
  why the numbers are the way they are. Use profilers (see -prof, -lprof), design factorial
  experiments, perform baseline and negative tests that provide experimental control, make sure
  the benchmarking environment is safe on JVM/OS/HW level, ask for reviews from the domain experts.
  Do not assume the numbers tell you what you want them to tell.
  
  Benchmark                                Mode  Cnt  Score   Error  Units
  Foo.benchmark1                           avgt    2  4.075          us/op
  Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 7.013 sec
  

• Runner.runeven returns RunResultobjects on which you can do assertions, etc.

• 将 JMH 依赖项添加到您的 POM：

  <dependency>
    <groupId>org.openjdk.jmh</groupId>
    <artifactId>jmh-core</artifactId>
    <version>1.21</version>
    <scope>test</scope>
  </dependency>
  <dependency>
    <groupId>org.openjdk.jmh</groupId>
    <artifactId>jmh-generator-annprocess</artifactId>
    <version>1.21</version>
    <scope>test</scope>
  </dependency>
  

  请注意，我已将它们放在 scope 中test。

  在 Eclipse 中，您可能需要手动配置注释处理器。NetBeans 会自动处理此问题。

• 创建您的 JUnit 和 JMH 类。我选择将两者合并为一个类，但这取决于您。请注意，OptionsBuilder.include这实际上决定了将从您的 JUnit 测试运行哪些基准测试！

  import java.util.ArrayList;
  import java.util.List;
  import java.util.Random;
  import java.util.concurrent.TimeUnit;
  import org.junit.Test;
  import org.openjdk.jmh.annotations.*;
  import org.openjdk.jmh.infra.Blackhole;
  import org.openjdk.jmh.runner.Runner;
  import org.openjdk.jmh.runner.options.*;
  
  
  public class TestBenchmark 
  {
  
        @Test public void 
      launchBenchmark() throws Exception {
  
              Options opt = new OptionsBuilder()
                  // Specify which benchmarks to run. 
                  // You can be more specific if you'd like to run only one benchmark per test.
                  .include(this.getClass().getName() + ".*")
                  // Set the following options as needed
                  .mode (Mode.AverageTime)
                  .timeUnit(TimeUnit.MICROSECONDS)
                  .warmupTime(TimeValue.seconds(1))
                  .warmupIterations(2)
                  .measurementTime(TimeValue.seconds(1))
                  .measurementIterations(2)
                  .threads(2)
                  .forks(1)
                  .shouldFailOnError(true)
                  .shouldDoGC(true)
                  //.jvmArgs("-XX:+UnlockDiagnosticVMOptions", "-XX:+PrintInlining")
                  //.addProfiler(WinPerfAsmProfiler.class)
                  .build();
  
              new Runner(opt).run();
          }
  
      // The JMH samples are the best documentation for how to use it
      // http://hg.openjdk.java.net/code-tools/jmh/file/tip/jmh-samples/src/main/java/org/openjdk/jmh/samples/
      @State (Scope.Thread)
      public static class BenchmarkState
      {
          List<Integer> list;
  
            @Setup (Level.Trial) public void
          initialize() {
  
                  Random rand = new Random();
  
                  list = new ArrayList<>();
                  for (int i = 0; i < 1000; i++)
                      list.add (rand.nextInt());
              }
      }
  
        @Benchmark public void 
      benchmark1 (BenchmarkState state, Blackhole bh) {
  
              List<Integer> list = state.list;
  
              for (int i = 0; i < 1000; i++)
                  bh.consume (list.get (i));
          }
  }
  

• JMH 的注释处理器似乎不能很好地与 NetBeans 中的 compile-on-save 配合使用。Clean and Build每当您修改基准时，您可能需要做一个完整的。（任何建议表示赞赏！）

• 运行您的launchBenchmark测试并观察结果！

  -------------------------------------------------------
   T E S T S
  -------------------------------------------------------
  Running com.Foo
  # JMH version: 1.21
  # VM version: JDK 1.8.0_172, Java HotSpot(TM) 64-Bit Server VM, 25.172-b11
  # VM invoker: /usr/lib/jvm/java-8-jdk/jre/bin/java
  # VM options: <none>
  # Warmup: 2 iterations, 1 s each
  # Measurement: 2 iterations, 1 s each
  # Timeout: 10 min per iteration
  # Threads: 2 threads, will synchronize iterations
  # Benchmark mode: Average time, time/op
  # Benchmark: com.Foo.benchmark1
  
  # Run progress: 0.00% complete, ETA 00:00:04
  # Fork: 1 of 1
  # Warmup Iteration   1: 4.258 us/op
  # Warmup Iteration   2: 4.359 us/op
  Iteration   1: 4.121 us/op
  Iteration   2: 4.029 us/op
  
  
  Result "benchmark1":
    4.075 us/op
  
  
  # Run complete. Total time: 00:00:06
  
  REMEMBER: The numbers below are just data. To gain reusable insights, you need to follow up on
  why the numbers are the way they are. Use profilers (see -prof, -lprof), design factorial
  experiments, perform baseline and negative tests that provide experimental control, make sure
  the benchmarking environment is safe on JVM/OS/HW level, ask for reviews from the domain experts.
  Do not assume the numbers tell you what you want them to tell.
  
  Benchmark                                Mode  Cnt  Score   Error  Units
  Foo.benchmark1                           avgt    2  4.075          us/op
  Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 7.013 sec
  

• Runner.run甚至返回RunResult您可以对其进行断言等的对象。

回答by myset

Other example

其他例子

@State(Scope.Benchmark)
@Threads(1)
public class TestBenchmark {

    @Param({"10","100","1000"})
    public int iterations;


    @Setup(Level.Invocation)
    public void setupInvokation() throws Exception {
        // executed before each invocation of the benchmark
    }

    @Setup(Level.Iteration)
    public void setupIteration() throws Exception {
        // executed before each invocation of the iteration
    }

    @Benchmark
    @BenchmarkMode(Mode.AverageTime)
    @Fork(warmups = 1, value = 1)
    @Warmup(batchSize = -1, iterations = 3, time = 10, timeUnit = TimeUnit.MILLISECONDS)
    @Measurement(batchSize = -1, iterations = 10, time = 10, timeUnit = TimeUnit.MILLISECONDS)
    @OutputTimeUnit(TimeUnit.MILLISECONDS)
    public void test() throws Exception {
       Thread.sleep(ThreadLocalRandom.current().nextInt(0, iterations));
    }


    @Test
    public void benchmark() throws Exception {
        String[] argv = {};
        org.openjdk.jmh.Main.main(argv);
    }

}