Java Microbenchmark Harness (JMH)

The JMH is an open source project that helps you to implement microbenchmarking correctly. You can be sure that this toolkit is one of the best because it's developed by the same people who work on the JVM. 

The recommended way to work with JMH is by using Maven. You can install Maven using the instructions at the following link: http://maven.apache.org/install.html. Then, you can execute the following command to generate a test project:

mvn archetype:generate \
          -DinteractiveMode=false \
          -DarchetypeGroupId=org.openjdk.jmh \
          -DarchetypeArtifactId=jmh-java-benchmark-archetype \
          -DgroupId=org.sample \
          -DartifactId=test \
          -Dversion=1.0

The generated project will have the following structure:

And the MyBenchmark file will look like this:

public class MyBenchmark {

    @Benchmark
    public void testMethod() {
        // This is a demo/sample template for building your JMH benchmarks. Edit as needed.
        // Put your benchmark code here.
    }

}

If you want to generate a project with another name and package, you have to specify arguments using the -DgroupId and -DartifactId parameters. Now, you can write your own implementation of testMethod(), for instance:

@Benchmark
public void testMethod() {
    int a = 3;
    int b = 4;
    int c = a + b;
}

Run the mvn clean install command in the root folder of the project and the target folder will be generated. At this point, the structure of your project will look as follows:

The most important file for our present purposes is benchmarks.jar. It contains the compiled MyBenchmark class and all JMH classes needed to run the .jar file. If you're going to add any external dependencies, you have to declare them inside the pom.xml file. Now you have a platform-independent .jar file, so you can run your test on any machine that has the JVM installed. Here's the command for running the test:

java -jar target/benchmarks.jar

In the output, you can see that the default configuration is 10 forks by 20 warm-ups, with simple iterations:

# Run progress: 90.00% complete, ETA 00:00:40
# Fork: 10 of 10
# Warmup Iteration 1: 2268891595.825 ops/s
# Warmup Iteration 2: 2028918125.250 ops/s
# Warmup Iteration 3: 2410600803.077 ops/s
......
# Warmup Iteration 17: 2224632990.097 ops/s
# Warmup Iteration 18: 2190560330.537 ops/s
# Warmup Iteration 19: 2117820907.659 ops/s
# Warmup Iteration 20: 2364498478.602 ops/s
Iteration 1: 2851568284.147 ops/s
Iteration 2: 2539539878.294 ops/s
Iteration 3: 2332746312.271 ops/s
.....
Iteration 17: 2389690978.812 ops/s
Iteration 18: 2285020290.690 ops/s
Iteration 19: 2220938195.822 ops/s
Iteration 20: 2242128929.564 ops/s

And the result looks like this:

Result "org.sample.MyBenchmark.testMethod":
2273140780.281 ±(99.9%) 60161283.208 ops/s [Average]
(min, avg, max) = (1065162420.546, 2273140780.281, 2989552931.957), stdev = 254726640.198
CI (99.9%): [2212979497.074, 2333302063.489] (assumes normal distribution)
# Run complete. Total time: 00:06:49
Benchmark              Mode  Cnt Score Error Units
MyBenchmark.testMethod thrpt 200 2273140780.281 ± 60161283.208 ops/s

The ops/s measure is the number of operations per second, or how many times the JMH can execute your method in one second. This number points to the fact that the default mode is Throughput. You can change this using the OptionsBuilder class:

public static void main(String[] args) throws RunnerException {
    Options opt = new OptionsBuilder()
            .include(MyBenchmark.class.getSimpleName())
            .forks(1)
            .measurementIterations(1)
            .mode(Mode.AverageTime)
            .build();

    new Runner(opt).run();
}

Or by using the special annotation @BenchmarkMode for methods. You can also use command-line arguments to set up the number of iterations and forks; for instance:

java -jar target/benchmarks.jar MyBenchmark -wi 5 -i 5 -f 1

In this case, we requested five warm-up and measurement iterations and a single fork.