Oracle vs IBM Java performance

Question

I'm trying to compare the performance of the two main Java implementations: Oracle and IBM on running the following test:

public class HarmonicSeriesTest {

    public static void main(String[] args) {
        long startTime = System.currentTimeMillis();
        final int limit = 20;
        double sum = 0.0;
        long n = 0;
        while (sum < limit) {
            n++;
            sum += 1.0 / n;
        }
        long duration = System.currentTimeMillis() - startTime;
        System.out.printf("n is %d\n", n);
        System.out.printf("Executed in %d miliseconds\n", duration);
    }
}

By running the above code with:

1. IBM JRE 1.8.0

java version "1.8.0" Java(TM) SE Runtime Environment (build pwa6480sr3fp22-20161213_02(SR3 FP22)) IBM J9 VM (build 2.8, JRE 1.8.0 Windows 10 amd64-64 Compressed References 20161209_329148 (JIT enabled, AOT enabled) J9VM - R28_20161209_1345_B329148 JIT - tr.r14.java.green_20161207_128946 GC - R28_20161209_1345_B329148_CMPRSS J9CL - 20161209_329148) JCL - 20161213_01 based on Oracle jdk8u111-b14

2. Oracle JRE 1.8.0_131

java version "1.8.0_131" Java(TM) SE Runtime Environment (build 1.8.0_131-b11) Java HotSpot(TM) 64-Bit Server VM (build 25.131-b11, mixed mode)

3. Sun JRE 1.5.0_22

java version "1.5.0_22" Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_22-b03) Java HotSpot(TM) 64-Bit Server VM (build 1.5.0_22-b03, mixed mode)

I consistently get the following results:

1. IBM JRE 1.8.0 Compiler compliance level 1.8

n is 272400600
Executed in 1024 miliseconds

2. Oracle JRE 1.8.0_131 Compiler compliance level 1.8

n is 272400600
Executed in 2002 miliseconds

3. Sun JRE 1.5.0_22 Compiler compliance level 1.5

n is 272400600
Executed in 1506 miliseconds

As you can see, the Oracle JRE is slower by 100% compared to IBM JRE and 25% compared to Sun JRE. The gap between Oracle and IBM implementations is huge, also Oracle JRE looks like a regression over Sun JRE.

Does anyone have an explanation why is the 'official' implementation of Java so slow?

I have no interest in using JMH since I don't want to benchmark anything.

Bellow is the hardware configuration I used for testing:

Answer 1

First off, I am not a performance expert, but I play around with JMH.

It's best to test this in a tool like JMH which will allow you to compare apples with apples.

Different Java runtimes will start up faster, and optimise later. For instance a loop can be unrolled and the hotspot will kick in.

Running a test once will not give you an accurate sample. i.e. The more the code runs,the faster it will get. There is a C1, C2, JIT and a bunch of magic the the different vendors implement to make code run faster and faster.

These are the steps to create a project in JHM that will hopefully help you compare apples with apples.

1. Create an JHM project:

   mvn -DarchetypeGroupId=org.openjdk.jmh -DarchetypeArtifactId=jmh-java-benchmark-archetype -DarchetypeVersion=1.18 -DarchetypeRepository=http://repo.maven.apache.org/maven2/ -DgroupId=com.yourcompany -DartifactId=HarmonicSeriesTest -Dversion=1.0-SNAPSHOT -Dpackage=com.yourcompany.harmonicseriestest -Darchetype.interactive=false --batch-mode archetype:generate

2. Build the project

   cd HarmonicSeriesTest/

   mvn package

3. Edit the generated class

Edit MyBenchmark.java

import java.util.concurrent.TimeUnit;
import org.openjdk.jmh.annotations.Benchmark;
import org.openjdk.jmh.annotations.BenchmarkMode;
import org.openjdk.jmh.annotations.Mode;
import org.openjdk.jmh.annotations.OutputTimeUnit;

public class MyBenchmark {
    @Benchmark
    @BenchmarkMode(Mode.AverageTime)
    @OutputTimeUnit(TimeUnit.MILLISECONDS)
    public double harmonicSeriesTest() {
        final int limit = 20;
        double sum = 0.0;
        long n = 0;
        while (sum < limit) {
            n++;
            sum += 1.0 / n;
        }
        return sum;
    }

}

4. Run the project on the command line and not an IDE as this can effect things

Execute on the command line:

mvn package
cd target
java -jar benchmarks.jar

My output. As you can see the first iteration is slow and then it speeds up. Look for the first Iteration after the warmup as this is the value you should compare across Java implementations and using different JVM flags.

JMH 1.18 (released 61 days ago)
VM version: JDK 1.8.0_102, VM 25.102-b14
VM invoker: /home/richard/install/java/jdk1.8.0_102/jre/bin/java
VM options: <none>
Warmup: 20 iterations, 1 s each
Measurement: 20 iterations, 1 s each
Timeout: 10 min per iteration
Threads: 1 thread, will synchronize iterations
Benchmark mode: Average time, time/op
Benchmark: com.yourcompany.harmonicseriestest.MyBenchmark.harmonicSeriesTest

Run progress: 0.00% complete, ETA 00:06:40
Fork: 1 of 10
Warmup Iteration   1: 1773.454 ms/op
Warmup Iteration   2: 1782.517 ms/op
Warmup Iteration   3: 1545.739 ms/op
Warmup Iteration   4: 1542.968 ms/op
Warmup Iteration   5: 1530.740 ms/op
Warmup Iteration   6: 1539.304 ms/op
Warmup Iteration   7: 1538.079 ms/op
Warmup Iteration   8: 1535.280 ms/op
Warmup Iteration   9: 1547.716 ms/op
Warmup Iteration  10: 1520.056 ms/op
Warmup Iteration  11: 1503.892 ms/op
Warmup Iteration  12: 1513.037 ms/op
Warmup Iteration  13: 1521.966 ms/op
Warmup Iteration  14: 1519.931 ms/op
Warmup Iteration  15: 1515.179 ms/op
Warmup Iteration  16: 1514.342 ms/op
Warmup Iteration  17: 1525.555 ms/op
Warmup Iteration  18: 1519.022 ms/op
Warmup Iteration  19: 1533.529 ms/op
Warmup Iteration  20: 1525.547 ms/op
Iteration   1: 1524.958 ms/op

NB These are the golden samples for testing the JVM. Micro-benchmarking is tricky and it's easy to make one that is 'wrong'.

Also have a look at blogs like Nitsan's and slides like Aleksey Shipilev's as they really know what they are doing :).