How can I use JMH for Scala benchmarks together with sbt?

11
votes

I have tried to use together with , but so far I have not managed to set it up properly so that .scala based benchmarks work.

As the combination sbt + .java based benchmarks works, I tried to start from that base. I am using sbt 0.13.1.

.java based benchmarks using sbt

build.sbt

import AssemblyKeys._

name := "scala-benchmark"

version := "1.0"

scalaVersion := "2.10.3"

scalacOptions += "-deprecation"

libraryDependencies += "org.openjdk.jmh" % "jmh-core" % "0.5.5"

libraryDependencies += "org.openjdk.jmh" % "jmh-java-benchmark-archetype" % "0.5.5"

libraryDependencies += "org.openjdk.jmh" % "jmh-generator-annprocess" % "0.5.5"

libraryDependencies += "org.openjdk.jmh" % "jmh-generator-bytecode" % "0.5.5"

assemblySettings

jarName in assembly := "microbenchmarks.jar"

test in assembly := {}

mainClass in assembly := Some("org.openjdk.jmh.Main")

To get a single "fat" jar at the end, the sbt-assembly plugin is required:

project/assembly.sbt

addSbtPlugin("com.eed3si9n" % "sbt-assembly" % "0.11.2")

A simple benchmark:

src/main/java/app/benchmark/java/benchmark2/Benchmark2.java

package app.benchmark.java.benchmark2;

import org.openjdk.jmh.annotations.GenerateMicroBenchmark;
import org.openjdk.jmh.runner.Runner;
import org.openjdk.jmh.runner.RunnerException;
import org.openjdk.jmh.runner.options.Options;
import org.openjdk.jmh.runner.options.OptionsBuilder;

public class Benchmark2 {
    @GenerateMicroBenchmark
    public int run() {
        int result = 0;
        for (int i = 0; i < 10; i++) {
            result += i * i;
        }
        return result;
    }
}

Running sbt assembly gives this output:

$ sbt assembly
[...]
[info] Compiling 2 Scala sources and 2 Java sources to ...
[warn] warning: Supported source version 'RELEASE_6' from annotation processor 'org.openjdk.jmh.generators.GenerateMicroBenchmarkProcessor' less than -source '1.8'
[warn] 1 warning
[info] Including: jmh-java-benchmark-archetype-0.5.5.jar
[info] Including: jmh-generator-bytecode-0.5.5.jar
[info] Including: jopt-simple-4.6.jar
[info] Including: jmh-generator-reflection-0.5.5.jar
[info] Including: jmh-generator-annprocess-0.5.5.jar
[info] Including: asm-4.2.jar
[info] Including: commons-math3-3.2.jar
[info] Including: jmh-core-0.5.5.jar
[info] Including: scala-library.jar
[...]
[info] Packaging /home/scala-2.10/vc/rhaag/scala/scala-benchmark/target/scala-2.10/microbenchmarks.jar ...

and the resulting microbenchmarks.jar contains everything required to run the benchmarks:

$ java -jar target/scala-2.10/microbenchmarks.jar -wi 3 -i 3 -f 1 app.benchmark.java.benchmark2.Benchmark2.run 

[...] 

Benchmark                  Mode   Samples         Mean   Mean error    Units 

a.b.j.b.Benchmark2.run    thrpt         3   607555,968    70243,275   ops/ms

So far so good.

Scala benchmarks using sbt

From that base I tried to switch to .scala based benchmarks:

build.sbt

Replacing the Java archetype with the Scala one

libraryDependencies += "org.openjdk.jmh" % "jmh-scala-benchmark-archetype" % "0.5.5"

does not work, as the download fails.

This works:

libraryDependencies += "org.openjdk.jmh" % "jmh-scala-benchmark-archetype" % "0.5.5" from "http://repo1.maven.org/maven2/org/openjdk/jmh/jmh-scala-benchmark-archetype/0.5.5/jmh-scala-benchmark-archetype-0.5.5.jar"

Another simple benchmark:

src/main/scala/app/benchmark/scala/benchmark2/Benchmark2.scala

package app.benchmark.scala.benchmark2

import org.openjdk.jmh.annotations.GenerateMicroBenchmark
import org.openjdk.jmh.runner.Runner
import org.openjdk.jmh.runner.RunnerException
import org.openjdk.jmh.runner.options.Options
import org.openjdk.jmh.runner.options.OptionsBuilder

class Benchmark2 {
  @GenerateMicroBenchmark
  def run() = {
    Seq.range(0, 10).map(i => i * i).sum
  }
}

Now sbt assembly creates the jar file, but target/scala-2.10/microbenchmarks.jar#META-INF/MicroBenchmarks does not list the Scala benchmarks, and these are not shown by java -jar target/scala-2.10/microbenchmarks.jar -l either.

Resources:

How can I integrate the (bytecode based) JMH processor for Scala? Or from another perspective: Why is the (annotation based) JMH processor picked up automatically and produces Java based benchmarks?

1
scalasbtmicrobenchmarkjmh
Including archetype dependencies is futile, those are just the project templates. – Aleksey Shipilev
I don't know how SBT works internally, but here's how Maven compiles JMH benchmarks with Scala: hg.openjdk.java.net/code-tools/jmh/file/tip/jmh-archetypes/…. You might want to replicate it in SBT :) – Aleksey Shipilev
hmmm, @GenerateMicroBenchmark seems to be ignored in .scala files. I looked at the Japanese page and the work around he did was to put the code to be benchmarked in a separate scala file and call it from a Java method which has the annotation assigned. No idea other than that why SBT is not picking it up. – M.K.
We discussed that with Jamie Allen before, and figured SBT does not process Java annotations. That is when JMH bytecode generator was born, and JMH Scala archetypes use that generator to parse out the bytecode from scalac, and generate the JMH benchmarks. The link above outlines what it takes in Maven. – Aleksey Shipilev
@AlekseyShipilev, thanks for that info. I will try to get around to writing a task (maybe even a plugin?) to generate the JMH benchmarks via sbt. – M.K.

1 Answers

12
votes

I have implemented an sbt-jmh plugin that actually works: https://github.com/ktoso/sbt-jmh

Currently building benchmarks.jar is not supported, but you can simply type run -i 10 .*MyBenchmark.* and it will work as expected (doing all the multi-step compilation for you).

I hope this helps, cheers!