Established benchmark suites for the Java Virtual Machine (JVM), such as DaCapo, ScalaBench, and SPECjvm2008, lack workloads that take advantage of the parallel programming abstractions and concurrency primitives offered by the JVM and by the Java Class Library. However, such workloads are fundamental for understanding the way in which modern applications and data-processing frameworks use the JVM’s concurrency features, and for validating new just-in-time (JIT) compiler optimizations that enable more efficient execution of such workloads.
In collaboration with the Graal team in Zurich and Linz, Charles University and Johannes Kepler University Linz, we have significantly contributed to Renaissance, a new benchmark suite which consists of 21 parallelism- and concurrency-oriented benchmarks for the JVM, which exercise a wide variety of concurrency primitives and rely on multiple existing state-of-the-art Java and Scala frameworks. Our contribution spans workload selection and discovery, maintenance of the open-source release, as well as workload analysis and assessment wrt. other prominent benchmark suites. We have shown that the proposed benchmarks are diverse (i.e., they span the space of concurrency metrics better than the existing suites), while being as complex as DaCapo and ScalaBench and much more complex than SPECjvm2008. Moreover, we have shown that the use of concurrency primitives in these workloads reveals optimization opportunities in the JIT compiler that are not visible in the workloads of the previous benchmark suites. In particular, we have used Renaissance to expose four new compiler optimizations in Graal.
This work has been published at PLDI’19 [1] and received two badges related to the artifact evaluation process (“Artifacts Functional” and “Artifacts Available”). We also published a technical report unveiling further aspects of Renaissance [2]. The benchmark suite has been released open-source [A][B] and is being actively maintained.
Key Publications
[1] Aleksandar Prokopec, Andrea Rosà, David Leopoldseder, Gilles Duboscq, Petr Tuma, Martin Studener, Lubomír Bulej, Yudi Zheng, Alex Villazón, Doug Simon, Thomas Würthinger, Walter Binder: Renaissance: Benchmarking Suite for Parallel Applications on the JVM. PLDI 2019: 31-47 [pdf]
[2] Aleksandar Prokopec, Andrea Rosà, David Leopoldseder, Gilles Duboscq, Petr Tuma, Martin Studener, Lubomír Bulej, Yudi Zheng, Alex Villazón, Doug Simon, Thomas Würthinger, Walter Binder: On Evaluating the Renaissance Benchmarking Suite: Variety, Performance, and Complexity. CoRR abs/1903.10267 (2019) [pdf]
Software
[A] Website: https://renaissance.dev
[B] GitHub repository: https://github.com/renaissance-benchmarks/renaissance
Other Resources
[a] Slides of the talk at SE’20
[b] Poster presented at SPLASH’19