Renaissance 0.16.0

This release brings improved benchmark validation, numerous platform and library updates, and improved compatibility with recent Java versions. Even though the source code of most benchmarks remains unchanged (apart from fixes and changes due to introducing validation in several benchmarks), the actual code executed at runtime may be affected by the dependency updates and exhibit changes in performance.

Workload validation

Validation of workload results ensures that changes in the underlying libraries or the JVM do not break the workloads. This release adds or improves validation in 7 workloads, bringing Renaissance closer to having fully validated workloads (there are 4 workloads left). In most cases, adding validation does not impact the reported benchmark performance, because it is performed outside the timed phase. However, in some cases, adding validation required changing the timed part of the benchmark, which is the case of the future-genetic and gauss-mix benchmarks (see the next section for a summary of changes with performance impact).

• Add validation to fj-kmeans benchmark (#431)
• Add validation to future-genetic benchmark (#432)
• Add validation to als benchmark (#435)
• Improve validation in log-regression benchmark (#436)
• Add validation to movie-lens benchmark (#441)
• Add validation to philosophers benchmark (#446)
• Improve validation in gauss-mix validation (#448)

Workload changes (with impact on performance)

In some benchmarks, adding validation required changes to the benchmark code that impact performance. In the future-genetic benchmark, obtaining stable results for validation required associating random number generators with work items, not worker threads. This change eliminated work thread contention on a shared Random instance, leading to higher performance. In the gauss-mix benchmark, adding a more in-depth validation required training multiple models with reduced size and dimensionality of the input data.

In other cases, the benchmark code was modified to get a more reasonable behavior. In the philosophers benchmark, the rendering of system state snapshots (into string) was moved out of the transaction to avoid blocking the philosopher threads. In addition, the number of state snapshots was fixed to prevent the benchmark from performing variable amount of work. The movie-lens benchmark now loads input data directly from resource files, avoding unnecessary IO, but also uses Spark-provided methods to load CSV files to properly handle quoted values. This results in different partitioning defaults in subsequent computations, impacting performance.

• Avoid contention on shared Random in the future-genetic benchmark (#432)
• Train multiple models in the gauss-mix benchmark (#441)
• Reduce contention and make the amount of work in the philosophers benchmark constant (#447)
• Use Spark-provided methods to load CSV files and avoid extra IO in movie-lens (#455)

Externally visible changes

The chromosome_count parameter name in the future-genetic benchmark has been renamed to population_size to better match what it really represents in the benchmark.

• Refactor parameter and variable names for clarity in the future-genetic benchmark (#463)

Java Compatibility

To improve compatibility with Java 23, the hadoop-client-api library used by the Apache Spark framework has been patched to avoid using the deprecated security manager API which prevented running on Java 23 without using the -Djava.security.manager=allow. The database libraries used in the db-shootout benchmark were updated to recent versions and, together with a fix to Chronicle initialization, the benchmark is now compatible with current Java releases.

• Patch Hadoop Client API jar to avoid deprecated Java Security API on Java 23 (#453)
• Update benchmark dependencies (#456)
• Set 'java.class.path' temporarily during Chronicle intialization (#459)

Dependency updates

The Scala 2.12 and 2.13 versions used by various benchmarks were updated to versions 2.12.20 and 2.13.15, respectively. The philosophers and scala-stm-bench7 benchmarks are now using Scala 3.3.4, together with the other benchmarks brought to Scala 3 in previous releases. This leaves only the reactors benchmark in the Scala 2.12 camp.

The following primary frameworks/libraries used by various benchmarks we updated:

• apache-spark benchmarks now use Spark version 3.5.3
• twitter-finagle benchmarks now use Finagle version 24.2.0
• neo4j-analytics now uses Neo4J version 5.25.1
• future-genetic now uses Jenetics version 6.3.0
• rx now uses RXJava3 version 3.1.9
• scala-dotty now uses Scala 3 compiler version 3.3.4
• scala-stm benchmarks now use Scala STM version 0.11.1
• db-shootout benchmark now uses H2 MVStore version 2.3.232, MapDB
  version 3.1.0, and Chronicle Map version 3.26ea4.

There are also numerous updates to the secondary/transitive dependencies such as Arrow, ASM, Caffeine, Guava, Jackson, Jersey, JNA, Netty, Parquet, and many others. See the relevant pull requests (especially #456) for details.

• Switch to external scala-stm and update to version 0.11.1 (#444)
• Migrate scala-stm benchmarks from Scala 2.12 to Scala 3 (#445)
• Update benchmark dependencies (#456)

Build system and CI changes

• Update SBT to version 1.9.9 and drop SBT binaries (#450)
• Speed up CI by building Renaissance once and running it on multiple JDKs (#454)

Thanks

This release includes contributions from @lovisek, @BohdanQQ, @guilhas07, and @reneleonhardt in addition to @vhotspur and @lbulej. Many thanks!

Renaissance 0.15.0

This release is primarily aimed at improving compatibility with Java 21 (LTS) and Java 22 (EA), but it also marks the departure from Java 8 by requiring at least Java 11 to build the suite and to run the benchmarks. Several benchmarks now use Scala 3 and a memory leak was plugged in one of the benchmarks.

To improve compatibility with Java 21, the Apache Spark framework used by the apache-spark benchmarks was updated to version 3.5.0 and the Neo4j framework used by the neo4j-analytics benchmark was updated to version 5.12.0 (the benchmark now requires Java 17 to run). The database libraries used in the db-shootout benchmark were updated to more recent versions, which makes the benchmark compatible with Java 18 (while still far from Java 21, it is an improvement over Java 11, where the benchmark was stuck for some time).

The Scala 2.12 and 2.13 versions used by various benchmarks were updated to versions 2.12.18 and 2.13.12, respectively. The akka-uct, dotty, and scala-kmeans are now using Scala 3.3.1, and the neo4j-analytics benchmark now uses Scala 2.13. This leaves only the philosophers, reactors and stala-stm-bench7 benchmarks in the Scala 2.12 camp.

Even though the source code of most benchmarks remains unchanged (apart from bug fixes and changes due to library APIs), the actual code executed at runtime may be affected by the dependency updates.

Other changes include a fix for a memory leak in the reactors benchmark, and two fixes that improve compatibility with JMH.

The actual changes are spread over various pull requests, with the most important categorized below.

Dependency updates

• Update apache-spark benchmarks to use Spark 3.5.0 (#399) and Hadoop 3.3.6 (#425)
  • Makes the apache-spark benchmarks compatible with Java 21 (up from Java 20)
  • The Hadoop update improves compatibility with the (open) IBM Semeru OpenJ9-based JVM builds
• Update rx-scrabble benchmark to use RxJava 3.1.8 and migrate to rxjava3 API (#414)
• Update db-shootout benchmark to use H2 MVStore version 2.2.224, MapDB version 3.0.10, and Chronicle Map version 3.22.9 (#419, #412)
  • Makes the benchmark compatible with Java 18 (up from Java 11)
• Update dotty benchmark to use scala3-compiler 3.3.1 and migrate to Scala 3.3 (#407)
• Update akka-uct benchmark to use Akka Actors 2.6.32 and migrate to Scala 3.3 (#406)
• Migrate scala-stdlib benchmarks to Scala 3.3 (#405)
• Update twitter-finagle benchmarks to use Finagle 22.12.0 (#404)
• Update neo4j-analytics benchmark to use Neo4j 5.12.0 and migrate to Scala 2.13 (#419, #422)
  • Makes the benchmark compatible with Java 21 (up from Java 15, but requires at least Java 17)
• Update Scala 2 benchmarks to use Scala 2.12.18 and 2.13.12 (#411)
• Update various shared dependencies (#409, #419, #423)
  • Most notably JNA 5.13.0, Netty 4.1.99, Guava 32.1.2-jre, ASM 9.6, Eclipse Collections 11.1.0, Jackson 2.15.2, and Jersey 2.40

Workload changes/fixes

• Prevent memory leaks in reactors benchmark (#389)
• Ensure that forked tasks get executed by threads from the fork-join pool in fj-kmeans (#420)
• Ensure that JMH invokes benchmark methods with correct context class loader (#421)

Build system and CI changes

• Target Java 11 as the default minimal JVM for benchmarks (#427)
• Update SBT to version 1.9.6 and JMH to version 1.37 (#398)
• Streamline CI to focus on Java LTS versions and the latest JDK only (#394, #426)

Renaissance 0.14.2

This is a maintenance release featuring minor updates to the underlying libraries to achieve the following two goals:

1. Avoid packaging vulnerable versions of the log4j framework. Even though we believe that the Renaissance suite itself (and the way it is used) is not susceptible to the vulnerabilities, the presence of JAR files with vulnerable classes can trigger alarms from various security scanners in some environments. (#366, #367)
2. Improve compatibility with some processor architectures (loongarch64) by using an updated version of the JNA library. (#371)

There have been no changes to the benchmark code.

See also the release announcement.

Renaissance 0.14.1

This release fixes race conditions that were discovered in the finagle-chirper and reactors benchmarks:

• finagle-chirper: avoid using iterator that can be modified during iteration (#356, #357)
• reactors: avoid incorrect usage of partially-initialized objects (#360)

Apart from the bug fixes, there have been no other changes to the benchmark code or the underlying libraries.

See also the release announcement.

Renaissance 0.14.0

This release contains internal cleanups and benchmark dependency updates to improve compatibility and to support execution of benchmarks using a single classloader, which makes the runtime less convoluted when using the benchmark in ahead-of-time compilation scenarios.

The source code of the benchmarks remains unchanged, but several dependencies were updated, potentially affecting the code executed by the benchmarks at runtime. The most significant change is the update of Apache Spark to version 3.2.0 and the associated migration of the apache-spark benchmarks from Scala 2.12 to Scala 2.13.

See also the release announcement.

The actual changes are spread roughly over 15 pull requests, with the most important categorized below.

Workload changes

• Update apache-spark benchmarks to use Spark 3.2.0 and Scala 2.13 (#327)
• Update Scala 2.13 benchmarks to use Scala 2.13.8 (#344)
• Update akka-uct benchmark to use Akka Actors 2.6.18 (#344)
• Update twitter-finagle benchmarks to use Finagle 21.12.0 (#344)
• Update neo4j-analytics benchmark to use Neo4j 4.4.2 (#344)
• Update future-genetic benchmark to use Jenetics 5.2.0 (#342)
• Update rx-scrabble benchmark to use RX Java 1.3.8 (#342)
• Update dotty benchmark to use Scala3 compiler 3.0.2 (#342)
• Update and commonize various dependencies (#342, #344)
  • Use common JNA version (5.10.0) to support more architectures and fix architecture mapping on SAP JVM, resolving issues with db-shootout on non-Intel architectures.
  • Use common Netty version (4.1.72) to avoid bundling several different versions and avoid component version mismatch in standalone (single classloader) benchmark execution.
  • Use common versions of several other components (see PR for details).

Workload fixes

• Expand 'Add-Opens' to prevent use of 'NilJvm' interface in Finagle (#330)

Harness changes

• Add --benchmark-metadata CLI option to override the location benchmark metadata (#341)
• Add --standalone CLI option to disable module loading (#341)
• Add Renaissance-Use-Modules manifest attribute to disable module loading (#341)
• Add plugin for collecting basic memory information (#335)
• Tolerate missing implementation of JMX system info methods (#302)

Build system and CI changes

• Update SBT to version 1.6.1 and JMH to version 1.34 (#344)
• Generate metadata-only jars that allow running benchmarks with a single class loader (#341)
  • Include renaissance-harness package for both Scala 2.12 and Scala 2.13.
• Refactor SBT project definition to speed up compilation and packaging (#336)
  • Compile benchmarks in parallel, avoid bundling duplicated jars, avoid extra filesystem copies.
  • Use JMH directly when generating benchmark JMH wrappers.
• Split testing into multiple workflows, keeping the important platforms in the main workflow (#325)

Renaissance 0.13.0

This release is primarily a JDK compatibility update, introducing support for JDK17. As a result, this version of the Renaissance suite supports JDK8, JDK11, and JDK17 on Linux, MacOS X, and Windows (on x86_64 architectures).

The changes in this release are minimal. The most important one (#297) just brings Scala 2.12 benchmarks to Scala 2.12.15, which now properly supports JDK17 and we just pass these improvements on. No benchmark code has been changed, only project documentation (#294) and parts of the harness responsible for plugin loading (#292, #293).

See also the release announcement.

Renaissance 0.12.0

Apart from a number of internal cleanups, this release focuses on improving compatibility with modern platforms, and on moving away from Scala 2.11 and Spark 2 towards Scala 2.12, Scala 2.13, and Spark 3. Please see the release announcement for a high-level overview and more details.

The changes in this release are spread over more than 30 pull requests, some of which were fairly extensive. Because it is difficult to neatly attribute individual changes to individual PRs, we only list some of the more important PRs for reference.

Workload changes

• Update benchmarks to use Scala 2.12 or 2.13 (#242)
• Clean up apache-spark benchmarks (#248)
• Update scala-dotty to use scalap sources and do hash-based validation (#263)
• Remove executor control and explicit repartitioning (and few other cleanups) (#274)
• Set benchmark-specific thread limits for the Spark local executor (#284)

Harness changes

• Remove old temporary directory API from benchmark context (#246)
• Allow overriding benchmark parameters (#262)

Build system and CI changes

• Switch to git-based versioning and rework the CI to follow suit (#253)
• Enable GitHub Actions (#271)

Renaissance 0.11.0

Workload changes:

• Avoid unnecessary I/O in page-rank (#197)
• Avoid creating response handlers per request in finagle-http (#215)
• Fix RDD caching issues and add checks in Spark workloads (#228)
• Fix resource path issues in finagle-chirper on Windows (#222)
• Fix classpath issues in dotty on Windows (#183)
• Update the Neo4J benchmark (#224)

Harness changes:

• Use single shot time mode in JMH wrappers (#185)
• Hardcode locale to improve output stability (#198)
• Force GC between iterations by default (#196)
• Add hardware counter collection plugin (#195)
• Add compilation time collection plugin (#218)
• Support named benchmark configurations (#191)
• Support benchmark parameters (#182)
• Record benchmark failure in JSON (#178)
• Record revision information in JSON (#177)
• Do not include dummy benchmarks in CLI list (#229)
• Add environment information to JSON (#216)

Renaissance 0.10.0

The following changes are part of the Renaissance release 0.10.0 :

• Introduced the scala-doku benchmark, a Sudoku solver written in Scala (#173 )
• Increased parallelism of finagle-http (#149, #148 )
• finagle-http and finagle-chirper bug fixes (#161, #164 )
• Fixed parallelism of movie-lens and als to 4 executor threads to prevent contention (#145 )
• Provide support for bundling a JMH jar (#120 )
• Support of JDK8 and JDK11 (#73, #127 )
• Support PowerPC infrastructure (#153 )

Renaissance 0.9.0

v0.9.0

Renaissance 0.9.0