reproducible builds for Spoon

[monperrus]

concept: the creation of Jar files should be deterministic, i.e. we should have a reproducible build, see https://reproducible-builds.org/

maybe it is already, maybe it is not, to be checked.

then, we have different solutions:

• https://maven.apache.org/guides/mini/guide-reproducible-builds.html
• https://zlika.github.io/reproducible-build-maven-plugin/

Tasks:

• build is reproducible (manually)
• check reproducibility in CI (see chore: Check reproducible state in CI #4653)
• create buildspec file, see reproducible builds for Spoon #4151 (comment)
• periodically rebuild past releases (handled by https://github.com/jvm-repo-rebuild/reproducible-central)

related to #3867

[monperrus]

See also thread "Reproducible builds on Java", https://lists.reproducible-builds.org/pipermail/rb-general/2021-September/002374.html

[I-Al-Istannen]

diff --git a/spoon-pom/pom.xml b/spoon-pom/pom.xml
index 7a352b09..eef300ae 100644
--- a/spoon-pom/pom.xml
+++ b/spoon-pom/pom.xml
@@ -24,6 +24,7 @@
         <java.test.version>1.8</java.test.version>
         <runtime.log>target/velocity.log</runtime.log>
         <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
+        <project.build.outputTimestamp>1</project.build.outputTimestamp>
     </properties>
 
     <dependencies>
@@ -108,6 +109,11 @@
     <build>
         <defaultGoal>clean install</defaultGoal>
         <plugins>
+            <plugin>
+                <groupId>org.apache.maven.plugins</groupId>
+                <artifactId>maven-jar-plugin</artifactId>
+                <version>3.2.0</version>
+            </plugin>
             <plugin>
                 <groupId>org.apache.maven.plugins</groupId>
                 <artifactId>maven-surefire-plugin</artifactId>

Yields builds that consistently have the same hash for me. So at least in the core that seems to work (after updating the jar plugin).

I haven't checked the other submodules.

[monperrus]

oh nice! would you make a PR?

[monperrus]

Thanks a lot @I-Al-Istannen for #4152!

I would say that the next step is to check reproducibility in CI, by packaging twice.

Thoughts?

[seintur]

Hi Martin @monperrus

Great idea to introduce reproductible builds for Spoon. +1

Just a minor notification about one technical issue that arises on Inria CI with project.build.outputTimestamp set to 1. See for instance [1] where you can search for

java.time.DateTimeException: Invalid value for MonthOfYear (valid values 1 - 12): 0

It seems to be a known bug with JDK 9. See for instance [2] and other references can be found on Google. Moving to a different JDK, 10 or later, seems to fix the issue. If someone can commit interest and time in upgrading the instance of Jenkins and/or pinging the Inria admins to do so...

Cheers,
Lionel.

[1] https://ci.inria.fr/sos/job/juliac/531/consoleText
[2] bazelbuild/bazel#6173

[seintur]

Many thanks to the one who configured the juliac job on Inria CI with JDK 11. This indeed solved the issue. I should have try it myself, but the notice "JDK11 (MUST BE PREINSTALLED ON NODE)" somewhat confused me and I dare not try, although I should have.

[monperrus]

you're welcome Lionel :) --Martin

[slarse]

Many thanks to the one who configured the juliac job on Inria CI with JDK 11. This indeed solved the issue. I should have try it myself, but the notice "JDK11 (MUST BE PREINSTALLED ON NODE)" somewhat confused me and I dare not try, although I should have.

@seintur I put that note there to avoid confusion, hehe. Basically, selecting that JDK only sets it as the default, but if JDK 11 is not installed it will just cause the build to fail. Which might be confusing as the other selections actually install a JDK.

I've edited the note to be less scary: note: JDK 11 must preinstalled on agent, this only sets it as the default

[monperrus]

At some point, we'll also provide secure signing with Linux Foundation's http://sigstore.dev

See @bobcallaway's Java component (yet to be documented): https://github.com/sigstore/sigstore-maven-plugin

[monperrus]

Eventually, Spoon will be listed on the reference table of reproducible maven artifacts: https://github.com/jvm-repo-rebuild/reproducible-central

For this, we need to contribute with a buildspec file, see https://github.com/jvm-repo-rebuild/reproducible-central/blob/master/BUILDSPEC.md#writing-a-new-buildspec

cc/ @hboutemy

[I-Al-Istannen]

@monperrus I tried that out and generated a buildspec, but we have a fundamental problem. Your tagged release commit in git has the version 10.0.0-SNAPSHOT, while the maven central release omits the -SNAPSHOT. This causes the maven artifact plugins compare to fail:

[INFO] Reference buildinfo file not found: it will be generated from downloaded reference artifacts
[WARNING] Reference artifact not found fr.inria.gforge.spoon:spoon-core:pom:10.0.0-SNAPSHOT
[WARNING] Reference artifact not found fr.inria.gforge.spoon:spoon-core:jar:10.0.0-SNAPSHOT
[WARNING] Reference artifact not found fr.inria.gforge.spoon:spoon-core:jar:jar-with-dependencies:10.0.0-SNAPSHOT

and results in a total KO.

Manually changing the version to 10.0.0 in the tagged commit in git results in:

[WARNING] Reproducible Build output summary: 2 files ok, 1 different
rebuild from wip/spoon-core-10.0.0.buildspec
  results in wip/spoon-core-10.0.0.buildinfo
compared to Central Repository wip/spoon-core-10.0.0.buildcompare:
    ok=2
    okFiles="spoon-core-10.0.0.pom spoon-core-10.0.0.jar"
    ko=1
    koFiles="spoon-core-10.0.0-jar-with-dependencies.jar"
    check .buildspec jdk=11 vs reference java.version=11 (from MANIFEST.MF Build-Jdk-Spec)
    check .buildspec newline=lf vs reference os.name=Unix (from pom.properties newline) (newline should be crlf if os.name is Windows, lf instead)

Which is a lot better. I am not sure if we want to have the jar with deps reproducible, but it sounds like @algomaster99 fixed that in #4614 anyways.

So moving forward would likely require you to use the actual version without SNAPSHOT for the release commit.

[hboutemy]

yes, fixing your release Git tag looks like a good idea
waiting for that, I suppose you can add mvn versions:set -D... in your buildspec: not ideal, but can do the job in the interim time.
notice that defining the project.build.outputTimestamp property in your pom.xml instead of inheriting a fixed value, and updating it during the release process would be a nice improvement also

last: did you diffoscope spoon-core-10.0.0-jar-with-dependencies.jar to see what are the non-reproducible bits?

[I-Al-Istannen]

waiting for that, I suppose you can add mvn versions:set -D... in your buildspec: not ideal, but can do the job in the interim time.

I am not sure how that would work. I tried command="mvn versions:set -DnewVersion=${version} clean package -DskipDepClean -DskipTests -Dmaven.javadoc.skip -Dgpg.skip" but as that is a single invocation, maven still seems to use the old version for package. The output is:

[INFO] --- versions-maven-plugin:2.10.0:set (default-cli) @ spoon-core ---
[INFO] Searching for local aggregator root...
[INFO] Local aggregation root: /var/maven/app
[INFO] Processing change of fr.inria.gforge.spoon:spoon-core:10.0.0-SNAPSHOT -> 10.0.0
[INFO] Processing fr.inria.gforge.spoon:spoon-core
[INFO]     Updating project fr.inria.gforge.spoon:spoon-core
[INFO]         from version 10.0.0-SNAPSHOT to 10.0.0
[INFO] 
[INFO] 
[INFO] ------------------< fr.inria.gforge.spoon:spoon-core >------------------
[INFO] Building Spoon Core 10.0.0-SNAPSHOT
[INFO] --------------------------------[ jar ]---------------------------------
[INFO] 

The buildspec file also does not seem to offer a way to massage the repository outside of and before the maven invocation, so I can't do this sequentially. I also tried using && in the hope that the command was passed as-is to the docker image, but that turned out to not be the case.

[hboutemy]

ok, no luck, it will have to be fixed for next releases :)

on spoon-core-10.0.0-jar-with-dependencies.jar, it seems it's basically the version of maven-assembly-plugin to upgrade

[I-Al-Istannen]

ok, no luck, it will have to be fixed for next releases :)

I guess you don't allow manual command fiddling to ensure projects are actually easily reproducible, but it bites us here :P

on spoon-core-10.0.0-jar-with-dependencies.jar, it seems it's basically the version of maven-assembly-plugin to upgrade

Yea, that is algomaster's conclusion in the PR I linked above as well. Should be doable :)

[hboutemy]

I guess you don't allow manual command fiddling

yes, the intent being to keep the rebuild script as simple as possible to maintain, not really forcing to have simple reproducibility

[I-Al-Istannen]

If #4652 is merged and Martin fixes the artifact name in the upcoming release, spoon 10.0.1 should be reproducible if I haven't missed something.

Once we have #4652, violating that would also break the CI. Not sure it is worth the runtime, but it isn't really heavy as it skips depclean and tests.

[MartinWitt]

Thanks to @I-Al-Istannen work, we now have check in the CI for each PR if the build is reproducible.

@monperrus What do you mean exactly with "periodically rebuild past builds"? Do you want to check if old builds are still reproducible?

[monperrus]

What do you mean exactly with "periodically rebuild past builds"?

For the last N releases on central, we take the associated commit, we build it, and we check that we get the same SHA256 as the one on central.

[I-Al-Istannen]

@monperrus The 10.1.0 release still has a -SNAPSHOT suffix in git, blocking progress in this issue. I didn't quite understand if or where you want to tackle that. If you don't intend to fix the git release commits, we'll need to reconsider adding spoon to reproducible-maven.

Automatically releasing a commit if the version does not end in -SNAPSHOT sounds a bit more logical to me and would fix the problem. We'd work in 10.2.0-SNAPSHOT and once it is ready it will become 10.2.0 and is released on central. The next commit would then need to change the version to 10.3.0-SNAPSHOT or whatever. This also allows us to later change our minds and change the version number (minor/patch/major) if any later feature changes semantics. Maybe I am missing something though.

[monperrus]

Thanks @I-Al-Istannen for moving forward on this important topic.

The 10.1.0 release still has a -SNAPSHOT suffix in git,

Yes, per our CD process, the commit sha1 is the reference.

blocking progress in this issue.

Why is that blocking? Which tool is complaining?

[I-Al-Istannen]

Why is that blocking? Which tool is complaining?

Here: #4151 (comment)

The reproducible-central skript works by checking out the git commit corresponding to a maven release. It then builds the project locally on that commit and compares the results with the artifact found in maven central. This does not work for spoon, as the remote pom on maven central and the local pom are different: The local pom has a version of 10.1.0-SNAPSHOT and the remote pom has 10.1.0. Therefore, the script does not find any reference release on maven central.

Even if it did though, the comparison would fail. The pom and jar files can not match as the pom.xml differs.

[monperrus]

I see, maven-artifact-plugin complains. We cannot claim a bug there.

We have to change the process.

In the meantime, I propose to manually release 10.1.1 (hotpatch) with an appropriate commit to make progress here.

WDYT?

[I-Al-Istannen]

Yea, that plugin works as intended. Our workflow is just not reproducible -- there is no commit for the release on maven as the version differs.

In the meantime, I propose to manually release 10.1.1 (hotpatch) with an appropriate commit to make progress here.

That would work fine, but it is also not really urgent. If you want to make a hotpatch feel free and I will update the reproducible-central PR. I am also fine with waiting until a new process has been found and settled on :)

[monperrus]

10.1.1 is released, we can proceed from there

This work is very timely because we have a research project about to start on reproducible builds with openings as research assistant, research engineer and PhD student.

Would you be interested?

[slarse]

The Maven release plugin automates creating the release commit and the "next dev cycle" commit. I say we just use that and include those commits in the repo, as is done in Sorald's release script.

[I-Al-Istannen]

As an update, the PR to reproducible-maven was merged and spoon-core 10.1.1 is now listed in the readme: https://github.com/jvm-repo-rebuild/reproducible-central#fr.inria.gforge.spoon:spoon-core.

We could add a badge to the readme if we want or just leave it as is :)

With this we can probably tick of the third point in the offpost.

---

[![Reproducible Builds](https://img.shields.io/badge/Reproducible_Builds-ok-success?labelColor=1e5b96)](https://github.com/jvm-repo-rebuild/reproducible-central#fr.inria.gforge.spoon:spoon-core)

[monperrus]

Hurray, we can close the issue then. Thanks a lot @I-Al-Istannen

[monperrus]

FTR, added a step to update the buildspec file on https://github.com/jvm-repo-rebuild/reproducible-central for each release

see #3890 (comment)

[monperrus]

If you like this issue, I'd recommend to subscribe to the rb-general mailing list https://lists.reproducible-builds.org/listinfo/rb-general

[algomaster99]

The status of the reproducible build of spoon releases is published here. Any idea why they do not check for beta releases or have we disabled checks for them intentionally?