Enable reproducible builds on Windows x64 jdk-20

[andrew-m-leonard]

• Test reproducible builds for Windows x64 jdk-19 via Jenkins
• Ensure reproducible across multiple nodes and build directories

[andrew-m-leonard]

jdk-20 built temurin WIndows x64 are identically reproducible on the same build machine, if the binaries are not signed.

Signed JDK binaries are different, much like for MacOS.

They are signed using the Eclipse signing service:
https://github.com/eclipse-cbi/org.eclipse.cbi/blob/dd82d290ba8695cf6b08fb3b1acba042889bec17/webservice/signing/windows/src/main/java/org/eclipse/cbi/webservice/signing/windows/OSSLCodesigner.java#L66

Which uses open source osslsigncode: https://github.com/mtrojnar/osslsigncode

osslsigncode has a -remove_signature option, which we maybe able to use to remove the signature to do a comparison.

[andrew-m-leonard]

Successful reproducible Win64 JDK comparison of Eclipse Adoptium built Temurin JDK binaries can be achieved by:

1. Download, build and install osslsigncode from here: https://github.com/mtrojnar/osslsigncode
2. Create a local self-signed certificate p12, eg.as follows:

openssl genrsa -out test.key -passout pass:testpw 2048                 
openssl req -new -key test.key -out test.csr -subj "/emailAddress=username@me.com, CN=Me, C=US"
openssl req -x509 -sha256 -days 365 -key test.key -in test.csr -out testcert.pem       
openssl pkcs12 -export -out test.p12 -inkey test.key -in testcert.pem -passout pass:testpw

Creates the test.p12 with password testpw
4. Run the following bash script against both unzipped JDK.zip binaries to be compared:

#!/bin/bash

JDK_DIR="$1"
SELF_CERT_FILE="$2"
SELF_CERT_PASS="$3"

if [ ! -d "${JDK_DIR}" ]; then
  echo "$JDK_DIR does not exist"
  exit 1
fi

echo "Expanding the 'modules' Image to remove signatures from within.."
java --add-exports java.base/jdk.internal.jimage=ALL-UNNAMED ExpandJImage "${JDK_DIR}/lib/modules"
rm ${JDK_DIR}/lib/modules

echo "Expanding the 'src.zip' to normalize file permissions"
unzip ${JDK_DIR}/lib/src.zip -d ${JDK_DIR}/lib/src_zip_expanded
rm ${JDK_DIR}/lib/src.zip

echo "Removing all osslsigncode Signatures from ${JDK_DIR}"
FILES=$(find "${JDK_DIR}" -type f -path '*.exe' && find "${JDK_DIR}" -type f -path '*.dll')
for f in $FILES
  do
    echo "Removing signature from $f"
    if osslsigncode remove-signature $f $f.unsigned ; then
        rm $f
        mv $f.unsigned $f
	echo "  ==> Successfully removed signature from $f"
    else
	echo "  ==> $f contains no signature"
    fi
  done

echo "Successfully removed all osslsigncode Signatures from ${JDK_DIR}"

echo "Adding SELF_SIGN osslsigncode Signatures for ${JDK_DIR}"
FILES=$(find "${JDK_DIR}" -type f -path '*.exe' && find "${JDK_DIR}" -type f -path '*.dll')
for f in $FILES
  do
    echo "Signing $f"
    if osslsigncode sign -pkcs12 $SELF_CERT_FILE -pass $SELF_CERT_PASS -n aaa -i "http://aaa" -in $f -out $f.signed ; then
        rm $f
        mv $f.signed $f
        echo "  ==> Successfully signed $f"
    else
        echo "  ==> ERROR: $f failed to be signed!!"
        exit 1
    fi
  done

echo "Successfully SELF_CERT signed all osslsigncode Signatures in ${JDK_DIR}"

echo "Removing all SELF_CERT osslsigncode Signatures from ${JDK_DIR}"
FILES=$(find "${JDK_DIR}" -type f -path '*.exe' && find "${JDK_DIR}" -type f -path '*.dll')
for f in $FILES
  do
    echo "Removing signature from $f"
    if osslsigncode remove-signature $f $f.unsigned ; then
        rm $f
        mv $f.unsigned $f
	echo "  ==> Successfully removed signature from $f"
    else
	echo "  ==> $f contains no signature"
    fi
  done

echo "Successfully removed all SELF_CERT osslsigncode Signatures from ${JDK_DIR}"

5. Compare both unsigned jdk dirs:

diff -r jdk1 jdk2

Example:

anleonar@anleonar-mac Downloads % diff -r jdk1 jdk2
Binary files jdk1/jdk-20+5/bin/server/classes.jsa and jdk2/jdk-20+5/bin/server/classes.jsa differ
Binary files jdk1/jdk-20+5/bin/server/classes_nocoops.jsa and jdk2/jdk-20+5/bin/server/classes_nocoops.jsa differ
diff -r jdk1/jdk-20+5/release jdk2/jdk-20+5/release
15c15
< BUILD_INFO="OS: Windows Server 2012 R2 Version: 6.3"
---
> BUILD_INFO="OS: Windows Server 2022 Version: 10.0"

[andrew-m-leonard]

The fix level of MSVC toolkit is important for reproducibility, it must be identical, even a "fix level" version makes a difference. So for example using 19.27.29112 compared to 19.27.29111, will be non-reproducible.

In an attempt to ensure identical toolkits, I am testing creating an offline toolkit network installation using the instructions here: https://docs.microsoft.com/en-us/visualstudio/install/create-a-network-installation-of-visual-studio?view=vs-2022

[andrew-m-leonard]

To create a installation layout using vs_community.exe, the latest version is required: https://aka.ms/vs/16/release/vs_community.exe
Using the current infrastructure reference one does not work: https://github.com/adoptium/infrastructure/blob/74b81d9f3722edaa9ec978b5e13544b02e930c9d/ansible/playbooks/AdoptOpenJDK_Windows_Playbook/roles/MSVS_2019/tasks/main.yml#L15

The latest version does work with --layout, and that redirects to: https://download.visualstudio.microsoft.com/download/pr/e33403d5-ac1e-4600-b624-d59ccd7b9a13/6a344354ece1743aa117a396f7fc8248957571a24d542ce1b2aaa61dafe99f89/vs_Community.exe
@sxa fyi

[andrew-m-leonard]

Our temurin builds seem to pickup the vcruntime140.dll from the bootjdk being used:

15:22:58  configure: Found vcruntime140.dll at /cygdrive/c/workspace/openjdk-build/jdk-19/bin/vcruntime140.dll using well-known location in Boot JDK

I believe that is because the logic in make/autoconf/toolchain_microsoft.m4 searches:

$VCToolsRedistDir/$vs_target_cpu/microsoft.vc${VS_VERSION_INTERNAL}.crt

which resolves to:
C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Redist\MSVC\14.29.30133\x64\microsoft.vc142.crt

Which does not exist, the following do:

./Microsoft Visual Studio/2019/Community/VC/Redist/MSVC/14.16.27012/onecore/x64/Microsoft.VC141.CRT/vcruntime140.dll
./Microsoft Visual Studio/2019/Community/VC/Redist/MSVC/14.16.27012/onecore/x86/Microsoft.VC141.CRT/vcruntime140.dll
./Microsoft Visual Studio/2019/Community/VC/Redist/MSVC/14.16.27012/x64/Microsoft.VC141.CRT/vcruntime140.dll
./Microsoft Visual Studio/2019/Community/VC/Redist/MSVC/14.16.27012/x86/Microsoft.VC141.CRT/vcruntime140.dll
./Microsoft Visual Studio/2019/Community/VC/Redist/MSVC/14.29.30133/arm64/Microsoft.VC142.CRT/vcruntime140.dll
./Microsoft Visual Studio/2019/Community/VC/Redist/MSVC/14.29.30133/onecore/arm64/Microsoft.VC142.CRT/vcruntime140.dll

[andrew-m-leonard]

It looks like the installation layout only contains the arm64 redist runtime DLLs, even though the layout was created on an x64 box... odd!
Will try re-creating the layout.

[andrew-m-leonard]

So the re-created layout now has the x64 redist DLLs, it seems because I had the redist packages installed when I created the layout, it did not add them to the layout image.

[andrew-m-leonard]

(Edit: Updated for VS2022)

1. First ensure on the machine you're creating the layout that no VS components/programs are installed, including the ReDist programs, otherwise the layout creation seems to pick these up.... I uninstalled these:

• Microsoft C++ <year..year> Redistributeables
• Visual Studio Community 20xx ...
• Visual Studio Installer
• Windows Software Development Kit
• Windows SDK Addon

3. Get the latest VS2022 Installer:

wget https://aka.ms/vs/17/release/vs_community.exe
chmod +x vs_community.exe

4. Created the VS2022 layout using the following:

vs_community.exe --layout VSLayout2022 --arch all --add Microsoft.VisualStudio.Workload.NativeDesktop;includeRecommended;includeOptional --add Microsoft.VisualStudio.Component.VC.Tools.x86.x64 --add Microsoft.VisualStudio.Component.VC.Tools.ARM64 --add Microsoft.VisualStudio.Component.VC.ATL.ARM64 --add Microsoft.VisualStudio.Component.VC.MFC.ARM64

To install from the layout

1. From the VSLayout2022 directory run the following, use "--passive" to see dialogs, or "--quiet" to run silently in background (Note:--wait does not seem to work vs_community.exe process finished immediately having spawned Installer...):
   • Before installing ensure any old c:\Program Files\Microsoft Visual Studio, and c:\Program Files(x86)\Microsoft Visual Studio directories are deleted.

.\vs_community.exe --passive --norestart --wait --noWeb --add Microsoft.VisualStudio.Workload.NativeDesktop;includeRecommended;includeOptional --add Microsoft.VisualStudio.Component.VC.Tools.x86.x64 --add Microsoft.VisualStudio.Component.VC.Tools.ARM64 --add Microsoft.VisualStudio.Component.VC.ATL.ARM64 --add Microsoft.VisualStudio.Component.VC.MFC.ARM64

[andrew-m-leonard]

So the explicit VS2019 x64 redist DLLs were not added to the layout because they were not explicitly selected, instead for some reason the 2017 ones were installed. To select the VS2019 ones, we need to explicitly add the x86.x64 VC Tools using:

--add Microsoft.VisualStudio.Component.VC.Tools.x86.x64

[andrew-m-leonard]

@sxa I have recreated the layout with the necessary redist's, vslayout2019.tar.gz is available on my test machine. I we can recreate the test Jenkins node with this please? When installing we may need to also specify:

--add Microsoft.VisualStudio.Component.VC.Tools.x86.x64

To ensure the 2019 x64 redist's get installed.
thanks

[sxa]

@andrew-m-leonard When I run the above command it starts up the GUI installer and gives me options for "Install while downloading" or "Download all, then install". Is that correct? --noweb is specified on my command line. Will it do the wrong thing if I let it install?

[andrew-m-leonard]

@andrew-m-leonard When I run the above command it starts up the GUI installer and gives me options for "Install while downloading" or "Download all, then install". Is that correct? --noweb is specified on my command line. Will it do the wrong thing if I let it install?

@sxa To run sliently, --quiet --norestart --wait need adding, i'm just testing that myself

[andrew-m-leonard]

@sxa yes, that worked for me.

[sxa]

Thanks - re-running now. Hopefully that won't result it in pulling anything from the web in secret!

[sxa]

https://ci.adoptopenjdk.net/computer/build-azure-win2012r2-x64-3-sxa/ now installed and live.

[andrew-m-leonard]

Successfully reproduced (except for know CDS archive differences) two separately built jdk-20 Win64 JDKs:

1. First JDK built via Jenkins at ci.adopopenjdk.net (https://ci.adoptopenjdk.net/job/build-scripts/job/jobs/job/jdk/job/jdk-windows-x64-temurin/187/)
2. Second JDK built on a manually setup Windows Server2022 machine, using the same VS2019 Layout as (1)

The comparison script above was updated to handle specific differences caused by:

1. Signed signature presence (requires removal, then both signing with a temporary self-signed, which is then removed, this is to ensure the Windows PE file CheckSum value is identical)
2. Unpacking of "modules" Image, so embedded signed artifacts can have the signatures removed
3. Unpacking of src.zip, so that any differences in Group file permissions between build systems is neutralized.

Differences after script processing of jdk1 and jdk2:

anleonar@anleonar-mac Downloads % diff -r jdk1 jdk2
Binary files jdk1/jdk-20+5/bin/server/classes.jsa and jdk2/jdk-20+5/bin/server/classes.jsa differ
Binary files jdk1/jdk-20+5/bin/server/classes_nocoops.jsa and jdk2/jdk-20+5/bin/server/classes_nocoops.jsa differ
diff -r jdk1/jdk-20+5/release jdk2/jdk-20+5/release
15c15
< BUILD_INFO="OS: Windows Server 2012 R2 Version: 6.3"
---
> BUILD_INFO="OS: Windows Server 2022 Version: 10.0"

[andrew-m-leonard]

Full command with path options:

.\vs_community.exe --installPath d:\VS2019 --path shared="d:\VS2019Shared" --path cache="d:\VS2019Cache" --nocache --quiet --norestart --wait --noweb --add Microsoft.VisualStudio.Workload.NativeDesktop;includeRecommended;includeOptional --add Microsoft.VisualStudio.Component.VC.Tools.x86.x64 --add Microsoft.VisualStudio.Component.VC.Tools.ARM64 --add Microsoft.VisualStudio.Component.VC.ATL.ARM64 --add Microsoft.VisualStudio.Component.VC.MFC.ARM64

[andrew-m-leonard]

New repro build of jdk-20+35: https://ci.adoptopenjdk.net/job/build-scripts/job/jobs/job/jdk20/job/jdk20-windows-x64-temurin/25/

[andrew-m-leonard]

jdk-20 builds for Win x64 are identically reproducible, when built with the same MS VS2019 install image version.