Pinned Offline Buildpack chooses non included versions

[benjaminguttmann-avtq]

Hi there,

I am currently seeing a strange behavior when using offline buildpacks, where the version used is not included in the offline package and therefore fails to stage the app.

Steps to reproduce the issue:

git clone https://github.com/cloudfoundry/java-buildpack.git
cd java-buildpack
git checkout v4.37
bundle exec rake clean package OFFLINE=true PINNED=true STACK=cflinuxfs3 VERSION="v4.37"
mv build/java-buildpack-offline-v4.37.zip ./java-buildpack-offline-v4.37.zip
git checkout v4.39
bundle exec rake clean package OFFLINE=true PINNED=true STACK=cflinuxfs3 VERSION="v4.39"
cf login
cf create-buildpack java-offline-buildpack-v4-39 ./build/java-buildpack-offline-v4.39.zip 1
cf create-buildpack java-offline-buildpack-v4-37 ./build/java-buildpack-offline-v4.37.zip 1
cf push spring-music -f manifest.yml -b java-offline-buildpack-v4-37

Manifest looks like this:

---
applications:
- name: spring-music
  instances: 1
  memory: 1G
  env:
    JBP_CONFIG_OPEN_JDK_JRE: '{ jre: { version: 1.8.+}}'

We are using spring-music to test the behavior.

In our env we can see that the staging fails due to this error:

   2021-07-14T08:27:00.95+0000 [STG/0] ERR [Buildpack]                      ERROR Finalize failed with exception #<RuntimeError: Unable to find cached file for https://java-buildpack.cloudfoundry.org/openjdk/bionic/x86_64/bellsoft-jre8u292%2B10-linux-amd64.tar.gz>
   2021-07-14T08:27:00.95+0000 [STG/0] ERR Unable to find cached file for https://java-buildpack.cloudfoundry.org/openjdk/bionic/x86_64/bellsoft-jre8u292%2B10-linux-amd64.tar.gz
   2021-07-14T08:27:00.96+0000 [STG/0] ERR Failed to compile droplet: Failed to run finalize script: exit status 1

We would expect the buildpack to choose the included version 1.8.0_282.

Beside that, for me v4.37 pins version to 11.0.11_9 but according to the release notes it should be 11.0.10_9.

[dmikusa]

Thanks, I'll take a look at this and see what is going on here.

[dmikusa]

Beside that, for me v4.37 pins version to 11.0.11_9 but according to the release notes it should be 11.0.10_9.

This while not what you expect is what the code is supposed to do.

java-buildpack/rakelib/dependency_cache_task.rb

Lines 128 to 144 in fd50c13

	if repository_configuration?(configuration)
	configuration['component_id'] = component_id
	configuration['sub_component_id'] = sub_component_id if sub_component_id
	if component_id == 'open_jdk_jre' && sub_component_id == 'jre'
	c1 = configuration.clone
	c1['version'] = '11.+'
	configurations << c1
	end
	if component_id == 'open_jdk_jre' && sub_component_id == 'jre'
	c1 = configuration.clone
	c1['version'] = '16.+'
	configurations << c1
	end

When you pin, the code has to do some special handling for JREs because it's the only component where we bundle multiple versions. This is kind of a hack, but instead of plugging in a specific version, it'll insert records for 11.+ and 16.+ which means "get the latest version at the time you run this". That is why you get a more recent version of Java 11 and 16 when you package 4.37 after the fact.

I would agree that this is not what's intended and that you'd likely expect to get the time-locked versions. Unfortunately, those are not in the commit (there's only one config/open_jdk_jre.yml file and it's committed with the Java 8 version, so retrieving them is not really an option.

An easy fix, I could add env variables where you could override the Java 11 & 16 versions at package time? So it would default to 11.+ and 16.+ (and emit a warning so it's more clear what is happening) but you could set JRE_XX_VERSION=11.0.10_9 to force a specific version. Would that be sufficient for your needs?

Actually fixing this would only be possible in versions going forward, as we'd need to save the Java 11 & 16 versions in the commit somehow & we cannot go back and change past commits.

Still looking into the other half of this issue, just wanted to mention this first so you could provide feedback. Thanks

[benjaminguttmann-avtq]

Hi @dmikusa-pivotal thanks for the feedback, we will discuss that internally and will let you know afterwards. :)

[dmikusa]

OK, so in regards to the other issue:

   2021-07-14T08:27:00.95+0000 [STG/0] ERR [Buildpack]                      ERROR Finalize failed with exception #<RuntimeError: Unable to find cached file for https://java-buildpack.cloudfoundry.org/openjdk/bionic/x86_64/bellsoft-jre8u292%2B10-linux-amd64.tar.gz>
   2021-07-14T08:27:00.95+0000 [STG/0] ERR Unable to find cached file for https://java-buildpack.cloudfoundry.org/openjdk/bionic/x86_64/bellsoft-jre8u292%2B10-linux-amd64.tar.gz
   2021-07-14T08:27:00.96+0000 [STG/0] ERR Failed to compile droplet: Failed to run finalize script: exit status 1

This is happening for a similar reason. When you package the buildpack, it will go and download resources and include them with the buildpack. One of the resources that it is downloading is the index.yml file. This file is not versioned so it will download the most recent index.yml file from the time you package the buildpack not the time when the release was cut.

The problem this causes is that new versions of a resource, like the JRE, may have been added to index.yml since the buildpack was released. When you specify 1.8.+ as your version, it's saying take the most recent version, which it picks from index.yml and use that.

Because you packaged your buildpack later, after a new JRE has been released, your cached index.yml has a newer version of the JRE. This isn't the version packaged, because the packaged version of Java 8 is set in the config/ file that's pinned prior to cutting the release. When you tell it to pick the most recent, it does, you just don't have the most recent version in your buildpack.

You can get around this by not setting the version for Java 8, because that's the default, or set the exact version like 1.8.0_282. I do not think this will impact Java 11 or Java 16, because the other issue you noticed causes the packaging tools to always download the most recent, so the most recent in index.yml for those versions will always be what is packaged into the buildpack.

I'm not sure there's an easy fix for this. In theory you could modify the index.yml so that only pinned versions are present, but I'm not sure how practical that would be, or the level of effort to do so.

Can you expand on the impact of this particular issue to you? Would it work to simply not specify the version like that and rely on the default or set the exact version you want (probably not the best idea as you'd have to change this in the app when the buildpack updates)? Just trying to understand the impact of this issue. Thanks!

[benjaminguttmann-avtq]

Hi @dmikusa-pivotal thanks for that information. That really helps to understand the overall issue. I am wondering if we on our side should just check the timing when we prepare offline buildpacks instead of re-adjusting the whole logic of the buildpack.

[dmikusa]

That's one way you could handle it. You would need to package buildpacks around the time of the release. Then persist the packages you create somewhere since you can't easily recreate them at the moment.

I do feel like we can do better here. The way things are now, it's not possible to reproduce a package of a particular buildpack version. I think short-term, I can document these caveats in our packaging docs on the README. Long term, I think we should have a solution such that it does not depend on when you run a command and that you can get a reasonably consistent (different timestamps, but the same versions) package.

I'll work on getting the README updated this week and leave this issue open so that we can circle back and make some changes to the package tools at a later date. If you have any use-case details you can share, feel free. That helps with scheduling the work.

Thanks!

[dmikusa]

FYI, update to README -> https://github.com/cloudfoundry/java-buildpack#packaging-caveats

[metron2]

Could we provide a list of major java versions instead of hard coding them?

[dmikusa]

So perfect reproducible builds of previously released Java buildpack versions are never going to really work. There are a number of non-fixable problems (registries that have gone away, versions removed from registries, and the fact that older versions don't pin non-default JRE versions). Despite that, I have made some changes which will improve the behavior in #957. Please check out the PR for details, I've outlined the changes there & in the README update part of the PR.

If you are interested, please take a look and add comments to the PR. We'll likely merge the PR in about a week unless there are objections/active discussions.

[dmikusa]

PR #957 merged.