Cannot upgrade Google HTTP libraries

[chanseokoh]

#3415 downgraded Google HTTP libraries to resolve #3058 and #3409. However, we cannot delay upgrading libraries indefinitely.

Moreover, even if we downgraded them, Gradle and Maven can still pull in newer versions depending on how a project is set up. Examples:

• Premature end of chunk coded message body after upgrading to 3.1.3 #3409 (comment)
• Regression HTTPS verification javax.net.ssl.SSLPeerUnverifiedException on docker layer pull #3058 (comment)
• "Premature end of chunk coded message body" and TruncatedChunkException when calling HttpResponse.disconnect() googleapis/google-http-java-client#1409 (comment)

In the last case, we didn't release jib-core with the problematic Google HTTP library version, but Quarkus picked up the latest version; I think jib-core will be more susceptible to version overriding than Jib plugins.

These issues need to be investigated further before we can upgrade the libraries.

[suztomo]

I assume the challenge is that we cannot reproduce the problem. Is this correct?

[chanseokoh]

For #3409, I think I know what exactly is the problem, but I have trouble reproducing this. Filed a fix (googleapis/google-http-java-client#1427) against the library.

For #3058, yeah, I haven't been able to reproduce it. And the root cause is still unknown. It may or may not be a fault of Amazon.

[famod]

In Quarkus, we just ran into a dependency "conflict" because of this: quarkusio/quarkus#20507

I anyone looking at getting the remaining issues fixed upstream? I understand it's hard to reproduce?

[chanseokoh]

#3409, which caused an outright no-go friction to many people with the "premature end of chunk" error, has been fixed upstream. New versions of the Google HTTP libraries with the fix are live now.

The other issue, #3058, is a very subtle one. It seems to happen to only some small group of people specifically using AWS ECR. Also, I've seen elsewhere that people run into the failure out of the Jib context as well, so it seems like a general problem. It's unclear if it's the fault of AWS or the Apache HttpClient. It's hard to reproduce, and I asked if anyone seeing the failure can contribute their time to debug it with us, but so far, no one responded.

Therefore, for Quarkus, IMO it'd be acceptable to upgrade the Google HTTP libraries to the latest once new versions are released.

[famod]

Thanks @chanseokoh, just one question:

New versions of the Google HTTP libraries with the fix are live now.

Are you sure about that? 1.40.0 does not seem to include googleapis/google-http-java-client#1427?

[chanseokoh]

Sorry, I was dumb. The fix was merged on Sep 23, while 1.40.0 was released on Aug 26. We still have to wait for a new release. Sorry for the misinformation.

[famod]

@chanseokoh alright, no worries!

Apart from those known issues in newer Google HTTP libs, would anything break if using latest jib-core with 1.38.0 of Google HTTP?

[chanseokoh]

@famod apart from those listed here (one of which is critical), I'm not aware of anything else. But generally speaking, there's always a chance of behavior changes whenever you upgrade. But in general, definitely one needs to keep libraries up-to-date.

[chanseokoh]

@famod Google HTTP Client 1.40.1 is released.

[elefeint]

There were 3 issues that blocked upgrade:

1. premature end of chunk coded message body -- fixed in fix: revert the order of stream closure and disconnect googleapis/google-http-java-client#1427
2. certificate issue with docker containers stored on AWS S3
   org.apache.httpcomponents:httpclient:4.5.7 breaks fetching S3 objects with consecutive slashes in the key aws/aws-sdk-java#1919
   SSLPeerUnverifiedException on S3 actions aws/aws-sdk-java-v2#1786

In any case, Jib can't hold off upgrading core HTTP libraries indefinitely. We should upgrade, and then help users work with AWS support if necessary.

[emmileaf]

jib-core 0.23.0, jib-maven-plugin 3.3.1, jib-gradle-plugin 3.3.1, and jib-cli 0.12.0 have been released with the HTTP libraries upgraded.