Stripping AWS Mail component with TrueZIP blocks packaging for docker images #5274
Comments
|
Hey @poikilotherm , the reason we took this approach is due to dependency compatibility issues between the AWS SDK and Jackson/Glassfish. See around this post in 3921 for more info on that problem. The solution we went with was not the most elegant, but functioned well for our standard deployment strategy. I am unaware at this point why we didn't go with pointing to the individual components, that seems like a good strategy to try. If you do try to fix this weird part of our build please make sure it still operates for our norm of keeping all the dependencies in the war. Thanks! |
|
Will have a look at the Jackson problems tomorrow... Thx for pointing that out. Using the S3-only SDK seems to be alright, that's the only AWS stuff in use currently. This reduced the WAR size from 146MB to 128MB already, which seems good, right? 😉 I checked all of the involved jar archives, as far as I can tell, there is no javamail related stuff in there, thus the mail problems should be avoided thoroughly. |
|
you wrote the lines following Line 73 in c2dec15 As far as I can see from the MVN AWS SDK, MVN AWS SDK Complete Bundle, MVN AWS SDK S3, There are compile scope deps against Jackson 2.6 in MVN AWS SDK Core. But since the release of .172, nowadays v.11.442, there have been only marginal changes in versioning and not for Jackson. So these "unique references" never were absent...? What was the exact problem with Jackson you mentioned earlier? The included Jackson Databinding in Glassfish 4.1 is v2.3.2 (lookup from the Included in the resulting dataverse WAR are the following Jackson JARs: I'm puzzled... |
|
Maybe I have a hint about why you ran into problems and solved those when using the bundle: https://aws.amazon.com/de/blogs/developer/java-sdk-bundle/ To sum up that blog article: within the bundle, Jackson and other third party licenses are re-packaged under |
|
Using the Maven Enforcer plugin and its dependency convergence rule leads to the following errors (among ALOT of others): And as we saw above, the v2.5.3 wins. This will take some work to sort out, especially if done for all defined deps (which is best practise anyway...). |
|
Hey @poikilotherm , sorry for the lack of info earlier, I had forgotten the details and some of them were lost to the sands of time and intern involvement. The movement of the 3rd party licenses to I'll bring it up during our team's tech hours today, there is definitely debt in cleaning up our dependencies in general and this one specifically. |
|
Most likely this is a matter of adding excludes to the dependency definitions or use |
…ter. When AWS S3 storage support was introduced back in 2017 in IQSS#3921, the team experienced problems with the bundled Jackson library of Glassfish (2.3 instead of 2.6 minimum). By switching to the complete bundle, the bundled Jackson library was used and problems where avoided. This lead to a bigger WAR than necessary (~20 MB) and made a workaround necessary to remove some AWS specific `javamail.providers` to avoid email problems via WAR file manipulation. This commit: * removes the WAR file hacking * makes use of the S3 SDK part only, reducing the WAR size * enables proper <dependencyManagement> for the sake of avoiding dependency convergence problems. People unaware of direct and transitive dependencies and how to manage them are kindly requested to have a look at the Maven docs and tutorials: * https://maven.apache.org/guides/introduction/introduction-to-dependency-mechanism.html * https://www.davidjhay.com/maven-dependency-management * https://maven.apache.org/enforcer/maven-enforcer-plugin/index.html
Needed because the transitive dependency from AWS has been used before. Thus this relates to IQSS#5274. As best practise, code should not rely on those deps but add a direct dependency. For the sake of working on IQSS#4260 (Java EE 8) at some point in the future, please remember to refactor the code using Jackson (and Gson) and remove it in favor of Java EE 8 native JSON-B and JSON-P support.
Needed because the transitive dependency from AWS has been used before. Thus this relates to IQSS#5274. As best practise, code should not rely on those deps but add a direct dependency. For the sake of working on IQSS#4260 (Java EE 8) at some point in the future, please remember to refactor the code using Jackson (and Gson) and remove it in favor of Java EE 8 native JSON-B and JSON-P support.
…ter. When AWS S3 storage support was introduced back in 2017 in IQSS#3921, the team experienced problems with the bundled Jackson library of Glassfish (2.3 instead of 2.6 minimum). By switching to the complete bundle, the bundled Jackson library was used and problems where avoided. This lead to a bigger WAR than necessary (~20 MB) and made a workaround necessary to remove some AWS specific `javamail.providers` to avoid email problems via WAR file manipulation. This commit: * removes the WAR file hacking * makes use of the S3 SDK part only, reducing the WAR size * enables proper <dependencyManagement> for the sake of avoiding dependency convergence problems. People unaware of direct and transitive dependencies and how to manage them are kindly requested to have a look at the Maven docs and tutorials: * https://maven.apache.org/guides/introduction/introduction-to-dependency-mechanism.html * https://www.davidjhay.com/maven-dependency-management * https://maven.apache.org/enforcer/maven-enforcer-plugin/index.html
Needed because the transitive dependency from AWS has been used before. Thus this relates to IQSS#5274. As best practise, code should not rely on those deps but add a direct dependency. For the sake of working on IQSS#4260 (Java EE 8) at some point in the future, please remember to refactor the code using Jackson (and Gson) and remove it in favor of Java EE 8 native JSON-B and JSON-P support.
…ter. When AWS S3 storage support was introduced back in 2017 in IQSS#3921, the team experienced problems with the bundled Jackson library of Glassfish (2.3 instead of 2.6 minimum). By switching to the complete bundle, the bundled Jackson library was used and problems where avoided. This lead to a bigger WAR than necessary (~20 MB) and made a workaround necessary to remove some AWS specific `javamail.providers` to avoid email problems via WAR file manipulation. This commit: * removes the WAR file hacking * makes use of the S3 SDK part only, reducing the WAR size * enables proper <dependencyManagement> for the sake of avoiding dependency convergence problems. People unaware of direct and transitive dependencies and how to manage them are kindly requested to have a look at the Maven docs and tutorials: * https://maven.apache.org/guides/introduction/introduction-to-dependency-mechanism.html * https://www.davidjhay.com/maven-dependency-management * https://maven.apache.org/enforcer/maven-enforcer-plugin/index.html
Needed because the transitive dependency from AWS has been used before. Thus this relates to IQSS#5274. As best practise, code should not rely on those deps but add a direct dependency. For the sake of working on IQSS#4260 (Java EE 8) at some point in the future, please remember to refactor the code using Jackson (and Gson) and remove it in favor of Java EE 8 native JSON-B and JSON-P support.
…ter. When AWS S3 storage support was introduced back in 2017 in IQSS#3921, the team experienced problems with the bundled Jackson library of Glassfish (2.3 instead of 2.6 minimum). By switching to the complete bundle, the bundled Jackson library was used and problems where avoided. This lead to a bigger WAR than necessary (~20 MB) and made a workaround necessary to remove some AWS specific `javamail.providers` to avoid email problems via WAR file manipulation. This commit: * removes the WAR file hacking * makes use of the S3 SDK part only, reducing the WAR size * enables proper <dependencyManagement> for the sake of avoiding dependency convergence problems. People unaware of direct and transitive dependencies and how to manage them are kindly requested to have a look at the Maven docs and tutorials: * https://maven.apache.org/guides/introduction/introduction-to-dependency-mechanism.html * https://www.davidjhay.com/maven-dependency-management * https://maven.apache.org/enforcer/maven-enforcer-plugin/index.html
Needed because the transitive dependency from AWS has been used before. Thus this relates to IQSS#5274. As best practise, code should not rely on those deps but add a direct dependency. For the sake of working on IQSS#4260 (Java EE 8) at some point in the future, please remember to refactor the code using Jackson (and Gson) and remove it in favor of Java EE 8 native JSON-B and JSON-P support.
|
@poikilotherm @pdurbin The missing method stack trace has to do with Jackson: [2018-11-16T14:09:07.433-0500] [glassfish 4.1] [INFO] [] [edu.harvard.iq.dataverse.metrics.MetricsServiceBean] [tid: _ThreadID=54 _ThreadName=jk-connector(4)] [timeMillis: 1542395347433] [levelValue: 800] [[ [2018-11-16T14:09:21.844-0500] [glassfish 4.1] [WARNING] [] [javax.enterprise.resource.webcontainer.jsf.lifecycle] [tid: _ThreadID=52 ThreadName=jk-connector(2)] [timeMillis: 1542395361844] [levelValue: 900] [[ |
Needed because the transitive dependency from AWS has been used before. Thus this relates to IQSS#5274. As best practise, code should not rely on those deps but add a direct dependency. For the sake of working on IQSS#4260 (Java EE 8) at some point in the future, please remember to refactor the code using Jackson (and Gson) and remove it in favor of Java EE 8 native JSON-B and JSON-P support.
|
@poikilotherm sure, I moved it to Community Dev and assigned it to you. If there's a better place, please let me know. Thanks! |
…ter. When AWS S3 storage support was introduced back in 2017 in IQSS#3921, the team experienced problems with the bundled Jackson library of Glassfish (2.3 instead of 2.6 minimum). By switching to the complete bundle, the bundled Jackson library was used and problems where avoided. This lead to a bigger WAR than necessary (~20 MB) and made a workaround necessary to remove some AWS specific `javamail.providers` to avoid email problems via WAR file manipulation. This commit: * removes the WAR file hacking * makes use of the S3 SDK part only, reducing the WAR size * enables proper <dependencyManagement> for the sake of avoiding dependency convergence problems. People unaware of direct and transitive dependencies and how to manage them are kindly requested to have a look at the Maven docs and tutorials: * https://maven.apache.org/guides/introduction/introduction-to-dependency-mechanism.html * https://www.davidjhay.com/maven-dependency-management * https://maven.apache.org/enforcer/maven-enforcer-plugin/index.html
Needed because the transitive dependency from AWS has been used before. Thus this relates to IQSS#5274. As best practise, code should not rely on those deps but add a direct dependency. For the sake of working on IQSS#4260 (Java EE 8) at some point in the future, please remember to refactor the code using Jackson (and Gson) and remove it in favor of Java EE 8 native JSON-B and JSON-P support.
…ter. This prepares the creation of the container Maven profile, as the TrueZIP hacking is not usable with that.
Needed because the transitive dependency from AWS has been used before. Thus this relates to IQSS#5274. As best practise, code should not rely on those deps but add a direct dependency. For the sake of working on IQSS#4260 (Java EE 8) at some point in the future, please remember to refactor the code using Jackson (and Gson) and remove it in favor of Java EE 8 native JSON-B and JSON-P support.
…ter. This prepares the creation of the container Maven profile, as the TrueZIP hacking is not usable with that.
…ter. This prepares the creation of the container Maven profile, as the TrueZIP hacking is not usable with that.
…ter. This prepares the creation of the container Maven profile, as the TrueZIP hacking is not usable with that.
…ter. This prepares the creation of the container Maven profile, as the TrueZIP hacking is not usable with that.
Using only the S3 part of AWS SDK pom.xml gets lighter and speeds up builds. (And hopefully deployment times, too)
Dear devs,
Dear @bsilverstein as the one who wrote the code for #3921,
while working on #4172, I am crafting a new Maven profile to generate smallest possible Docker images based on the payara 5 server-full image using https://dmp.fabric8.io.
As I want to make things really small, I don't want to bundle the dataverse dependencies within the WAR file, but create a small WAR without all deps, containing only classes and ressources. The deps get moved in by copying them the
/libof the Glassfish/Payara domain (so they are kind of "provided" scoped in Maven terminology). (For those interested in Docker: this should provide us with smaller layer changes and smaller hub transfers!)The stripping of the mailing stuff from the AWS SDK makes this hard to achieve in two ways: a) I would to fiddle around with this, too and b) the TrueZIP plugin simply fails as I don't build the WAR with deps inside and it is not easy to skip it in a profile without making a mess in the
pom.xml...Back in #3921 where all of this was introduced, there has been a pointer to the AWS dev docs about using individual components.
Did you guys ever try to get that rolling? If not: maybe this would be the perfect time to do so now... Willing to contribute on this. 😄
This is related to a question I made on IRC (see logs).
EDIT:
@pdurbin suggested some clarification on IRC.
The
pom.xmlplugin section I would like to see removed:And something that needs to be changed (and complemented by some more stuff):
<dependency> <groupId>com.amazonaws</groupId> - <artifactId>aws-java-sdk-bundle</artifactId> + <artifactId>aws-java-sdk-s3</artifactId> <version>${aws.version}</version> </dependency>The text was updated successfully, but these errors were encountered: