Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Different versions of libraries packaged in war #1912

Open
josegar74 opened this issue Feb 24, 2017 · 15 comments
Open

Different versions of libraries packaged in war #1912

josegar74 opened this issue Feb 24, 2017 · 15 comments
Labels
enhancement stale stale issue flagged for closure.

Comments

@josegar74
Copy link
Member

josegar74 commented Feb 24, 2017

Checked in GeoNetwork 3.2.1 war:

  • batik-ext-1.6.jar
  • batik-ext-1.7.jar

  • common-2.6.0.jar
  • common-3.2.1-0.jar

  • commons-beanutils-1.6.jar
  • commons-beanutils-core-1.7.0.jar

  • commons-lang-2.6.jar
  • commons-lang3-3.1.jar

  • jsonic-1.2.0.jar
  • jsonic-1.2.7.jar

These should not be even in the war as provided by the container:

  • servlet-api-2.5-6.1.14.jar
  • servlet-api-2.5-20081211.jar

  • Mix of libraries related to jetty libs for 6.1.26 (jetty-6.1.26.jar, jetty-util-6.1.26.jar) and others for 8.1.15.v20140411 version. I don't think these should be packaged in the war.

I think this can lead to unexpected behaviours.

@josegar74 josegar74 added this to the 3.2.2 milestone Feb 24, 2017
@eswright
Copy link

eswright commented Feb 27, 2017

I'm trying to do a simple ubuntu install and getting
[ERROR] javac: invalid target release: 1.8

Is this related?

yet javac -version returns 1.7.0_121
?

@josegar74
Copy link
Member Author

Don't think this is related, it seems more like a Java 8 feature has been added and you are building with Java 7.

But I was under the impression that Java 7 was supported, will try it to verify.

@juanluisrp
Copy link
Contributor

About the commons-lang, note that Lang 3.0 (and subsequent versions) use a different package (org.apache.commons.lang3) than the previous versions (org.apache.commons.lang), allowing it to be used at the same time as an earlier version.

@pmauduit
Copy link
Contributor

But I was under the impression that Java 7 was supported, will try it to verify.

Not anymore with 3.2.x, from what I remember, you need Java 8 at compilation (there are some lambdas around IIRC) and runtime (because of dependencies needing it: solr, jetty, geotools).

@pmauduit
Copy link
Contributor

pmauduit commented Apr 20, 2017

I began to open some issues related to this one recently:

I opened a branch to address this (it is related to a current work for a french project at c2c, as well as the PR opened by @fgravin here: #1836): https://github.com/pmauduit/core-geonetwork/tree/classloading-cleanup

For now, here is the status of remaining duplicated classes in the WEB-INF/lib:

  • com/google/protobuf (twice) provided by netcdf-4.0.patch.jar and protobuf-java-3.0.2.jar
  • com/sun/istack (twice) provided by jaxb-core-2.2.7.jar and istack-commons-runtime-2.16.jar
  • javax/transaction (twice) provided by geronimo-jta_1.0.1B_spec-1.0.1.jar and jboss-transaction-api_1.2_spec-1.0.0.Final.jar
  • javax/xml/parsers (3 times) provided by xml-apis-1.0.b2.jar, xmlParserAPIs-2.0.2.jar, xmlParserAPIs-2.6.2.jar
  • javax/xml/transform (twice) provided by xml-apis-1.0.b2.jar and xmlParserAPIs-2.6.2.jar
  • net/arnx/jsonic (twice) provided by jsonic-1.2.7.jar and jsonic-1.2.0.jar
  • org/apache/activemq/protobuf (twice) provided by activemq-core-5.6.0.jar and activemq-protobuf-1.1.jar
  • org/apache/commons/beanutils (twice) provided by commons-beanutils-1.6.jar and commons-beanutils-core-1.7.0.jar

Not ready yet, and still needs some work from my side

@mprins
Copy link

mprins commented Nov 2, 2017

I just downloaded 3.2.2 and the case of the included servlet api has become twice a bad! You can now find the following versions of the servlet api in the war file:

  • servlet-api-2.5-6.1.14.jar
  • servlet-api-2.5-20081211.jar
  • geronimo-servlet_3.0_spec-1.0.jar
  • javax.servlet-api-3.1.0.jar

Also the JSP api is included:

  • jsp-api-2.1-6.1.14.jar

Note that these will cause things to break on various servlet containers and break the war contract as the servlet specification and implementation should be provided by the container

Having to remove these sort defeats the purpose of a downloadable .war artifact

@pmauduit
Copy link
Contributor

pmauduit commented Nov 2, 2017

These are mainly caused by the dependency onto activeMq (needed by wfsfeature-harvester):

[INFO] +- org.geonetwork-opensource:wfsfeature-harvester:jar:3.2.3-SNAPSHOT:compile
[...]
[INFO] |  +- org.apache.activemq:activemq-core:jar:5.6.0:compile
[INFO] |  |  +- org.apache.geronimo.specs:geronimo-jms_1.1_spec:jar:1.1.1:compile
[INFO] |  |  +- org.apache.activemq:kahadb:jar:5.6.0:compile
[INFO] |  |  +- org.apache.activemq.protobuf:activemq-protobuf:jar:1.1:compile
[INFO] |  |  +- org.fusesource.fuse-extra:fusemq-leveldb:jar:1.1:compile
[INFO] |  |  |  +- org.fusesource.hawtbuf:hawtbuf-proto:jar:1.9:compile
[INFO] |  |  |  +- org.fusesource.hawtdispatch:hawtdispatch-scala:jar:1.9:compile
[INFO] |  |  |  |  \- org.fusesource.hawtdispatch:hawtdispatch:jar:1.9:compile
[INFO] |  |  |  +- org.iq80.leveldb:leveldb:jar:0.2:compile
[INFO] |  |  |  |  +- org.iq80.leveldb:leveldb-api:jar:0.2:compile
[INFO] |  |  |  |  +- com.google.inject:guice:jar:3.0:compile
[INFO] |  |  |  |  |  \- javax.inject:javax.inject:jar:1:compile
[INFO] |  |  |  |  \- com.google.inject.extensions:guice-multibindings:jar:3.0:compile
[INFO] |  |  |  +- org.fusesource.leveldbjni:leveldbjni-osx:jar:1.2:compile
[INFO] |  |  |  |  \- org.fusesource.leveldbjni:leveldbjni:jar:1.2:compile
[INFO] |  |  |  |     \- org.fusesource.hawtjni:hawtjni-runtime:jar:1.5:compile
[INFO] |  |  |  +- org.fusesource.leveldbjni:leveldbjni-linux32:jar:1.2:compile
[INFO] |  |  |  +- org.fusesource.leveldbjni:leveldbjni-linux64:jar:1.2:compile
[INFO] |  |  |  +- org.xerial.snappy:snappy-java:jar:1.0.3:compile
[INFO] |  |  |  +- org.apache.hadoop:hadoop-core:jar:1.0.0:compile
[INFO] |  |  |  |  +- org.mortbay.jetty:jetty:jar:6.1.26:compile
[INFO] |  |  |  |  |  \- org.mortbay.jetty:servlet-api:jar:2.5-20081211:compile
[INFO] |  |  |  |  +- org.mortbay.jetty:jetty-util:jar:6.1.26:compile
[INFO] |  |  |  |  +- org.mortbay.jetty:jsp-api-2.1:jar:6.1.14:compile
[INFO] |  |  |  |  |  \- org.mortbay.jetty:servlet-api-2.5:jar:6.1.14:compile
[INFO] |  |  |  |  \- org.mortbay.jetty:jsp-2.1:jar:6.1.14:compile
[INFO] |  |  |  \- org.scala-lang:scala-library:jar:2.9.1:compile
[INFO] |  |  +- org.fusesource.mqtt-client:mqtt-client:jar:1.0:compile
[INFO] |  |  |  +- org.fusesource.hawtdispatch:hawtdispatch-transport:jar:1.9:compile
[INFO] |  |  |  \- org.fusesource.hawtbuf:hawtbuf:jar:1.9:compile
[INFO] |  |  +- org.osgi:org.osgi.core:jar:4.1.0:compile
[INFO] |  |  +- org.apache.geronimo.specs:geronimo-j2ee-management_1.1_spec:jar:1.0.1:compile
[INFO] |  |  \- org.jasypt:jasypt:jar:1.8:compile
  • geronimo-servlet is needed by camel-http4 (transitively needed by the same wfs harvester sub-project)

  • the 3.1.0 is needed by jetty-servlet, transitively needed by the core sub-project of GN

@mprins
Copy link

mprins commented Nov 2, 2017

the servlet api's only provide the interfaces and not the implementation so they cannot actually be "needed" any servlet container will provide for these so they must be excluded from the war

@pmauduit
Copy link
Contributor

pmauduit commented Nov 2, 2017

the servlet api's only provide the interfaces and not the implementation so they cannot actually be "needed" any servlet container will provide for these so they must be excluded from the war

I'm aware of that, I was just explaining why the jars actually ended up in the webapp (which dependency were responsible). but I'm still wondering if the correct way of avoiding them in the war is to blacklist them in the web/pom.xml file or at the maven sub-project level.

@mprins
Copy link

mprins commented Nov 3, 2017

ideally they should have a provided scope, but if you have a dependency not under your control that doesn't do this and has a compile/runtime scope to eg servlet-api then it needs to be excluded from that dependency to prevent it getting dragged in at the point the dependency is declared.

pmauduit added a commit to pmauduit/core-geonetwork that referenced this issue Nov 3, 2017
These jars should be provided by the servlet container.

Note: If Hadoop transitively requires jetty + jetty:jsp-api +
servlet-api (in wfsfeature-harvester subproject), there is probably a
good reason for this. I preferred to exclude the dependencies from the
web/pom.xml instead of acting on each maven subprojects.

Note 2: some classes actually require the servlet-api jar, so I added it
as a dependency, but keeping it away from the resulting artifact using
the 'provided' scope.

Tests: compilation + runtime on web, no issue so far.
@pmauduit
Copy link
Contributor

pmauduit commented Nov 3, 2017

ideally they should have a provided scope, but if you have a dependency not under your control that doesn't do this and has a compile/runtime scope to eg servlet-api then it needs to be excluded from that dependency to prevent it getting dragged in at the point the dependency is declared.

@mprins See above for a PR which should resolve your issue ; we do have (at least an "indirect") control over the dependency (wfsfeature-harvester is a subproject in the GN sourcecode tree), I don't really know if there is a way to have the wfs harvester working in a "standalone" mode (which might explain why it has a complete servlet container - jetty - as a dependency), so I am more confident on blacklisting the problematic jars from the web/ project rather than directly onto the wfs module.

@Delawen Delawen modified the milestones: 3.2.2, 3.2.3 Dec 15, 2017
@Delawen Delawen modified the milestones: 3.2.3, 3.4.3 Jun 27, 2018
pmauduit added a commit to pmauduit/core-geonetwork that referenced this issue Jun 29, 2018
These jars should be provided by the servlet container.

Note: If Hadoop transitively requires jetty + jetty:jsp-api +
servlet-api (in wfsfeature-harvester subproject), there is probably a
good reason for this. I preferred to exclude the dependencies from the
web/pom.xml instead of acting on each maven subprojects.

Note 2: some classes actually require the servlet-api jar, so I added it
as a dependency, but keeping it away from the resulting artifact using
the 'provided' scope.

Tests: compilation + runtime on web, no issue so far.
@pmauduit
Copy link
Contributor

Note: in geOrchestra, we usually deploy our webapps in official jetty docker images. Since recently, Jetty at startup provide a nice feature as it scans the Jars and tells about duplicated classes. This might simplify tracking down classloading pollutions. Example of output below:

[...]
2018-06-29 09:54:32.045:WARN:oeja.AnnotationParser:qtp1020391880-11: org.apache.log4j.spi.AppenderAttachable scanned from multiple locations: jar:file:///var/lib/jetty/webapps/geonetwork/WEB-INF/lib/apache-log4j-extras-1.1.jar!/org/apache/log4j/spi/AppenderAttachable.class, jar:file:///var/lib/jetty/webapps/geonetwork/WEB-INF/lib/log4j-1.2.17.jar!/org/apache/log4j/spi/AppenderAttachable.class
2018-06-29 09:54:32.045:WARN:oeja.AnnotationParser:qtp1020391880-11: org.apache.log4j.spi.Configurator scanned from multiple locations: jar:file:///var/lib/jetty/webapps/geonetwork/WEB-INF/lib/apache-log4j-extras-1.1.jar!/org/apache/log4j/spi/Configurator.class, jar:file:///var/lib/jetty/webapps/geonetwork/WEB-INF/lib/log4j-1.2.17.jar!/org/apache/log4j/spi/Configurator.class
2018-06-29 09:54:32.045:WARN:oeja.AnnotationParser:qtp1020391880-11: org.apache.log4j.spi.DefaultRepositorySelector scanned from multiple locations: jar:file:///var/lib/jetty/webapps/geonetwork/WEB-INF/lib/apache-log4j-extras-1.1.jar!/org/apache/log4j/spi/DefaultRepositorySelector.class, jar:file:///var/lib/jetty/webapps/geonetwork/WEB-INF/lib/log4j-1.2.17.jar!/org/apache/log4j/spi/DefaultRepositorySelector.class
2018-06-29 09:54:32.046:WARN:oeja.AnnotationParser:qtp1020391880-11: org.apache.log4j.spi.ErrorCode scanned from multiple locations: jar:file:///var/lib/jetty/webapps/geonetwork/WEB-INF/lib/apache-log4j-extras-1.1.jar!/org/apache/log4j/spi/ErrorCode.class, jar:file:///var/lib/jetty/webapps/geonetwork/WEB-INF/lib/log4j-1.2.17.jar!/org/apache/log4j/spi/ErrorCode.class
[...]

@mprins
Copy link

mprins commented Jun 29, 2018

that probably only works for concrete classes and not interface such as all the servlet-*-api jars I listed.

this issue still not having been resolved after a year and a half probably shows the interest of having this software being able to run reliably in a compliant servlet container.
#2287 has been sitting there for over a year and a half

juanluisrp pushed a commit that referenced this issue Jul 25, 2018
These jars should be provided by the servlet container.

Note: If Hadoop transitively requires jetty + jetty:jsp-api +
servlet-api (in wfsfeature-harvester subproject), there is probably a
good reason for this. I preferred to exclude the dependencies from the
web/pom.xml instead of acting on each maven subprojects.

Note 2: some classes actually require the servlet-api jar, so I added it
as a dependency, but keeping it away from the resulting artifact using
the 'provided' scope.
juanluisrp pushed a commit that referenced this issue Jul 27, 2018
Cherry-picked 3bba23e.

These jars should be provided by the servlet container.

Note: If Hadoop transitively requires jetty + jetty:jsp-api +
servlet-api (in wfsfeature-harvester subproject), there is probably a
good reason for this. I preferred to exclude the dependencies from the
web/pom.xml instead of acting on each maven subprojects.

Note 2: some classes actually require the servlet-api jar, so I added it
as a dependency, but keeping it away from the resulting artifact using
the 'provided' scope.
@jahow
Copy link
Contributor

jahow commented Oct 22, 2018

#2287 was merged in the trunk.

@juanluisrp @pmauduit can we close this issue?

@pmauduit
Copy link
Contributor

From my PoV, no because #2287 was only referring to servlet-related dependencies (servlet-api should not be in the webapp's classpath in any case because the code should be provided by the servlet-container).

For the other libs duplicated in the classpath, we just have to provide one and only one version of each.

@ticheler ticheler added the stale stale issue flagged for closure. label Mar 20, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement stale stale issue flagged for closure.
Projects
None yet
Development

No branches or pull requests

8 participants