-
Notifications
You must be signed in to change notification settings - Fork 1.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Issue #10164 - MetaInfConfiguration Overhaul #10816
Conversation
Ensure JPMS and TLDs can coexist happily Better method naming per reviews Reworking MetaInfConfiguration with newer Java techniques in mind
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think we will need to have a hangout, including @gregw to ensure we're all on the same page.
@@ -550,7 +550,7 @@ public void parse(final Set<? extends Handler> handlers, Resource r) throws Exce | |||
if (!r.exists()) | |||
return; | |||
|
|||
if (FileID.isJavaArchive(r.getPath())) | |||
if (FileID.isJavaArchive(r.getPath())) // TODO: this is now always false, as all Resource objects are directories |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I don't understand the TODO
comment. FileID just checks if the path ends in ".jar"
, so this should be fine?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The resource being passed in is never a raw path to a jar file in this method.
Even if the underlying resource was a JAR, by the time it reaches here the Resource a mounted JAR already, meaning r.getPath()
is just a directory inside the JAR.
This entire method parse(handlers , r)
is basically a no-op. nothing happens.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
You mean that after these changes `parse(handlers, r) is a no-op, because it certainly isn't now.
@@ -968,7 +968,7 @@ else if (entry.getValue() == null) //can't work out provenance of SCI, so can't | |||
{ | |||
for (Map.Entry<ServletContainerInitializer, Resource> entry : sciResourceMap.entrySet()) | |||
{ | |||
if (webInfJar.equals(entry.getValue())) | |||
if (webInfJar.isContainedIn(entry.getValue())) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I don't understand this change. We're checking here if the resource that contains the SCI is the webInfJar that we're looking at. Makes no sense to check containment.
@@ -754,7 +754,7 @@ public boolean isFromExcludedJar(WebAppContext context, ServletContainerInitiali | |||
boolean included = false; | |||
for (Resource r : orderedJars) | |||
{ | |||
included = r.equals(sciResource); | |||
included = r.isContainedIn(sciResource); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
See comment below.
* @throws RuntimeException if unable to convert the URI to URL | ||
* @see URI#toURL() | ||
*/ | ||
public static URL toURL(URI uri) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
What's the motivation for changing the checked exception into a runtime exception?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
So it can be used in a stream
jetty-core/jetty-util/src/main/java/org/eclipse/jetty/util/NullMap.java
Outdated
Show resolved
Hide resolved
} | ||
else | ||
{ | ||
resources.add(resourceFactory.newResource(file.toPath())); | ||
// Treat bundle as jar file that needs to be opened (so that resources within it can be found) | ||
resources.add(resourceFactory.newJarFileResource(file.toPath().toUri())); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
See previous comments about this.
...ee10/jetty-ee10-webapp/src/main/java/org/eclipse/jetty/ee10/webapp/MetaInfConfiguration.java
Outdated
Show resolved
Hide resolved
...ee10/jetty-ee10-webapp/src/main/java/org/eclipse/jetty/ee10/webapp/MetaInfConfiguration.java
Outdated
Show resolved
Hide resolved
|
||
scanJars(context); | ||
// Scan for META-INF/*.tld entries | ||
metaInfTlds.addAll(scanTlds(streamTargets(containerTargets, webappTargets, true), metaInfTldCache)); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think the idea was not to iterate over the resources multiple times, but iterate just once and interrogate for tlds, fragments, resources in one go.
} | ||
|
||
protected List<URI> getAllContainerJars(final WebAppContext context) | ||
/** | ||
* Only look for Servlet 3+ features ({@code META-INF/web-fragment.xml} and {@code META-INF/resources}) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is not the right algorithm: we only need to skip fragments iff metadata-complete or < servlet3 or not discovered configuration. The current code is correct:
if (context.getMetaData().isMetaDataComplete() || (context.getServletContext().getEffectiveMajorVersion() < 3) && !context.isConfigurationDiscovered())
scanTypes.remove(METAINF_FRAGMENTS);
if (scheme.equals("jar")) | ||
{ | ||
URI container = URIUtil.unwrapContainer(uri); | ||
if (container.getScheme().equals("file")) | ||
{ | ||
// Just add a reference to the | ||
loader.addClassPath(container.getPath()); | ||
continue; | ||
} | ||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is this converting things like "jar:file:/usr/jetty/lib/somedependency.jar!/" back to just "/usr/jetty/lib/somedependency.jar"?
Are we to assume that we will never get anything like: "jar:file:/usr/jetty/lib/somedependency.jar!/WEB-INF/classes" or "jar:file:/usr/jetty/lib/somedependency.jar!/WEB-INF/lib/some.jar" ?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@janbartel Can you give us a list of all the different uris that this method might see and what it is meant to do with each of them? i.e. what is "the brief" of this method?
@janbartel I've started testing performance of the MetaInfConfiguration in jetty 11 and jetty 12 using the war file produced by https://github.com/joakime/huge-war That war is 433MB in size, with 319 jars in WEB-INF/lib, and over 218,000 classes (it is ultimately a nonsense war that nobody would deploy, just using it as an example of a worst case scenario of discovery / scanning) The numbers so far.
I branched from While this PR is a vast improvement over Going back to 1 scan (with options) might bring us back down to jetty-11 numbers, but that approach messes up the stream processing behaviors this PR has (i'll noodle it this weekend) |
// BEGIN - slf4j 2.x | ||
/* SLF4J 2.0 */ | ||
// slf4j-api comes from paxexam now. | ||
// res.add(mavenBundle().groupId("org.slf4j").artifactId("slf4j-api").versionAsInProject().noStart()); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
it's probably not sync with ee9 :(
btw I noticed some issues recently and logging was a pain...
I'd like to extract some parts here which can be common and if possible really fix the logging
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@olamy but not in this PR, which is only for changes to MetaInfConfiguration
.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@janbartel sure but my point is either we don't change anything in this PR for this or we keep both files in sync (because something tell me we will forget :) ) BTW they are probably already not in sync anymore.
@joakime and @lorban I think this PR is missing the issue that I raised in #10164. I'm not that concerned about efficiency in our scanning of META-INF (if users want quick start then they can use quickstart). I'm concerned that for some reason If a webapp contains foo.jar and bar.jar, then I could kind of live with seeing mounts for:
but we also see mounts of :
I do not understand:
Thus my concerns are primarily with I have some secondary concerns about
Now I'm sure some of these problems come from bad code in |
I've had a bit more of a look with @janbartel and we can see a few of the problems.
With regards to using the Classloader, I think we could do something like However, it is wrong to use the lifespan of the context for these mounts. We only needed them during starting and not once started. So perhaps a dedicated configuration ResourceFactory that is closed once started? |
@gregw a dedicated start-time only ResourceFactory would be correct for some of the things that . o.e.j.u.resource.Resources in META-INF/resources: ones that actually exist need to hang around for the entire lifetime of the context as they form part of the |
This PR addresses this.
If you use
The MetaInfConfiguration in The extra mounts on Line 410 in 49b3442
That code causes a new mount. (and there's many such examples) Again, we WANT to open the mount for a JAR and keep it open for the entirety of the scanning, close when done all of the scanning. There's no reason to track references the Jar files as normal files during MetaInfConfiguration.
You can mount files, it is not known if that String points to a file or directory during the mount process. As for that mount, it comes from jetty-12.0.x HEAD here ... Line 477 in 49b3442
and here ... Line 482 in 49b3442
Again, this PR cleans all of this up, and more.
Nope, of the problems you have listed in this comment, they are all totally within
We could, but wouldn't we need to create the
That is what #10164 brought up, and this PR fixes this.
We greatly enhanced the unit testing of MetaInfConfiguration with PR #10282 This PR uses that new testcase, with minimal changes (only verifying behavior against mounts while in process). |
This PR finds the
These are stored in the context attribute at the key
These are stored as a context attribute at key |
I think this PR avoids the issue rather than addresses it. It might be doing so in a good way, but it is not addressing the actual issue I am concerned about.
I understand that is what is happening. But the issue is that it should not be happening. Consider the following code: Resource a = factory.newResource("jar:file:///tmp/jetty-xyz123/webapp/WEB-INF/lib/foo.jar!/");
Resource b = factory.newResource("jar:file:///tmp/jetty-xyz123/webapp/WEB-INF/lib/foo.jar!/bar/");
Resource c = a.resolve("bar/"); Currently this will create 2 mount points and b & c will be against different mounts. I think this is wrong.
I don't agree. There was never a distinction before between using newResource vs resolving of an existing one. Previously we used the underlying URL cache to cache a "mount" for the jar and the URL resulting from either approach would be identical. We have moved the cache from the JVM URL mechanism to an explicit Again, I'm not saying that we cannot improve I'll have a look at ResourceFactory and see if I can find exactly what I'm concerned about.... |
That example creates 1 mount point in FileSystemPool, with 2 uses being tracked in the pool.
And in the example of Jetty 9 / Jetty 10 / Jetty 11 that example would have 2 usages in the URL cache.
Keep in mind that you are seeing the ResourceFactory tracking, not the mounts themselves. |
@joakime I'm going to close this one, as any potential changes would need to be based off of head now. |
I agree. |
Replacement for PR #10171 for #10164 - now that we have better testing of original behavior, this PR should be smaller that #10171 was.
MetaInfConfiguration
use ofResource
objects #10171 (comment))