PathMatchingResourcePatternResolver provides duplicate resources when using classpath*: prefix combined with ant-style [SPR-15989]

[spring-projects-issues]

Maël OUAHI opened SPR-15989 and commented

PathMatchingResourcePatternResolver provides duplicate resources when using "classpath*" prefix combined with Ant-style patterns.

I'll attach an application that reproduces the problem.The application uses the location pattern "classpath*:/*-perm.json" to resolve Resources.
If you package and run it

mvn clean package && java -jar target/resources-0.0.1-SNAPSHOT.jar

You should see the following output:

file:/D:/resources/target/resources-0.0.1-SNAPSHOT.jar!/
file:D:\resources\target\resources-0.0.1-SNAPSHOT.jar!/
file:/C:/Program%20Files/Java/jdk1.8.0_40/jre/lib/ext/dnsns.jar!/
...
file:/D:/resources/target/resources-0.0.1-SNAPSHOT.jar!/domaine-perm.json
file:D:\resources\target\resources-0.0.1-SNAPSHOT.jar!/domaine-perm.json

---

Affects: 4.3.8, 4.3.9, 4.3.10, 4.3.11

Attachments:

• resources.zip (1.04 MB)

Issue Links:

• PathMatchingResourcePatternResolver does not consider manifest based classpaths [SPR-13685] #18260 PathMatchingResourcePatternResolver does not consider manifest based classpaths
• PathMatchingResourcePatternResolver returns duplicate resources when using classpath*: prefix [SPR-16117] #20665 PathMatchingResourcePatternResolver returns duplicate resources when using classpath* prefix

Referenced from: commits 9d8e3d4, a839477

Backported to: 4.3.12

[spring-projects-issues]

Juergen Hoeller commented

Running the attached app, all I can see are regular entries along the lines of D:\resources\target in your output - no duplicates here, I'm afraid. There have to be some specific conditions involved in your setup, so please try to debug PathMatchingResourcePatternResolver accordingly: Somewhere in doFindAllClassPathResources, possibly in addAllClassLoaderJarRoots / addClassPathManifestEntries, the same jar might be seen twice but with different URL format.

I'm pretty sure this only happens when searching for such patterns in the root of your jar files... since the root of such files is unfortunately not easily discoverable with a standard ClassLoader (since ClassLoader.getResources("") is not guaranteed to deliver a complete result). As a consequence, it is usually recommendable to put such files into a specific subdirectory (even META-INF) in each jar file which can be more reliably searched through a standard call (e.g. ClassLoader.getResources("META-INF")).

[spring-projects-issues]

Christophe Blanchard commented

I do reproduce the problem as well :

"C:\Program Files\Java\jdk1.8\bin\java.exe" -jar target/resources-0.0.1-SNAPSHOT.jar

file:/D:/tmp/resources/resources/target/resources-0.0.1-SNAPSHOT.jar!/
file:D:\tmp\resources\resources\target\resources-0.0.1-SNAPSHOT.jar!/
file:/C:/Program%20Files/Java/jdk1.8/jre/lib/ext/access-bridge-64.jar!/
file:/C:/Program%20Files/Java/jdk1.8/jre/lib/ext/cldrdata.jar!/
file:/C:/Program%20Files/Java/jdk1.8/jre/lib/ext/dnsns.jar!/
file:/C:/Program%20Files/Java/jdk1.8/jre/lib/ext/jaccess.jar!/
file:/C:/Program%20Files/Java/jdk1.8/jre/lib/ext/jfxrt.jar!/
file:/C:/Program%20Files/Java/jdk1.8/jre/lib/ext/localedata.jar!/
file:/C:/Program%20Files/Java/jdk1.8/jre/lib/ext/nashorn.jar!/
file:/C:/Program%20Files/Java/jdk1.8/jre/lib/ext/sunec.jar!/
file:/C:/Program%20Files/Java/jdk1.8/jre/lib/ext/sunjce_provider.jar!/
file:/C:/Program%20Files/Java/jdk1.8/jre/lib/ext/sunmscapi.jar!/
file:/C:/Program%20Files/Java/jdk1.8/jre/lib/ext/sunpkcs11.jar!/
file:/C:/Program%20Files/Java/jdk1.8/jre/lib/ext/zipfs.jar!/
file:/D:/tmp/resources/resources/target/resources-0.0.1-SNAPSHOT.jar!/domaine-perm.json
file:D:\tmp\resources\resources\target\resources-0.0.1-SNAPSHOT.jar!/domaine-perm.json

The problem does not appear when running with spring-boot-maven-plugin instead of maven-shade-plugin. weird...

[spring-projects-issues]

Maël OUAHI commented

thanks for your reply, i confirm that the problem does not appear when i put the file into subdirectory, but i reproduce the problem when i run the app on windows 7 OS and centOs 7.

[spring-projects-issues]

Juergen Hoeller commented

I'm afraid I still can't reproduce the problem on Windows 10 here... but it indeed seems to be related to the maven-shade-plugin.

Any chance you could attach a debugger to this? Which Resource type are those duplicated entries? Which step in PathMatchingResourcePatternResolver creates them (I suppose addAllClassLoaderJarRoots / addClassPathManifestEntries)?

[spring-projects-issues]

Maël OUAHI commented

I reproduce the problem when using maven-assembly-plugin to create the fat jar instead of maven-shade-plugin.

[spring-projects-issues]

Juergen Hoeller commented

I can reproduce the problem as well now: on JDK 8 but not on JDK 9 (which I was accidentally always running on before - sorry). On JDK 9, the system class loader never extends URLClassLoader but on JDK 8, it apparently may do so... even in combination with the "java.class.path" system property (which we're not taking into account in our code).

I'll try to sanitize the paths in our code accordingly before adding them to the result set, properly identifying the same jar there and only including it once.

[spring-projects-issues]

Juergen Hoeller commented

We're explicitly checking for duplicates in addClassPathManifestEntries now, with or without extra leading slash in the existing result (typically from URLClassLoader.getURLs()).

This is in master now for the 5.0 GA release. I'll also backport it to 4.3.12 later today. It'd be great if you could give 4.3.12.BUILD-SNAPSHOT a try once it's in; I'll let you know.

[spring-projects-issues]

Maël OUAHI commented

thanks for you responsiveness, let me know when backporting was done and i will try it.

[spring-projects-issues]

Maël OUAHI commented

It's OK. I tried it and this is the output

file:/D:/dev/workspaces/oxygen/hyperia/resources/target/resources-0.0.1-SNAPSHOT.jar!/
file:/C:/Program%20Files/Java/jdk1.8/jre/lib/ext/access-bridge-64.jar!/
...
file:/D:/dev/workspaces/oxygen/hyperia/resources/target/resources-0.0.1-SNAPSHOT.jar!/domaine-perm.json

[spring-projects-issues]

Udayakumar commented

I am facing exactly the same issue even with 5.0.1.RELEASE.
The pattern i am using is - classpath*:/**/*.rdrl to fetch all the .rdrl files inside jars in the class path. Basically we have multiple jars in the class path containing .rdrl file. Each file is returned two times one with / and another with \ in the path.

[spring-projects-issues]

Juergen Hoeller commented

Udayakumar, could you please create a new JIRA issue for your scenario? Please dump the full URLs that you're seeing, like above... Plain slash differences shouldn't matter since UrlResource.equals considers them equal, so there's probably something else involved. In any case, let's try to fix this for 5.0.2 / 4.3.13.

[spring-projects-issues]

Udayakumar commented

Thanks. opened a new issue.
#20665