Regression: Creating shadowJar fails when configurations contain directories (9.0.0)

[simschla]

Expected and Results

Expected: ShadowJar would contain all classes and resources from its configurations. (as in latest 8.x release)

Actual: The execution of shadowJar fails with the message:

Execution failed for task ':myproj:shadowJar'.
> Cannot expand ZIP '/Users/myuser/devel/myproj/build/classes/java/custom' as it is not a file.

(The referenced path is a output-directory of a custom source set, its contents to not need to be extracted, but should be included)

Related environment and versions

shadow 8.3.9 => ✅
shadow 9.0.0 => ❌

Reproduction steps

No response

Anything else?

No response

[Goooler]

Reproducer?

[simschla]

working on one 😄

[simschla]

Reproducer: https://github.com/simschla/gradleup-shadow-repro

gradlew shadowJar
Calculating task graph as configuration cache cannot be reused because file 'app/build.gradle' has changed.
> Task :app:shadowJar FAILED

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':app:shadowJar'.
> Cannot expand ZIP '/Users/simschla/tmp/gradleup-shadow-repro/utilities/build/classes/java/custom' as it is not a file.

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to generate a Build Scan (Powered by Develocity).
> Get more help at https://help.gradle.org.

BUILD FAILED in 740ms
14 actionable tasks: 2 executed, 2 from cache, 10 up-to-date
Configuration cache entry stored.

[Goooler]

Related to #1440.

api sourceSets.custom.output should be replaced by

tasks.named('shadowJar') {
  from sourceSets.custom.output
}

[simschla]

I'm not sure that would work:

• the api sourceSets.custom.output is in :utilities subproject
• shadow is applied in :app project

Or maybe I don't understand the suggestion correctly.

[Goooler]

// In app project:
tasks.named('shadowJar') {
  from project(':utilities').sourceSets.custom.output
}

[simschla]

I'm sorry, but that feels like a workaround to me.
It's requiring the build author to strongly couple these two projects to not hit this issue in the shadow plugin.
In the reproducer, :app is unaware that :utilities chose to separate classes into different source-sets - I don't think choosing to create a shadowJar should require the author of :app project to know about this internal decision of :utilities.

[Goooler]

This is a trade-off as I listed in #1440 (comment).

Or you can try out

plugins {
    id 'buildlogic.java-library-conventions'
}

sourceSets {
    custom
}

// I guess this could be removed after adding `customJar`?
classes.dependsOn customClasses

def customJar = tasks.register('customJar', Jar) {
    from sourceSets.custom.output
    archiveClassifier.set('custom')
}

dependencies {
    api project(':list')
    api files(customJar)
}

[simschla]

Thanks for the explanation — I see the reasoning, and I really appreciate the improvement that already zipped files no longer need to be unzipped and rezipped. That’s a nice optimization! 🙌

That said, from my perspective as the build author, the change does come with a noticeable trade-off: I now need to adapt build logic and introduce new tasks, just to avoid triggering an internal implementation detail of Shadow. That feels a bit heavy for something that’s not directly related to my build’s intent.

Regarding your comment:

But which incorrect files are correct and which incorrect files should fail? The current behavior is a tradeoff.

I totally get that this is a nuanced issue. Personally, I’d prefer a route that doesn’t result in a “only zip files are allowed due to implementation details” outcome. Would it be feasible to allow a few well-known cases (TBD), in addition to zip files, and treat the rest as truly incorrect? That might strike a better balance between correctness and flexibility.

Happy to continue the discussion — thanks again for the thoughtful work on this!

[Goooler]

Your use case is very rare. From my perspective, if you want to expose the dependency via api or implementation, it should be a jar instead of a folder of files.

Would it be feasible to allow a few well-known cases (TBD), in addition to zip files, and treat the rest as truly incorrect?

I have no idea about the standard of well-known cases, could you provide me with some examples?

If your custom source set looks like the structure:

gradleup-shadow-repro/utilities/src/custom/
├── java
│   └── org
│       └── example
│           └── utilities
│               └── CustomUtils.java
└── resources
    ├── META-INF
    │   ├── org
    │   │   └── apache
    │   │       └── logging
    │   │           └── log4j
    │   │               └── core
    │   │                   └── config
    │   │                       └── plugins
    │   │                           └── Log4j2Plugins.dat
    │   └── services
    │       └── org.junit.jupiter.api.extension.Extension
    └── junit-3.8.2.jar

Should we treat Log4j2Plugins.dat as a normal file? Should we unzip junit-3.8.2.jar?
How should we define the strategies for the resources?

[simschla]

@Goooler thank you for considering my comments! 🙏

Your use case is very rare. From my perspective, if you want to expose the dependency via api or implementation, it should be a jar instead of a folder of files.

I agree, the separation into multiple sourceSets could be handled better and I will rewrite that part in our build for sure after this discussion 👍

Should we treat Log4j2Plugins.dat as a normal file? Should we unzip junit-3.8.2.jar?
How should we define the strategies for the resources?

As there is no "correct" answer, my feeling is that both files should be included as is, since the are used as resources not das dependencies - but it's always hard to discuss on hypothetic data, because it almost always comes down to intentions: why has junit-3.8.2.jar been placed in resources/ directory?

Thanks for your investment and for #1613 🙏