[BUG] Depclean falsely reports a direct dependency as "UNUSED DIRECT DEPENDENCY"

[algomaster99]

Describe the bug
Upon running Depclean v2.0.1 on ASSERT-KTH/sorald@8762596, it reports four unused direct dependencies - out of which org.sonarsource.java:java-checks-testkit:6.9.0.23563 is definitely used here.

To Reproduce
Steps to reproduce the behavior:

1. Clone https://github.com/SpoonLabs/sorald.
2. Checkout to commit - 8762596558e49be194e4033e3094cc702d6768fd.
3. Run depclean-maven-plugin v2.0.1 without ignoring any scopes.

4.  -------------------------------------------------------
    D E P C L E A N   A N A L Y S I S   R E S U L T S
   -------------------------------------------------------
   USED DIRECT DEPENDENCIES [10]: 
        org.eclipse.jgit:org.eclipse.jgit:5.7.0.202003110725-r:compile (2 MB)
        org.sonarsource.java:java-checks:6.9.0.23563:compile (2 MB)
        fr.inria.gforge.spoon:spoon-core:9.1.0-beta-11:compile (1 MB)
        org.sonarsource.java:java-frontend:6.9.0.23563:compile (909 KB)
        info.picocli:picocli:4.5.2:compile (381 KB)
        ch.qos.logback:logback-classic:1.2.3:test (283 KB)
        commons-io:commons-io:2.7:compile (269 KB)
        org.junit.platform:junit-platform-launcher:1.7.0:test (133 KB)
        org.json:json:20190722:compile (63 KB)
        org.junit.jupiter:junit-jupiter:5.7.0:test (6 KB)
   USED INHERITED DEPENDENCIES [0]: 
   USED TRANSITIVE DEPENDENCIES [38]: 
        org.bouncycastle:bcprov-jdk15on:1.64:compile (4 MB)
        org.sonarsource.sonarqube:sonar-plugin-api:7.9:compile (3 MB)
        com.google.guava:guava:26.0-jre:compile (2 MB)
        com.fasterxml.jackson.core:jackson-databind:2.12.3:compile (1 MB)
        xerces:xercesImpl:2.12.0:compile (1 MB)
        org.apache.commons:commons-compress:1.20:compile (617 KB)
        org.apache.commons:commons-lang3:3.12.0:compile (573 KB)
        org.junit.jupiter:junit-jupiter-params:5.7.0:test (554 KB)
        com.fasterxml.woodstox:woodstox-core:5.2.0:compile (506 KB)
        ch.qos.logback:logback-core:1.2.3:test (460 KB)
        com.fasterxml.jackson.core:jackson-core:2.12.3:compile (356 KB)
        org.bouncycastle:bcpg-jdk15on:1.64:compile (321 KB)
        commons-lang:commons-lang:2.6:compile (277 KB)
        org.codehaus.plexus:plexus-utils:3.2.1:compile (255 KB)
        com.google.code.gson:gson:2.8.6:compile (234 KB)
        org.apache.maven:maven-model:3.8.1:compile (210 KB)
        org.sonarsource.sslr:sslr-core:1.23:compile (193 KB)
        org.checkerframework:checker-qual:2.5.2:compile (188 KB)
        org.junit.platform:junit-platform-engine:1.7.0:test (176 KB)
        org.junit.jupiter:junit-jupiter-api:5.7.0:test (170 KB)
        org.codehaus.woodstox:stax2-api:4.1:compile (165 KB)
        com.googlecode.javaewah:JavaEWAH:1.1.7:compile (162 KB)
        org.apache.maven.shared:maven-shared-utils:3.3.3:compile (150 KB)
        org.junit.platform:junit-platform-commons:1.7.0:test (97 KB)
        com.fasterxml.jackson.core:jackson-annotations:2.12.3:compile (73 KB)
        org.sonarsource.analyzer-commons:sonar-analyzer-commons:1.12.0.632:compile (72 KB)
        com.martiansoftware:jsap:2.1:compile (67 KB)
        org.slf4j:slf4j-api:1.7.25:compile (40 KB)
        com.google.code.findbugs:jsr305:1.3.9:compile (32 KB)
        org.apache.maven.shared:maven-invoker:3.1.0:compile (31 KB)
        org.sonarsource.analyzer-commons:sonar-xml-parsing:1.12.0.632:compile (26 KB)
        com.google.errorprone:error_prone_annotations:2.1.3:compile (13 KB)
        org.sonarsource.analyzer-commons:sonar-analyzer-recognizers:1.12.0.632:compile (9 KB)
        com.google.j2objc:j2objc-annotations:1.1:compile (8 KB)
        org.opentest4j:opentest4j:1.2.0:test (7 KB)
        javax.inject:javax.inject:1:compile (2 KB)
        org.apiguardian:apiguardian-api:1.1.0:test (2 KB)
        org.sonarsource.java:jdt:shaded:6.9.0.23563 (size unknown)
   POTENTIALLY UNUSED DIRECT DEPENDENCIES [4]: 
        info.picocli:picocli-codegen:4.5.2:provided (124 KB)
        org.hamcrest:hamcrest:2.2:test (120 KB)
        org.reflections:reflections:0.9.12:compile (103 KB)
        org.sonarsource.java:java-checks-testkit:6.9.0.23563:compile (5 KB)
   POTENTIALLY UNUSED INHERITED DEPENDENCIES [0]: 
   POTENTIALLY UNUSED TRANSITIVE DEPENDENCIES [17]: 
        org.eclipse.jdt:org.eclipse.jdt.core:3.23.0:compile (6 MB)
        org.eclipse.platform:org.eclipse.osgi:3.17.0:compile (1 MB)
        org.eclipse.platform:org.eclipse.core.resources:3.15.100:compile (879 KB)
        org.bouncycastle:bcpkix-jdk15on:1.64:compile (857 KB)
        org.javassist:javassist:3.26.0-GA:compile (764 KB)
        cglib:cglib-nodep:3.2.5:compile (344 KB)
        org.eclipse.platform:org.eclipse.text:3.12.0:compile (288 KB)
        com.jcraft:jsch:0.1.55:compile (275 KB)
        org.junit.jupiter:junit-jupiter-engine:5.7.0:test (206 KB)
        org.eclipse.platform:org.eclipse.equinox.common:3.15.0:compile (139 KB)
        org.eclipse.platform:org.eclipse.equinox.preferences:3.9.0:compile (136 KB)
        org.eclipse.platform:org.eclipse.core.commands:3.10.100:compile (114 KB)
        org.eclipse.platform:org.eclipse.core.jobs:3.12.0:compile (108 KB)
        org.eclipse.platform:org.eclipse.core.contenttype:3.8.0:compile (100 KB)
        org.eclipse.platform:org.eclipse.core.runtime:3.23.0:compile (70 KB)
        com.jcraft:jzlib:1.1.1:compile (67 KB)
        org.codehaus.mojo:animal-sniffer-annotations:1.14:compile (3 KB)
   

Expected behavior

Report should not include org.sonarsource.java:java-checks-testkit:6.9.0.23563:compile.

[algomaster99]

POTENTIALLY UNUSED DIRECT DEPENDENCIES [4]: 
     info.picocli:picocli-codegen:4.5.2:provided (124 KB)
     org.hamcrest:hamcrest:2.2:test (120 KB)
     org.reflections:reflections:0.9.12:compile (103 KB)
     org.sonarsource.java:java-checks-testkit:6.9.0.23563:compile (5 KB)

Out of these 4, only org.reflections:reflections:0.9.12:compile is unused and we removed that ASSERT-KTH/sorald#654. However, the rest are used.

1. hamcrest - https://github.com/SpoonLabs/sorald/blob/87ce9cc379184fe141dc6dea98e344f21f461818/src/test/java/sorald/sonar/RuleVerifierTest.java#L3
2. picocli - https://github.com/SpoonLabs/sorald/blob/87ce9cc379184fe141dc6dea98e344f21f461818/src/test/java/sorald/sonar/RuleVerifierTest.java#L3
3. sonarsource - https://github.com/SpoonLabs/sorald/blob/master/src/main/java/sorald/sonar/RuleVerifier.java#L14-L40

[cesarsotovalero]

Hi @algomaster99, thanks for reaching out.

The DepClean analysis results are ALL correct!

Let me explain each case separately:

Case java-checks-testkit

This dependency is indeed not used by sorald directly. Instead, the transitive dependency org.sonarsource.sonarqube:sonar-plugin-api:jar:7.9:compile is the one used here.

If you run the command mvn dependency:tree you will get the following output:

[INFO] +- org.sonarsource.java:java-checks-testkit:jar:6.9.0.23563:compile
[INFO] |  +- com.google.code.gson:gson:jar:2.8.6:compile
[INFO] |  +- com.google.guava:guava:jar:26.0-jre:compile
[INFO] |  |  +- org.checkerframework:checker-qual:jar:2.5.2:compile
[INFO] |  |  +- com.google.errorprone:error_prone_annotations:jar:2.1.3:compile
[INFO] |  |  +- com.google.j2objc:j2objc-annotations:jar:1.1:compile
[INFO] |  |  \- org.codehaus.mojo:animal-sniffer-annotations:jar:1.14:compile
[INFO] |  +- com.google.code.findbugs:jsr305:jar:1.3.9:compile
[INFO] |  \- org.sonarsource.sonarqube:sonar-plugin-api:jar:7.9:compile

So, as you can see, java-checks-testkit carries out 8 transitive dependencies to sorald.
It can be removed directly and all the transitive dependencies used included as direct dependencies.
DepClean provides a way of doing this automatically by using the flag -DcreatePomDebloated=true.

Case hamcrest

This dependency has test scope. DepClean by default does not analyze this dependencies because they are not packaged by Maven at the end of the build pipeline.
However, DepClean is capable of analyzing the usage of test dependencies.
To do so, first compile the test classes of the project with the command mvn compiler:testCompile and then you can run DepClean directly from the command line, eg mvn se.kth.castor:depclean-maven-plugin:2.0.2-SNAPSHOT:depclean.

This is part of the output that you will get:

-------------------------------------------------------
 D E P C L E A N   A N A L Y S I S   R E S U L T S
-------------------------------------------------------
USED DIRECT DEPENDENCIES [9]: 
        org.eclipse.jgit:org.eclipse.jgit:5.7.0.202003110725-r:compile (2 MB)
        org.sonarsource.java:java-checks:6.9.0.23563:compile (2 MB)
        fr.inria.gforge.spoon:spoon-core:9.1.0-beta-11:compile (1 MB)
        org.sonarsource.java:java-frontend:6.9.0.23563:compile (909 KB)
        info.picocli:picocli:4.5.2:compile (381 KB)
        commons-io:commons-io:2.7:compile (269 KB)
        org.junit.platform:junit-platform-launcher:1.7.0:test (133 KB)
        org.hamcrest:hamcrest:2.2:test (120 KB)
        org.json:json:20190722:compile (63 KB)
...

Case picocli-codegen

This dependency has aprovided scope. This means that the dependency is used at runtime (if it used). DepClean uses static analysis to detect unused dependencies, so if this library is used dynamically to do some bytecode generation on-the-fly then it is not possible for DepClean (or any other static analysis tool) to detect the usage.

However, I've removed this dependency and then build the project without this dependency and the build is successful (ie, sorald compiles and all the tests pass). So it is ultimately up to the developers to keep it or not, based ont the knowledge they have regarding what it is used dynamically by sorald or not.

[algomaster99]

@cesarsotovalero Thanks for the prompt and elaborate reply!

Case java-checks-testkit

This case is very clear to me. And thanks for pointing out that it could be replaced with the other transitive dependencies. It is always best to have the dependencies as direct if they are used in the program.

Case hamcrest

This is clear too. Thanks!

Case picocli-codegen

I think I got this confused with picocli and that is definitely required at compile time. Anyway, depclean reports picocli-codegen which is apparently not used like you pointed out. I am unsure where it is used and I will have to inspect it.