Extend HTTPClient usage for downloads to ArtifactorySearch #7254
Comments
aikebah
changed the title
|
@irrandon, @jthurne Are you still using Artifactory with authentication? I'm looking for someone who can verify the correct working of ODC snapshot against an artifactory server requiring authentication by the time I've finished coding the required changes for this enhancement in order to not break its connectivity in a future ODC release. |
|
@marcelstoer Spotted in other tickets that you have some items stored in Artifactory. Are you using Artifactory Pro version and would you be able to help validate correct working of authentication towards it (preferably both basic and bearer) by the ArtifactoryAnalyzer using a snapshot-build of a feature-branch in this project? |
|
@aikebah I'd be happy to help. I am authenticating through |
|
@marcelstoer any version would do, as long as it's the Artifactory Pro. Open source setup responds on the search with HTTP 400:
I hope to work on polishing up the ArtifactorySearch transformation to httpclient5 coming Wednesday and will ping back here when a testable branch has been pushed to this repo |
|
@marcelstoer Implementation is complete. I would appreciate a test using a snapshot built from the |
|
All good, the check passed w/o problem. Would I need to enable the Artifactory analyzer for what you need me to test? |
|
@marcelstoer Yes, artifactoryAnalyzer would have to be enabled. And would be best if at least one of your project's dependencies would be findable by sha1 on your Artifactory instance's search API - that/those dependencies should then have artifactory evidences included in the evidences section of the dependency-check report. |
|
I had no idea what the Artifactory analyzer is or does and the best documentation I could find appears to be #60. Hence, I didn't really know what I was doing here 😄 Anyway, I hope the output I collected is of any use for you. Command Debug snippet ODC report excerpt 
|
|
@marcelstoer at least no errors encountered, otherwise you'd shown those I guess. And if not: is |
|
Yes, no errors.
No, maybe because the dependencies are all present in the local repo?
It's returning the AWS SDK {
"results" : [ {
"uri" : "https://my-bin/artifactory/api/storage/maven-central-cache/com/amazonaws/aws-java-sdk-core/1.11.277/aws-java-sdk-core-1.11.277.jar"
} ]
} |
|
Ah... I'd missed the detail of... for (Evidence e : dependency.getEvidence(EvidenceType.VENDOR)) {
if ("pom".equals(e.getSource())) {
return;
}
}in the artifactoryAnalyzer so indeed... it would not use the Artifactory search as the jar will no doubt already have the pom associated with it already and therefor pom-based evidences are already present. I guess best way to try and get it to really hit Artifactory with a request would be to put the jar in some folder and use the CLI from within that folder to scan the current folder (with artifactory analyzer enabled and centralAnalyzer disabled) path/to/odc/bin/dependency-check.sh \
-s . \
--disableCentral \
--enableArtifactory \
--artifactoryUrl https://my-bin/artifactory \
--artifactoryUsername $ARTIFACTORY_USER \
--artifactoryApiToken $ARTIFCATORY_TOKEN \
-d ~/.m2/repository/org/owasp/dependency-check-data/11.0 \
--noupdate \
-l odc.logshould allow you to run it and simply re-use the already cached cveDB in your maven local repo, assuming that your register your login credentials for artifactory in the referenced env variables |
|
@marcelstoer Would you be able to run a trial with the CLI against your artifactory? When 'seen working' I would mark the PR ready and have it merged for the next release. I'd rather not merge and then find out that it 'breaks when used' if we can avoid that by testing the snapshot. |
This fails with the following stacktrace The {
"results": [
{
"uri": "https://bin.ti8m.ch:443/artifactory/api/storage/maven-central-cache/org/freemarker/freemarker/2.3.33/freemarker-2.3.33.jar"
},
{
"uri": "https://bin.ti8m.ch:443/artifactory/api/storage/gradle-plugins-extended-cache/org/freemarker/freemarker/2.3.33/freemarker-2.3.33.jar"
}
]
} |
|
@marcelstoer A simple HTTP-GET of the URL (e.g. from the browser) indeed returns only the URL, that is to be expected. The request that dependency-check sends includes (or at least is expected to include based on the code I wrote as a replacement for the old code) an additional HTTP Header With that header present Artifactory is expected to respond with a much more detailed response (as also indicated at the Headers (Optionally) at Artifactory REST API documentation for search |
|
I assumed you wanted me to try that and so I did 😄 The result, though, is the same: no checksums returned, only URIs. Their documentation says
Maybe this is to be interpreted as "all information is optional"? |
|
Based on stored responses in the testcases from when the artifactory analyzer was first contributed I would expect at least some of the additional details to be 'always available'. Based on the current Artifactory OSS sources for any found artifact I would expect at the very least its 'BaseStorageInfo' to be added into the response: storageInfoRest.repo = itemInfo.getRepoKey();
storageInfoRest.path = "/" + itemInfo.getRelPath();
storageInfoRest.created = RestUtils.toIsoDateString(itemInfo.getCreated());
storageInfoRest.createdBy = itemInfo.getCreatedBy();
storageInfoRest.lastModified = RestUtils.toIsoDateString(itemInfo.getLastModified());
storageInfoRest.modifiedBy = itemInfo.getModifiedBy();
storageInfoRest.lastUpdated = RestUtils.toIsoDateString(itemInfo.getLastUpdated()); |
|
@marcelstoer Regarding how it can be that you don't get them: is the connection from curl to your artifactory direct, or might there be some WebApplicationFirewall technology in-between that strips out any 'not explicitly allowed' HTTP headers from each request that passes through? |
|
Hah, spot on, genius! It's indeed the WAF dropping the I guess I need to redo the entire ODC CLI test on said host. It'll take a while but I'll report back once done. |
|
Well, at least some other useful outcome of this testing: the code should be prepared to receive responses without the detail and handle those gracefully (try to detect we get the 'basic' response and inform the user in some way that there is likely network infrastructure dropping the X-Result-Detail HTTP-header in between their system and the Artifactory server that is rendering the ArtifactoryAnalyzer useless) |
…of detail Resolves the NPE reported in #7254 (comment) Logs a warning for each entry in the response that does not have the checksums block that is to be expected for any request that has the `X-Result-Detail: info` HTTP header.
|
@marcelstoer Have you mixed up the search-URL and the search response, or is your artifactory actually hosting a jar for freemarker 2.3.33 that has a sha1 collision with the Maven Central hosted org.aspectj:aspectj-weaver:1.9.22.1)? |
Ok, all good now I suppose. The report is fine i.e. as expected and the |
Originally posted by @aikebah in #6949 (comment)
Follow-up issue to #6949 to use HTTPClient for ArtifactorySearch in order to honor the standard proxy configuration also for the (expected rare) cases in which an Artifactory repository is only reachable via a proxy
The text was updated successfully, but these errors were encountered: