[poc][wip] Cache in local Artifactory everything that is downloaded by Conan

[jgsogo]

Changelog: (Feature | Fix | Bugfix): Describe here your pull request
Docs: https://github.com/conan-io/docs/pull/XXXX

This is just to start talking about it and realize about different needs

• What do we want to cache? Everything like the DownloadCache does or just the things used by the recipe?
  The DownloadCache caches every file, the conanfile.py, conanmanifest.txt,... Here we probably want to cache only 3rdparty files:
  • can we filter them using the URLs in the remote.json (maybe only v2)?
  • activate some flag before running certain methods to enable/disable this cache?
• Do we want to provide some organization inside the generic repo or everything inside one folder?
• Credentials: use JFrog CLI under the hood and forget about credentials?
  • use-case: the user can try-get artifacts, but cannot upload them.

Refactor before? Chain download classes?

[memsharded]

Good start. A few ideas:

The DownloadCache caches every file, the conanfile.py, conanmanifest.txt,... Here we probably want to cache only 3rdparty files:

yes, I think this backup should only cache things outside the Conan remotes, it doesn't make sense to cache thing from Conan repos. To think: What happen with "git clone" source retrieval that is not a download?

can we filter them using the URLs in the remote.json (maybe only v2)?

Yes, I think this is the way to go. No need v2 only, this feature will be new and experimental, so it can have implemented.

activate some flag before running certain methods to enable/disable this cache?
I would prefer the cache to filter based on origins if possible, instead of methods.

Do we want to provide some organization inside the generic repo or everything inside one folder?

I would say it should be a hash based storage, no organization.

Lets do some brainstorming about this, this was a good kick off!

[memsharded]

Better avoid the "cache" term, and call it something like "sources backup"

[jgsogo]

• The problem with the API v1 is the get_download_urls, they might point to an URL different from the remote one. But, the failing behavior would be to store more things in Artifactory, not a big issue though.

• SCM sources. I see some integration for Git LFS and Artifactory, but nothing built-in. I would say this can wait for a second iteration, it will affect recipes using SCM feature and servers that cannot create a bundle from a commit, but this w won't happen for recipes in ConanCenter. IMO, this won't block this feature.

[memsharded]

The problem with the API v1 is the get_download_urls, they might point to an URL different from the remote one. But, the failing behavior would be to store more things in Artifactory, not a big issue though.

That is true, it is possible, but I am not aware of any server implementation right now that uses different URLs for the downloads, so it could be a good enough assumption. I am fine with providing this for revisions V2 only.

SCM sources. I see some integration for Git LFS and Artifactory, but nothing built-in. I would say this can wait for a second iteration, it will affect recipes using SCM feature and servers that cannot create a bundle from a commit, but this w won't happen for recipes in ConanCenter. IMO, this won't block this feature.

Yes, agree it can wait, it is just something to think at some point.

[jgsogo]

This PR needs to be implemented on top of #8116

[jgsogo]

Superseded by #8211