Exception when using partial / third party CDN sources

[esteluk]

• I've read and understood the CONTRIBUTING guidelines and have done my best effort to follow.

Report

What did you do?

If you configure a Podfile to use two CDN sources – one of which is the trunk, and the second is a private spec CDN that does not contain a complete mirror of the trunk, pod install fails to complete when I try to use dependencies from both.

source 'https://cdn.mycompany.com/api/pods/pods/'
source 'https://cdn.cocoapods.org/'

pod 'PrivateDependency'
pod 'Swiftlint'

What did you expect to happen?

Installs all pod dependencies successfully.

What happened instead?

### Error

Errno::ENOENT - No such file or directory @ rb_sysopen - /Users/nathan/.cocoapods/repos/pods-private/all_pods_versions_4_0_1.txt
/Users/nathan/Developer/Projects/open/cocoapods/Core/lib/cocoapods-core/cdn_source.rb:327:in `initialize'
/Users/nathan/Developer/Projects/open/cocoapods/Core/lib/cocoapods-core/cdn_source.rb:327:in `open'
/Users/nathan/Developer/Projects/open/cocoapods/Core/lib/cocoapods-core/cdn_source.rb:327:in `local_file'
/Users/nathan/Developer/Projects/open/cocoapods/Core/lib/cocoapods-core/cdn_source.rb:286:in `ensure_versions_file_loaded'
/Users/nathan/Developer/Projects/open/cocoapods/Core/lib/cocoapods-core/cdn_source.rb:208:in `search'
/Users/nathan/Developer/Projects/open/cocoapods/Core/lib/cocoapods-core/source/aggregate.rb:83:in `block in search'
/Users/nathan/Developer/Projects/open/cocoapods/Core/lib/cocoapods-core/source/aggregate.rb:83:in `select'
/Users/nathan/Developer/Projects/open/cocoapods/Core/lib/cocoapods-core/source/aggregate.rb:83:in `search'
/Users/nathan/.rbenv/versions/3.2.3/lib/ruby/gems/3.2.0/gems/cocoapods-1.15.2/lib/cocoapods/resolver.rb:416:in `create_set_from_sources'
/Users/nathan/.rbenv/versions/3.2.3/lib/ruby/gems/3.2.0/gems/cocoapods-1.15.2/lib/cocoapods/resolver.rb:385:in `find_cached_set'
/Users/nathan/.rbenv/versions/3.2.3/lib/ruby/gems/3.2.0/gems/cocoapods-1.15.2/lib/cocoapods/resolver.rb:360:in `specifications_for_dependency'
/Users/nathan/.rbenv/versions/3.2.3/lib/ruby/gems/3.2.0/gems/cocoapods-1.15.2/lib/cocoapods/resolver.rb:165:in `search_for'
/Users/nathan/.rbenv/versions/3.2.3/lib/ruby/gems/3.2.0/gems/cocoapods-1.15.2/lib/cocoapods/resolver.rb:274:in `block in sort_dependencies'
/Users/nathan/.rbenv/versions/3.2.3/lib/ruby/gems/3.2.0/gems/cocoapods-1.15.2/lib/cocoapods/resolver.rb:267:in `each'
/Users/nathan/.rbenv/versions/3.2.3/lib/ruby/gems/3.2.0/gems/cocoapods-1.15.2/lib/cocoapods/resolver.rb:267:in `sort_by'
/Users/nathan/.rbenv/versions/3.2.3/lib/ruby/gems/3.2.0/gems/cocoapods-1.15.2/lib/cocoapods/resolver.rb:267:in `sort_by!'
/Users/nathan/.rbenv/versions/3.2.3/lib/ruby/gems/3.2.0/gems/cocoapods-1.15.2/lib/cocoapods/resolver.rb:267:in `sort_dependencies'

Swiftlint uses CDN shard 4_0_1 and Cocoapods queries each CDN repo for the contents of this shard.
If this CDN shard doesn't exist on a repo (which is very likely for any private CDN that doesn't copy the trunk, but very unlikely for the real trunk), this file returns a 404 and isn't downloaded successfully. This leads to the above exception ☝🏽

If I remove any trunk dependencies from my Podfile, pod install completes as expected.

CocoaPods Environment

Stack

   CocoaPods : 1.15.2
        Ruby : ruby 3.2.3 (2024-01-18 revision 52bb2ac0a6) [arm64-darwin23]
    RubyGems : 3.5.5
        Host : macOS 14.3.1 (23D60)
       Xcode : 15.3 (15E204a)
         Git : git version 2.39.3 (Apple Git-146)
Ruby lib dir : /Users/nathan/.rbenv/versions/3.2.3/lib
Repositories : 
               pods-private - CDN - https://cdn.mycompany.com/api/pods/pods/
               trunk - CDN - https://cdn.cocoapods.org/

Installation Source

Executable Path: /Users/nathan/.rbenv/versions/3.2/bin/pod

Plugins

cocoapods-deintegrate : 1.0.5
cocoapods-plugins     : 1.0.0
cocoapods-search      : 1.0.1
cocoapods-trunk       : 1.6.0
cocoapods-try         : 1.2.0
slather               : 2.7.4

Project that demonstrates the issue

Will try to identify this later.

[santam85]

@amorde We use JFrog as a CDN repo and we are running into the same. Any clue if they incompletely implemented the protocol, or is it a bug in the client?

[igor-makarov]

@santam85 @esteluk
CDN support isn't limited to the Trunk.

It's entirely possible that JFrog's Artifactory has done something wrong - they've had limited contact with CocoaPods when they developed their products. It could also be a misconfiguration of some sort.

Try contacting their support for assistance.

[santam85]

@igor-makarov do you have any other CDN source different from the trunk we can use to test a configuration similar to the above example? To pinpoint if the issue is with JFrog or not.

[igor-makarov]

@santam85 sadly, I do not. There was one I personally had for experiments, but I've shut it down due to lack of use.

You could build your own CDN if you follow the format as specified at https://github.com/CocoaPods/cdn.cocoapods.org

[santam85]

@igor-makarov how would you explain the error we are getting then? From what i gather, the Cocoapods client tries to download the sharded podspec (4/0/1) from our private repo for Swiftlint (available on trunk), fails to do so as we have no Pod matching that particular shard, and then crashing.

[igor-makarov]

@santam85 It's okay if you don't have the pod in the shard, but you need to have an empty shard that corresponds to 4/0/1.

[santam85]

@igor-makarov the code at https://github.com/CocoaPods/cdn.cocoapods.org/blob/main/Scripts/create_pods_and_versions_index.rb does not seem to behave that way. There might be gaps in the hashes if there is no pod matching a particular prefix, so this needs to be addressed either in the client (my preference, more economical solution) or in the pod version index generation.

[igor-makarov]

The CDN support was intended to deal with the size the trunk repo, and its unique performance needs. Trunk does not have missing shards, due to the sheer number of pods.

There's very limited badwidth to address changes to either the client or server components - my realistic recommendation is to switch to using git for your non-trunk spec repo.

Also, I would like to reiterate that JFrog's implementation of "CDN-like" spec repos is not affiliated with CocoaPods and I do not know how it works.

[santam85]

Unfortunately that recommendation does not work for our enterprise, so we are stuck with what JFrog offers us. We'll work with them trying to address potential bugs in their implementation, but would you be open to us to keep working on #771 to allow the client to keep looking for pods on other sources if it can't find the required shard?

[igor-makarov]

CocoaPods is a project that's entirely dependent on the mainteneace work of volunteers.

Personally, I cannot justify devoting my time to a PR trying to fix a problem whose root cause is a non-compliant implementation by a third party (JFrog) of a feature that's being used in an unintended manner.

@eyalbe4 @yahavi - there's a long-standing compatibility problem in Artifactory that I think you need to address.

[santam85]

I fail to see how their implementation is non-compliant: if they were running the code you posted in this issue to generate the json shards, they would end up with exactly the problem at hand, and gaps in the generated shards. I couldn't find any reference documentation on the CocoaPods docs that also spells out this requirement for empty shards in private CDN repos, can you point me to it?

We are willing to contribute any needed effort to get it fixed, as we are heavy users of CocoaPods and can provide insights on requirements for Enterprise users.

[igor-makarov]

@santam85 there was never an intention for the CDN format to be used for third party repos. The documentation in https://github.com/CocoaPods/cdn.cocoapods.org is not instructions on how to set up your own CDN repo. This is out of scope for CocoaPods. It's a description of how the code works.

Please understand that there's absolutely no bandwidth in CocoaPods to deal with any of this. Talk to your vendor.

[santam85]

You said yourself earlier:

CDN support isn't limited to the Trunk.

but now you seem to have backtracked on that:

there was never an intention for the CDN format to be used for third party repos

We are definitely talking to our vendor about it, but they will also likely require some form of support from Cocoapods to comply with these undocumented standards and expectations.

I don't understand why this much resistance to improve Cocoapods, in the spirit of open-source we're happy to contribute and volunteer the resources needed to address this, as well as financially sponsor and help maintain the project.

[maikelvdh]

Great to see this ticket (and PR), as I have raised similar one with comparing patch here: CocoaPods/CocoaPods#12338! I was awaiting some feedback on the issue which unfortunately didn't reached any further attention yet!

[adam-browning]

Dear @santam85 & @esteluk ,
My name is Adam and I’m the product leading package integrations @jfrog Artifactory. I’d like to thank you for raising this important issue and I apologize for not responding sooner.

I wanted to share that our engineering team has conducted a thorough investigation and found that this is indeed a unique use case as the practice of specifying 2 sources in the global configuration of the podfile is somewhat of a gap that is not mentioned anywhere in the Cocoapods Official Documentation.

Our investigation into the pod client confirms that this issue stems from what is presumably an intentional lack of support in the pod client that validates this issue is not a vendor or server-side issue.

Now to the great news, we come bearing great news for you and our CocoaPods community.
🐸 JFrog Artifactory’s Virtual repository offering is exactly what you need to overcome this gap.
CocoaPods Virtual Repository is a single source URL you can specify in your podfile that will aggregate any local and/or remote repositories you want. Virtual repositories are available from Artifactory version 7.84.3 and free for all who use our CocoaPods repositories.

Additionally, there is another workaround available 🩹 that is mentioned in Cocoapods Official Documentation.
💡 It’s important to note that it can be a little tedious to maintain but nonetheless usable to overcome the issue.

Please see here under the “Sources” section an example that shows a form of "Explicit source declaration" that allows you to specify a different source for a specific package.

For example;

source 'https://customer.jfrog.io/api/pods/pods' # point to Jfrog CDN

pod 'package', :source => 'https://cdn.cocoapods.org/' # The specific package to will resolve from the explicit specified source

pod 'MyProject' # Will resolve from the main source JFrog CDN

It’s amazing to see how our community and customers find exciting new ways of using CocoaPods and pushing the envelope further. We particularly see these use cases as great opportunities to innovate and collaborate together.

Lastly, I would like to thank @igor-makarov for his collaboration and valuable insight into the CocoaPods community perspective as a maintainer.
I see we are in the same neighborhood so we’d love connect and see how we can collaborate together 😀

Hope this information helps,
Adam

[igor-makarov]

@adam-browning great comment.

I could also suggest two more things that jFrog can do to avoid tricky situations like this. Either one would work on its own, but both are rather simple.

• return an empty text file for empty shards instead of 404
• use an unsharded repo

[santam85]

@adam-browning thanks for your message.

I disagree with your statement that this use case is not contemplated in the CocoaPods documentation: the podspec sources page speaks about "a list of sources", and while the example here uses a Git repo and a single CDN repository, it also doesn't state anywhere that multiple CDNs shouldn't work either; it's understandable how the reader might extrapolate that expectation.

The workaround you suggest implies including each and every single transitive dependency of our projects dependencies in the podfile, I hope you can see why that's not something we are open to do: it would mean a sizable ongoing effort for us, as we maintain more than 60 different applications, and transitive dependencies might change with every library update. I'd appreciate if you also replace our company's repo url from your example above with some generic url and comunicate with us through the official support channels of JFrog instead in a more timely manner.

As per the virtual repository suggestion, that is not going to work unless we mirror the main cocoapods repo. That's also something that we are not willing to do, for 2 main reasons:

• We pay JFrog based on the traffic we generate on our cloud instance, so we want to ship only our private packages from there
• If we were to mirror all of CocoaPods, we'd be in breach of several OSS licenses that do not allow us to redistribute software without proper acknowledgement or permission.

The fix for the issue is implemented in this PR: #771, I'd appreciate more feedback on it, happy to continue working on it until it meets the maintainers expectations. That would solve this problem for us and any other party interested in leveraging JFrog or any other private CocoaPod CDN repo.

Unless that gets merged and released, we'd need JFrog to return an empty text file for empty shards like @igor-makarov suggests, otherwise the virtual repository feature for CocoaPods (which requires us to use the CDN APIs) is effectively unusable.

@igor-makarov can you elaborate more on the merits of an unsharded repo? How would we achieve that?

Thanks both for your reaction and support!

[adam-browning]

Hi @santam85
My sincerest apologies for the URL.
I was under the impression it was a generic example, clearly I was mistaken - I have updated accordingly.