[long-running] Changing the storage contract requires 2x publishes to crates.io

[slawlor]

The problem

Due to the change in the storage contract, there are delays in the time between publishing akd to crates.io and the new version being available to cargo to build akd_mysql.

This happens because when breaking the contract between akd and akd_mysql, akd_mysql will rely on the latest version of akd which doesn't exist in crates.io until it's published. For example, if both crates are at v1.0.0, and we change the contract, and publish v2.0.0, akd_mysql depends on akdv2.0.0 which was _just_published. There appears to be a race in crates.io that the package isn't instantly available after publish, so the cargo publish --dry-run balks saying akdv2.0.0 can't be found but since we don't have a multi-publish, akdv2.0.0 is published and cannot be reverted. Therefore we're in a "broken" state where half of the crates are published.

Current workaround

The current workaround is we need to publish akd with the update, akd_mysql will fail, and we version bump both and publish again. The 2nd time crates.io has updated and the index is updated, akd_mysql can build and publish.

See PRs tagged with publish_race to identify instances where this is needed.

Reason to fix

This is ugly and just clutters crates.io with partial versions of akd with sometimes matching akd_mysql crates.

[slawlor]

Some ideas to investigate

1. Delay trying to dry-run build akd_mysql via some timeout, letting crates.io update in time. Or perhaps some polling loop?
2. Maybe there's a flag to force a cargo update we're missing?

[slawlor]

An example of the publish CI pipeline failing for this reason

[eozturk1]

Does the PR close the issue?

[slawlor]

Unfortunately not, that PR is an example of the problem.

[eozturk1]

It seems there is no official solution to this. This issue suggests querying the published index at the Github repo crates-io.index as a hack but the corresponding akd index seems to be missing there. Any ideas why?

[slawlor]

OK so a little debugging, it works for akd_mysql with

https://raw.githubusercontent.com/rust-lang/crates.io-index/master/ak/d_/akd_mysql

but the url you put still has the d_ even though the crate akd doesn't have a _ in its name. There must be something wrong with the construction of that URI, I'm not sure where they build this but we need to see how they handle < 4 char long crate names. This could be a good potential solution however, with something like an exponential backoff in the polling times.

[slawlor]

Ah found it searching the crates.io-index repo. There's a special handling for 3-char names.

https://github.com/rust-lang/crates.io-index/search?q=akd

yields a pointer to

https://raw.githubusercontent.com/rust-lang/crates.io-index/master/3/a/akd

[slawlor]

It seems there is no official solution to this. This issue suggests querying the published index at the Github repo crates-io.index as a hack but the corresponding akd index seems to be missing there. Any ideas why?

@eozturk1 good catch on finding the related issue! It indeed is the same problem referenced there.

[eozturk1]

Ah found it searching the crates.io-index repo. There's a special handling for 3-char names.

https://github.com/rust-lang/crates.io-index/search?q=akd

yields a pointer to

https://raw.githubusercontent.com/rust-lang/crates.io-index/master/3/a/akd

@slawlor, great find! Since we have akd's path now, we can implement a similar solution, working on it.

[slawlor]

@eozturk1 Awesome! It'll need to be in the publish workflow: https://github.com/novifinancial/akd/blob/main/.github/workflows/publish.yml, which maybe it'd be worthwhile to make a version of the publish workflow which publishes a series of "beta" releases while you test this (e.g. for debugging)? We can land that new pipeline that could run with each merge to master w/tag for example, rather than on real releases.

[eozturk1]

No issues in the latest publish operation. Closing! :)