Parity Publish

Parity publish is a tool to publish and manage publishing cargo workspaces to crates.io.

Parity publish aims to be be an all in one tool to manage everything to do with crates, crate.io and publishing releases. Manage maintaining past releases with backports and patch releases. And ensure polkadot-sdk git repro stays in a healthy state when it comes to publishing.

Commands

There are a bunch of commands for doing various things.

Check

Checks crates in the workspace for errors that would prevent publishing. Currently this tool checks for:

• No description (non fatal)
• No license
• Specified README file doesn't exist
• Crate is unpublished on crates.io (non fatal)
• Crate is taken by some one else on crates.io
• Crate is publish = false and is a dependent of a crate with publish = true
• Crate is publish = true but depends on a crate that is publish = false

Publish issues are solved recursively down the dependency chain and up the dependency chain if --recursive is passed.

CI

There is a parity-publish check CI workflow running on https://github.com/paritytech/polkadot-sdk/.

Example

morganamilo@songbird % parity-pubish check

pallet-tx-pause (substrate/frame/tx-pause):
    readme specified in Cargo.toml doesn't exist

bp-test-utils (cumulus/bridges/primitives/test-utils):
    'publish = false' is set but this crate is a dependency of others
    either publish this crate or don't publish the dependants:
        pallet-bridge-grandpa (cumulus/bridges/modules/grandpa)

pallet-bridge-grandpa (cumulus/bridges/modules/grandpa):
    no description

pallet-bridge-parachains (cumulus/bridges/modules/parachains):
    no description

cumulus-client-cli (cumulus/client/cli):
    no description
    no license

polkadot-performance-test (polkadot/node/test/performance-test):
    'publish = false' is set but this crate is a dependency of others
    either publish this crate or don't publish the dependants:
        polkadot-cli (polkadot/cli)

cumulus-test-relay-sproof-builder (cumulus/test/relay-sproof-builder):
    'publish = false' is set but this crate is a dependency of others
    either publish this crate or don't publish the dependants:
        cumulus-primitives-parachain-inherent (cumulus/primitives/parachain-inherent)

parachain-info (cumulus/parachains/pallets/parachain-info):
    no description
    no license
    owned by some one else on crates.io

...

Claim

The claim command looks for unpublished crates in the workspace and publishes empty v0.0.0 releases so that other's can't take our crates between the time they are added to git and a release happens.

The PARITY_PUBLISH_CRATESIO_TOKEN must be set for the tool to be able to claim crates.

CI

There is a parity-publish claim CI workflow running on https://github.com/paritytech/polkadot-sdk/.

Example

morganamilo@songbird % parity-pubish claim

publishing node-executor...
publishing node-testing...
node-inspect is set to not publish
rococo-parachain-runtime is set to not publish
parachain-info exists and is owned by someone else
sp-api-test is set to not publish
sp-application-crypto-test is set to not publish
sp-arithmetic-fuzzer is set to not publish
sp-consensus-sassafras is set to not publish
sp-npos-elections-fuzzer is set to not publish
sp-runtime-interface-test is set to not publish

Status

This command gives a general overview of crate versions and ownership.

Example

morganamilo@songbird % parity-pubish check

Crate                                             Local Ver       crates.io Ver   Owner
bridge-runtime-common                             0.1.0           0.5.0           Parity
bp-header-chain                                   0.1.0           0.5.0           Parity
bp-runtime                                        0.1.0           0.5.0           Parity
frame-support                                     4.0.0-dev       26.0.0          Parity
frame-support-procedural                          4.0.0-dev       21.0.0          Parity
frame-support-procedural-tools                    4.0.0-dev       9.0.0           Parity
frame-support-procedural-tools-derive             3.0.0           10.0.0          Parity
sp-api                                            4.0.0-dev       24.0.0          Parity
sp-api-proc-macro                                 4.0.0-dev       13.0.0          Parity
sp-core                                           21.0.0          26.0.0          Parity
sp-core-hashing                                   9.0.0           13.0.0          Parity
sp-debug-derive                                   8.0.0           12.0.0          Parity
sp-externalities                                  0.19.0          0.23.0          Parity
sp-std                                            8.0.0           12.0.0          Parity
sp-storage                                        13.0.0          17.0.0          Parity
sp-runtime-interface                              17.0.0          22.0.0          Parity
sp-runtime-interface-proc-macro                   11.0.0          15.0.0          Parity
...

Config

This command applies the changes specified in Plan.config to the cargo workspace.

This config file allows you to specify changes you wish to be made every release.

Changed

The changed command shows which crates have changed between two git commits.

Currently this command can detect three kinds of changes:

• File changes - source code and other file have changed
• Manifest changes - the Cargo.toml has changed
• Dependency - a dependency of a crate has changed

Example

morganamilo@songbird % parity-pubish changed FROM TO

bridge-runtime-common (bridges/bin/runtime-common):
    manifest

sp-io (substrate/primitives/io):
    files

sp-keystore (substrate/primitives/keystore):
    manifest

sp-state-machine (substrate/primitives/state-machine):
    files

sp-runtime (substrate/primitives/runtime):
    files

sp-application-crypto (substrate/primitives/application-crypto):
    manifest

polkadot-node-subsystem (polkadot/node/subsystem):
    dependency

polkadot-statement-distribution (polkadot/node/network/statement-distribution):
    dependency

sp-mmr-primitives (substrate/primitives/merkle-mountain-range):
    dependency

Plan

The plan command is the starting point for running a release.

The plan command generates a Plan.toml which outlines all the crates in the workspace, which ones should be published, what version numbers should be used and if any git dependencies should be removed or substituted for registry dependencies.

A plan can be generated a few ways:

# generate a new plan where we release all crates
parity-publish plan --new --all

# generate a new plan where we release crates that changed since the v1.2.3 git tag
parity-publish plan --new --since=v1.2.3

# generate a new plan where we release specific crates
parity-publish plan --new foo bar

--pre=dev.1 can be used to generate pre release version numbers.

A plan file doesn't do much on it's own. It's just a record of everything we would like to be done for the release. Once a plan file is generated, running a release from the plan should be a reproducible process, always ending up with the same end result when applied.

Once a release has been ran, patch releases can be done by running parity-publish plan --patch foo. This will patch version bump the crate foo in the plan ready to be applied.

Example

morganamilo@songbird % parity-pubish plan --new --since=release-crates-io-v1.4.0

looking up crate data, this may take a while....
checking crates....
349 packages changed 146 indirect
calculating order...
looking up crates...
calculating plan...
plan generated 461 packages 349 to publish

Apply

Apply reads the Plan.toml file and makes the changes needed to make crates suitable for publishing to crates.io. It does quite a handful of things, following the crate order in Plan.toml.

parity-publish apply does the following things:

• Removes any reference to dev dependencies from feature values
• Fill in missing descriptions
• Replace workspace path dependencies with crates.io version numbers
• Replaces git dependencies with crates.io releases if there are any
• If there are optional git dependencies with no releases
  • The dependency will be removed completely
  • The features that activate this dependency will be removed completely
  • Features that weakly reference the dependency will be removed from the feature values
  • Any references to the features that have been removed will be removed recursively
  • Any crates that require the feature unconditionally are removed from the workspace

Once the changes have been applied, they can be double checked and commited. Then parity-publish apply --publish will start publishing the crates. It takes a few minutes per each crate to publish. With the amount of crates we have this takes an extremely long amount of time. 5 minutes x 350 crates = 29 hours. This task needs to be let run overnight and then some.

Post release

After the initial plan has been generated and release pushed out, the plan file can then be patch bumped and applied as needed for backports and maintenance.

git cherry-pick BACKPORT
parity-publish plan --patch foo bar
parity-publish apply --publish

Example

morganamilo@songbird % parity-pubish apply --publish

rewriting manifests...
Publishing 349 packages (0 skipped)
(1/349) publishing binary-merkle-tree-12.0.0...

Limitations / Future plans

Changes

Currently the diffing code to calculate changes between commits is not yet perfect but is being worked on. The idea is to more semantically diff Manifest files to see if what changed is breaking or not and if the change warrants a release.

Semver

Currently parity-publish bumps the major version on every release (apart from --patch bumping). The plan is to eventually make this tool semver aware via https://github.com/paritytech/prdoc. This would not only be a lot less ugly and annoying for users, but will allow us to skip pushing out new releases for dependency changes.