[Templates] Systematic way to synchronize the templates in this repo and substrate-developer-hub

[kianenigma]

https://github.com/substrate-developer-hub/substrate-node-template and https://github.com/substrate-developer-hub/substrate-parachain-template are both in the substrate-developer-hub org, and as the commit dates suggest not very up to date.

The existence of substrate-developer-hub in itself is questionable in my opinion. It is not well maintained, and I much rather everything to be in /paritytech, and ideally in polkadot-sdk repo.

But, whatever we decide to do here, we should also do them in a backwards compatible way. substrate-developer-hub is used in many external resources and should be kept as an archive.

Options include:

0. Rebuild a bot that tries to update deps in the other repos, and/or a human owner who makes sure they are kept up to date.
1. Archive the external templates, only use the ones here. Downside is that they will not be easy to clone.
2. Archive the old ones, make two new repos in this org, use git submodules to make sure the code remains up to date, but is clone-able separately.

Moreover, this also gives the opportunity to re-consider our templates. I personally think we should scrap the node-template as it is mainly targeted towards "solo-chain development". The two main templates we want should be:

1. A minimal template that has almost nothing, and is mainly there to help you test your custom FRAME pallet.
2. A parachain template.

cc @gupnik

Tasks

Beta Give feedback

1. [EngAut] Automate Sync/Release the polakdot-sdk templates eng-automation#6
   +0
   

Outcome:

• https://github.com/paritytech/polkadot-sdk-minimal-template
• https://github.com/paritytech/polkadot-sdk-solochain-template
• https://github.com/paritytech/polkadot-sdk-parachain-template

Next Steps

#5242

[nuke-web3]

Related to paritytech/polkadot-sdk-docs#67 (comment) . I am an advocate to archive the who devhub org as soon as a MVP for the Rust Docs vision to replace it is in place (perhaps unarchive to update links/redirects when ready to promote the maintained version(s)).

IIRC (although I didn't deeply eval the tooling myself) the CI scripts to modify a few files and make a top level Cargo.toml was fairly simple, likely easy to update for this. One other path I would suggest in making a tool (perhaps something for cargo-generate or a just file w/ dual purposed CI scripts) that would let anyone derive a template for themselves at any commit/tag/release. CI running tool this and making a separate repo that is a template with tags to clone at a version would.

RE: deprecation of even any MVP solo-chain option seems anti-ethos to what I understand Substrate to be: a blockchain building framework. Not exclusively a parachain building one. I sure hope that the SDK doesn't move to try and make it difficult to use Substrate as a product on it's own. That and I would argue maintaining a solo-chain option has benefits to learning (consensus, tokenomics, governance, etc.) for users on how a blockchain works more than just a dev-mode like FRAME engine. A MVP template that is a fully functional blockchain is a great way to read and play with the "real thing" I would surely miss. I would hope maintaining this template would be a relatively light lift, as the relay-chain's needs as a solo-chain are quite closely related to a MVP solo-chain template.

[kianenigma]

After brainstorming with @gupnik about this a bit more, using any type of git-based system like submodules and subtrees totally defies the purpose as these nested templates cannot use relative paths in their Cargo.toml and must use the released crates, so being in the same file tree does not help.

What we suggest doing is the following:

1. bring all templates of polkadot-sdk under one folder. /templates/solo-chain, /templates/parachain and /templates/minimal are the main 3. These are the source of truth and always kept up to date.
2. Create repositories corresponding for each, called polkadot-sdk-solo-chain-template, polkadot-sdk-solo-parachain and polkadot-sdk-minimal-tempalte.
3. Write a script that receives a commit hash somewhere in the stable branch as input, and mirrors the templates into their corresponding repo. This script needs to replace all Cargo.toml files to use the released crates.
4. We run the above script upon each update of stable branch, i.e. every 3 month.

[kianenigma]

now that #3460 is merged, before attempting to automate the rest, this is what is more or less needed.

// one-off operations
for each sub-folder t in /templates 
    // eg. paritytech/polkadot-sdk-solochain-template
    // eg. paritytech/polkadot-sdk-parachain-template
    create a repository paritytech/polkadot-sdk-$t-template

on event  new polkadot-sdlk release with version x.y.z do
    for t in /templates
        // important to replace as there might be files removed as well. 
        replace the entire code in paritytech/polkadot-sdk-$t-template with content of templates/$t at the commit of polakadot-sdk#x.y.z
        in each `Cargo.toml` in paritytech/polkadot-sdk-$t-template
            replace each 'path' dep with crates.io versions polakadot-sdk#x.y.z
            replace version number x.y.z
        end for
    if compiles + cargo test passes:
        push to master
    else: 
        open a PR, should never happen.
end do

This should guarantee that there is a new template that will 100% work from the versions maintained in this repo.

Then, we should archive all the ones in https://github.com/substrate-developer-hub/

[Polkadot-Forum]

This issue has been mentioned on Polkadot Forum. There might be relevant details there:

https://forum.polkadot.network/t/paritytech-update-for-april/7646/1

[bkchr]

@rzadp as part of synchronizing the template, could you please ensure that any reference to docify in the code of the templates is removed? This includes the dependency in the Cargo.toml and the macro attributes.

[kianenigma]

@rzadp as part of synchronizing the template, could you please ensure that any reference to docify in the code of the templates is removed? This includes the dependency in the Cargo.toml and the macro attributes.

docify being a dependency of many production crates is something that we were aware of as a downside with Sam when we built this last year. I would actually not put so much effort into removing it from some non-production templates. What I would be worried about is mainly supply chain attacks.

[rzadp]

Looking at master, I see that the minimal template references polkadot-sdk-docs - which is not being published.

So I don't have a choice but to remove this reference, otherwise it will not build outside of the monorepo.

[bkchr]

I would actually not put so much effort into removing it from some non-production templates

I mean it should be mainly a simple sed search&replace. I find it weird that if someone wants to get a minimal template and then has their stuff in it that belongs docs in the SDK repo. This will just confuse people IMO.

[kianenigma]

This was added in #4119 and I kinda disagree with it even to stay in master. @gupnik can you adjust accordingly?

I am of the opinion that polkadot-sdk-docs should bring in a lot of crates, but should not be the dependent on any crate. One issue with making polkadot-sdk-docs dependent of any crate is that it would prevent the reverse from happening due to circular dependencies.

[gupnik]

I am of the opinion that polkadot-sdk-docs should bring in a lot of crates, but should not be the dependent on any crate.

IMO, the doc-only crate in templates should ideally link to polkadot-sdk-docs for all references so that the developer knows how to find further details.

One issue with making polkadot-sdk-docs dependent of any crate is that it would prevent the reverse from happening due to circular dependencies.

I was not imagining thepolkadot-sdk-docs crate to ever pull in the doc-only template crate, thus, avoiding the circular dependency.

Happy to adjust if we have other ways to reference such items.

[kianenigma]

IMO, the doc-only crate in templates should ideally link to polkadot-sdk-docs for all references so that the developer knows how to find further details.

Okay this sounds good. If need be, then we remove the reverse link (polkadot-sdk-docs -> xxx-template). The templates will soon live in their own repo anyways, and the internal copies are basically just for staging.

[kianenigma]

I see a lot of great progress here, thanks @rzadp and @gupnik! Some action items that I see is:

(most of these changes are applied to this repository, and upon next release they would get synchronized)

1. Update the README of all 3 templates. Make them look nice and shiny. It is not that hard, and fun to do, I recently did it too ;)
2. Make the repos be homogenous. They should all have the same engineering components, such as .env files, .editorconfig, .rustfmt.toml
3. As noted by @PierreBesson good github workflows would be great. These should be directly added to the template repo, and they should not be synchronized. Some ideas are:

• Action to build the rust-docs
• Action to run cargo test
• Action to run cargo clippy
• Similarly, Dockerfile for all would be great, and would allow parity to also test these, if need be.
• We should put a lot more focus on the parachain template; add links to the new agile-coretime guides being made by @CrackTheCode016 and @DrW3RK, maybe even use it there.
• The parachain template still uses polkadot-launch, which tbh is a shame. Should be updated to zombienet.
• Update the list of templates here: https://paritytech.github.io/polkadot-sdk/master/polkadot_sdk_docs/polkadot_sdk/templates/index.html
• FINALLY mark the existing templates as archive and forward them here.
• Mark the new templates as actually Template in gh so others can use them as a template :)
• Update all templates to use frame + polkadot-sdk-umbrella crate.
• Remove the solochain template.

Once #5 and omni-node is done, all templates should be updated with an option to not build the node side and instead use the omni-node.

[rzadp]

(github workflows) should be directly added to the template repo, and they should not be synchronized.

Agree. I have already added a simple starter github workflow to the templates with cargo check && cargo test. It is directly in the repo, not synchronized - so can be further modified with direct PRs to the templates.

[DrW3RK]

@kianenigma I think it makes a lot of sense to use Docify and create docs for the parachain template in this repo. We put the guides on the Wiki as an interim resource, and our end goal was to have the content in the Polkadot SDK repo. We can start adding content here right away.

[rzadp]

Sorry @rzadp, can you adjust? thank you!

No problem, updated the PR to Unlicense (for all 3 templates).

[rzadp]

Just as a small update, all three templates have been updated to 1.13.0 last week, which include Dockerfiles.
The Dockerfiles are being built on CI in the individual repos.

[rzadp]

Hey @kianenigma, I see you have added more action items - could you elaborate a bit?

For example, Remove the solochain template - why is that?

I could continue to help if I can. Is the umbrella crate ready to be used, or is that for the future?

[kianenigma]

Update all templates to use frame + polkadot-sdk-umbrella crate.

Is something that I'd say @gupnik should do.

Remove the solochain template.

I am less sure about this, it is an idea for now. Let's ignore it.

FINALLY mark the existing templates as archive and forward them here.

We are ready to do this IMO.

[kianenigma]

closing in favor of #5242