Synchronize templates (#4040)

- Progresses #3155 ### What's inside A job, that will take each of the three [templates](https://github.com/paritytech/polkadot-sdk/tree/master/templates), yank them out of the monorepo workspace, and push to individual repositories ([1](https://github.com/paritytech/polkadot-sdk-minimal-template), [2](https://github.com/paritytech/polkadot-sdk-parachain-template), [3](https://github.com/paritytech/polkadot-sdk-solochain-template)). In case the build/test does not succeed, a PR such as [this one](paritytech-stg/polkadot-sdk-solochain-template#2) gets created instead. I'm proposing a manual dispatch trigger for now - so we can test and iterate faster - and change it to fully automatic triggered by releases later. The manual trigger looks like this: <img width="340px" src="https://github.com/paritytech/polkadot-sdk/assets/12039224/e87e0fda-23a3-4735-9035-af801e8417fc"/> ### How it works The job replaces dependencies [referenced by git](https://github.com/paritytech/polkadot-sdk/blob/d733c77ee2d2e8e2d5205c552a5efb2e5b5242c8/templates/minimal/pallets/template/Cargo.toml#L25) with a reference to released crates using [psvm](https://github.com/paritytech/psvm). It creates a new workspace for the template, and adapts what's needed from the `polkadot-sdk` workspace. ### See the results The action has been tried out in staging, and the results can be observed here: - [minimal stg](https://github.com/paritytech-stg/polkadot-sdk-minimal-template/) - [parachain stg](https://github.com/paritytech-stg/polkadot-sdk-parachain-template/) - [solochain stg](https://github.com/paritytech-stg/polkadot-sdk-solochain-template/) These are based on the `1.9.0` release (using `release-crates-io-v1.9.0` branch).
paritytech · Apr 12, 2024 · c963dc2 · c963dc2
1 parent 480d5d0
commit c963dc2
Showing 1 changed file with 158 additions and 0 deletions.
diff --git a/.github/workflows/sync-templates.yml b/.github/workflows/sync-templates.yml
@@ -0,0 +1,158 @@
+name: Synchronize templates
+
+
+# This job is used to keep the repository templates up-to-date.
+# The code of the templates exist inside the monorepo, and upon releases we synchronize the repositories:
+# - https://github.com/paritytech/polkadot-sdk-minimal-template
+# - https://github.com/paritytech/polkadot-sdk-parachain-template
+# - https://github.com/paritytech/polkadot-sdk-solochain-template
+#
+# The job moves the template code out of the monorepo,
+# replaces any references to the monorepo workspace using psvm and toml-cli,
+# checks that it builds successfully,
+# and commits and pushes the result to each respective repository.
+# If the build fails, a PR is created instead for manual inspection.
+
+
+on:
+  # A manual dispatch for now - automatic on releases later.
+  workflow_dispatch:
+    inputs:
+      crate_release_version:
+        description: 'A release version to use, e.g. 1.9.0'
+        required: true
+
+
+jobs:
+  sync-templates:
+    runs-on: ubuntu-latest
+    strategy:
+      fail-fast: false
+      matrix:
+        template: ["minimal", "solochain", "parachain"]
+    env:
+      template-path: "polkadot-sdk-${{ matrix.template }}-template"
+    steps:
+
+      # 1. Prerequisites.
+
+      - name: Configure git identity
+        run: |
+          git config --global user.name "Template Bot"
+          git config --global user.email "163342540+paritytech-polkadotsdk-templatebot[bot]@users.noreply.github.com"
+      - uses: actions/checkout@v3
+        with:
+          path: polkadot-sdk
+          ref: "release-crates-io-v${{ github.event.inputs.crate_release_version }}"
+      - name: Generate a token for the template repository
+        id: app_token
+        uses: actions/create-github-app-token@v1.9.3
+        with:
+          owner: "paritytech"
+          repositories: "polkadot-sdk-${{ matrix.template }}-template"
+          app-id: ${{ secrets.TEMPLATE_APP_ID }}
+          private-key: ${{ secrets.TEMPLATE_APP_KEY }}
+      - uses: actions/checkout@v3
+        with:
+          repository: "paritytech/polkadot-sdk-${{ matrix.template }}-template"
+          path: "${{ env.template-path }}"
+          token: ${{ steps.app_token.outputs.token }}
+      - name: Install toml-cli
+        run: cargo install --git https://github.com/gnprice/toml-cli --rev ea69e9d2ca4f0f858110dc7a5ae28bcb918c07fb # v0.2.3
+      - name: Install Polkadot SDK Version Manager
+        run: cargo install --git https://github.com/paritytech/psvm --rev c41261ffb52ab0c115adbbdb17e2cb7900d2bdfd psvm # master
+      - name: Rust compilation prerequisites
+        run: |
+          sudo apt update
+          sudo apt install -y \
+            protobuf-compiler
+          rustup target add wasm32-unknown-unknown
+          rustup component add rustfmt clippy rust-src
+      
+      # 2. Yanking the template out of the monorepo workspace.
+
+      - name: Use psvm to replace git references with released creates.
+        run: find . -type f -name 'Cargo.toml' -exec psvm -o -v ${{ github.event.inputs.crate_release_version }} -p {} \;
+        working-directory: polkadot-sdk/templates/${{ matrix.template }}/
+      - name: Create a new workspace Cargo.toml
+        run: |
+          cat << EOF > Cargo.toml
+          [workspace.package]
+          license = "MIT-0"
+          authors = ["Parity Technologies <admin@parity.io>"]
+          homepage = "https://substrate.io"
+
+          [workspace]
+          members = [
+              "node",
+              "pallets/template",
+              "runtime",
+          ]
+          resolver = "2"
+          EOF
+        shell: bash
+        working-directory: polkadot-sdk/templates/${{ matrix.template }}/
+      - name: Update workspace configuration
+        run: |
+          set -euo pipefail
+          # toml-cli has no overwrite functionality yet, so we use temporary files.
+          # We cannot pipe the output straight to the same file while the CLI still reads and processes it.
+
+          toml set templates/${{ matrix.template }}/Cargo.toml 'workspace.package.repository' "https://github.com/paritytech/polkadot-sdk-${{ matrix.template }}-template.git" > Cargo.temp
+          mv Cargo.temp ./templates/${{ matrix.template }}/Cargo.toml
+
+          toml set templates/${{ matrix.template }}/Cargo.toml 'workspace.package.edition' "$(toml get --raw Cargo.toml 'workspace.package.edition')" > Cargo.temp
+          mv Cargo.temp ./templates/${{ matrix.template }}/Cargo.toml
+
+          toml get Cargo.toml 'workspace.lints' --output-toml >> ./templates/${{ matrix.template }}/Cargo.toml
+
+          toml get Cargo.toml 'workspace.dependencies' --output-toml >> ./templates/${{ matrix.template }}/Cargo.toml
+        working-directory: polkadot-sdk
+      - name: Print the result Cargo.tomls for debugging
+        if: runner.debug == '1'
+        run: find . -type f -name 'Cargo.toml' -exec cat {} \;
+        working-directory: polkadot-sdk/templates/${{ matrix.template }}/
+
+      - name: Clean the destination repository
+        run: rm -rf ./*
+        working-directory: "${{ env.template-path }}"
+      - name: Copy over the new changes
+        run: |
+          cp -r polkadot-sdk/templates/${{ matrix.template }}/* "${{ env.template-path }}/"
+
+      # 3. Verify the build. Push the changes or create a PR.
+
+      # We've run into out-of-disk error when compiling in the next step, so we free up some space this way.
+      - name: Free Disk Space (Ubuntu)
+        uses: jlumbroso/free-disk-space@54081f138730dfa15788a46383842cd2f914a1be # 1.3.1
+        with:
+          android: true # This alone is a 12 GB save.
+          # We disable the rest because it caused some problems. (they're enabled by default)
+          # The Android removal is enough.
+          dotnet: false
+          haskell: false
+          large-packages: false
+          swap-storage: false
+
+      - name: Check if it compiles
+        id: check-compilation
+        run: cargo check && cargo test
+        working-directory: "${{ env.template-path }}"
+        timeout-minutes: 90
+      - name: Create PR on failure
+        if: failure() && steps.check-compilation.outcome == 'failure'
+        uses: peter-evans/create-pull-request@5b4a9f6a9e2af26e5f02351490b90d01eb8ec1e5 # v5
+        with:
+          path: "${{ env.template-path }}"
+          token: ${{ steps.app_token.outputs.token }}
+          add-paths: |
+            ./*
+          title: "[Don't merge] Update the ${{ matrix.template }} template"
+          body: "The template has NOT been successfully built and needs to be inspected."
+          branch: "update-template/${{ github.event_name }}"
+      - name: Push changes
+        run: |
+          git add -A .
+          git commit --allow-empty -m "Update template triggered by ${{ github.event_name }}"
+          git push
+        working-directory: "${{ env.template-path }}"