Zombie tests build pipeline rework (#2159)

* add usrmerge to zombie-builder build step

* new zombie_builder.Dockerfile

* testing ci for zombie-builder

* Replace minideb with the official Rust image.

* testing images for zombie_tester

* test pkg dependencies for zombie-builder

* kagome-dev minideb-testing Docker

* add libraries for zombie_builder docker image

* enable cache on zombie-builder step

* zombie builder image group update

* change repo for zombie builder, separate builder steps

* add working directory

* update image path for zombie-builder

* building_binary step for zombie builder

* github inputs test

* test sccache

* Exit code disabled

* check sccache working

* Revert "check sccache working"

This reverts commit ab18767.

* make dirs when CI zombie-builder starts

* show build result for polkadot_binary

* change runner for zombie-builder

* zombie builder, change paths

* build apt package pipeline

* docker-builder - add debug info

* zombie-builder ci architecture fix

* New kagome dev minideb builder

* temporarily ci changes (autotests)

* enable kagome-builder-deb workflow

* zombie_builder and zombie_tester updated

* zombie_builder syntax fixes

* zombie builder - build logic variable added

* save zombie-builder env to file and restore it

* New build kagome Makefile for kagome-dev

* Zombie-Tests kagome build, release by default

* new build kagome pipeline - activate cache

* enable show info when build deb packet

* update dependencies for packet

* test update

* Check package version exists in build steps

* show package version in build step

* add date to version apt package name

* Test zombie-tests job

* update run zombienet test

* update Google Auth for zombie-tests ci

* Run zombie-tests on actions-runner-controller

* wasmedge get SHA script

* Runtime cache build packet - first iteration

* ci runtime build fix

* Kagome runtime ci update

* zombie test -> ubuntu latest

* Revert "zombie test -> ubuntu latest"

This reverts commit ff582d2.

* add hunter cache to safe.directory

* CI change cache using config

* Enable swap on Ubuntu zombie-tests

* change cache logic in kagome build step

* Reduce kagome validators in 0001 test

* change swap size - 4G (for ubuntu runners)

* update cache save settings - build kagome step

* Fix zndsl

* zombie-tests runs on actions-runner-controller

* disable swap

* disable broken zombie tests

* zombie test - kagome build - google cache

* disable broken zombie-tests

* zombie tests - disable always save cache, change cache action

* return github action version

* change gcp github action version

* testing actions step

* return github cache save

* select working zombie tests

* runtime cache build update

* Rebuild runtime option in CI

* update zombienet simlinks for tester image

* Zombie-Tests runs-on -> ubuntu-latest

* Build runtime -> actions-runner-controller

* Clean old files, clean Zombie tests Makefile, change version numeric

* Zombienet Makefile new line fix

* Clear Kagome docker Build makefile

* use default kagome_dev version in runtime_cache ci build

* runtime cache version ci fix

* Add kagome-crates dependency

* Switch to tag

* Update main build pipeline (zombie tests new pipeline)

* fix ci errors in test.yaml

* Build type for Zombie tests added

* code cleanup

* Zombie tests in main pipeline -> actions-runner-controller

* change mount dirs for docker build

* cleaning cache and default build typo for main build workflow

* disable external project

* add CI env to workflows

* -Werror param support for CI building

* add check -Werror compiler support

* Change the order of applying -Werror

* add_flag cmake function update

---------

Co-authored-by: Kirill Azovtsev <k.azovtsev@qdrvm.io>
Co-authored-by: kamilsa <kamilsa16@gmail.com>
Co-authored-by: Zak Fein <zak@zerg.su>
Co-authored-by: Harrm <abrehchs@gmail.com>

.github/workflows/build.yaml

@@ -0,0 +1,22 @@
+#
+# Copyright Quadrivium LLC
+# All Rights Reserved
+# SPDX-License-Identifier: Apache-2.0
+#
+
+name: Manual Workflow Dispatch
+
+on:
+  workflow_dispatch:
+
+jobs:
+  example-job:
+    runs-on: ubuntu-latest
+
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@v3
+
+      - name: Print template message
+        run: echo "This is a template for testing workflows in branches."
+

.github/workflows/kagome-builder-deb.yaml

@@ -0,0 +1,102 @@
+#
+# Copyright Quadrivium LLC
+# All Rights Reserved
+# SPDX-License-Identifier: Apache-2.0
+#
+
+name: Kagome Builder | Debian (Bookworm)
+
+on:
+  #  schedule:
+  #    - cron: '0 2 * * 0' # Every Sunday at 2:00 AM
+  workflow_dispatch:
+    inputs:
+      kagome_image_rebuild:
+        description: 'Rebuild Kagome Builder Minideb image'
+        required: false
+        default: 'false'
+        type: 'boolean'
+      runtime_cache_rebuild:
+        description: 'Rebuild Kagome Runtime Cache'
+        required: false
+        default: 'false'
+        type: 'boolean'
+
+env:
+  DOCKER_REGISTRY_PATH: ${{ secrets.GCP_REGISTRY }}/${{ secrets.GCP_PROJECT }}/kagome-dev/
+
+jobs:
+  build_polkadot_builder:
+    runs-on: ubuntu-latest
+    timeout-minutes: 60
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@v4
+
+      - name : "Authenticate with Google Cloud"
+        uses: 'google-github-actions/auth@v2'
+        with:
+          credentials_json: ${{ secrets.GCP_SERVICE_ACCOUNT_KEY }}
+
+      - name: "Set up Cloud SDK"
+        uses: 'google-github-actions/setup-gcloud@v2'
+
+      - name: "Configure Docker for GCR"
+        run: |
+          gcloud auth configure-docker --quiet
+          gcloud auth configure-docker ${{ secrets.GCP_REGISTRY }} --quiet
+
+      - name: "Check if image tag exists"
+        id: check-tag
+        working-directory: ./housekeeping/docker/kagome-dev
+        run: |
+          IMAGE_TAG=$(grep '^MINIDEB_IMAGE' Makefile | awk -F '[@:]' '{print $3}' | cut -c 1-7)_rust-$(grep '^RUST_VERSION' Makefile | sed 's/.*?= //')
+          KAGOME_BUILDER_DEB_IMAGE=${DOCKER_REGISTRY_PATH}kagome_builder_deb:${IMAGE_TAG}
+          TAG_EXISTS=$(docker manifest inspect ${KAGOME_BUILDER_DEB_IMAGE} > /dev/null 2>&1 && echo "true" || echo "false")
+          echo "exists=$TAG_EXISTS" >> $GITHUB_ENV
+          echo "kagome_builder_deb=KAGOME_BUILDER_DEB_IMAGE" >> GITHUB_OUTPUT
+
+      - name: "Build Polkadot Builder"
+        working-directory: ./housekeeping/docker/kagome-dev
+        if: env.exists == 'false' || github.event.inputs.kagome_image_rebuild == 'true'
+        run: make kagome_builder_deb \
+          DOCKER_REGISTRY_PATH=${DOCKER_REGISTRY_PATH}
+
+      - name: "Push Polkadot Builder"
+        working-directory: ./housekeeping/docker/kagome-dev
+        if: env.exists == 'false' || github.event.inputs.kagome_image_rebuild == 'true'
+        run: make kagome_builder_deb_push \
+          DOCKER_REGISTRY_PATH=${DOCKER_REGISTRY_PATH}
+
+  build_kagome_runtime:
+    runs-on: [ actions-runner-controller ]
+    timeout-minutes: 60
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@v4
+
+      - name : "Authenticate with Google Cloud"
+        uses: 'google-github-actions/auth@v2'
+        with:
+          credentials_json: ${{ secrets.GCP_SERVICE_ACCOUNT_KEY }}
+
+      - name: "Set up Cloud SDK"
+        uses: 'google-github-actions/setup-gcloud@v2'
+
+      - name: "Configure Docker for GCR"
+        run: |
+          gcloud auth configure-docker --quiet
+          gcloud auth configure-docker ${{ secrets.GCP_REGISTRY }} --quiet
+
+      - name: "Build Runtime Cache"
+        working-directory: ./housekeeping/docker/kagome-dev
+        if: github.event.inputs.runtime_cache_rebuild == 'true'
+        run: make runtime_cache \
+          DOCKER_REGISTRY_PATH=${DOCKER_REGISTRY_PATH}
+
+      - name: "Push Runtime Cache"
+        working-directory: ./housekeeping/docker/kagome-dev
+        if: github.event.inputs.runtime_cache_rebuild == 'true'
+        run: make upload_apt_package_runtime \
+          DOCKER_REGISTRY_PATH=${DOCKER_REGISTRY_PATH}
+

.github/workflows/telegram_notify.yml

@@ -1,3 +1,9 @@
+#
+# Copyright Quadrivium LLC
+# All Rights Reserved
+# SPDX-License-Identifier: Apache-2.0
+#
+
 name: Telegram Notification
 
 on: