scripts, CI, workflows: remove submodule handling

Remove submodule references from build scripts, devcontainer, .github workflows The change includes a transient change to emerge-gitclone-9999.ebuild for handling the new unified scripts/ repo. Co-authored-by: Krzesimir Nowak <knowak@microsoft.com>
flatcar · Apr 3, 2023 · 439f118 · 439f118
1 parent bbd8370
commit 439f118
Show file tree

Hide file tree

Showing 26 changed files with 70 additions and 266 deletions.
diff --git a/.github/workflows/ci.yaml b/.github/workflows/ci.yaml
@@ -11,24 +11,6 @@ on:
           Space-separated vendor formats to build.
         required: true
         default: qemu_uefi
-      portage_remote:
-        description: |
-          The remote we should pull portage-stable from. This defaults to whatever the submodule is set to in this repo.
-          If triggered by a change in the portage repo, please set this to the remote which is proposing a change.
-        required: false
-      portage_ref:
-        description: |
-          This is the ref we will use to pull the changes from the portage_remote.
-        required: false
-      coreos_remote:
-        description: |
-          The remote we should pull coreos-overlay from. This defaults to whatever the submodule is set to in this repo.
-          If triggered by a change in the portage repo, please set this to the remote which is proposing a change.
-        required: false
-      coreos_ref:
-        description: |
-          This is the ref we will use to pull the changes from the coreos_remote.
-        required: false
 
 concurrency:
   group: ${{ github.workflow }}-${{ github.head_ref || github.ref_name }} 
@@ -75,7 +57,6 @@ jobs:
         with:
           path: scripts
           fetch-depth: 0
-          submodules: true
 
       # Hack alert: actions/checkout will check out the (disjunct) merge commit of a PR
       #  instead of its head commit. That commit is not connected to any branch.
@@ -96,23 +77,10 @@ jobs:
         shell: bash
         run: |
           arch="${{ matrix.arch }}"
-          COREOS_REMOTE=""
-          COREOS_REF=""
-          PORTAGE_REMOTE=""
-          PORTAGE_REF=""
-          IMAGE_FORMATS="qemu_uefi"
+          echo "arch=${arch}" >> $GITHUB_ENV
 
-          [ -z "${{ github.event.inputs.coreos_remote }}" ] || COREOS_REMOTE="${{ github.event.inputs.coreos_remote }}"
-          [ -z "${{ github.event.inputs.coreos_ref }}" ] || COREOS_REF="${{ github.event.inputs.coreos_ref }}"
-          [ -z "${{ github.event.inputs.portage_remote }}" ] || PORTAGE_REMOTE="${{ github.event.inputs.portage_remote }}"
-          [ -z "${{ github.event.inputs.portage_ref }}" ] || PORTAGE_REF="${{ github.event.inputs.portage_ref }}"
+          IMAGE_FORMATS="qemu_uefi"
           [ -z "${{ github.event.inputs.image_formats }}" ] || IMAGE_FORMATS="${{ github.event.inputs.image_formats }}"
-
-          echo "arch=${arch}" >> $GITHUB_ENV
-          echo "COREOS_REMOTE=${COREOS_REMOTE}" >> $GITHUB_ENV
-          echo "COREOS_REF=${COREOS_REF}" >> $GITHUB_ENV
-          echo "PORTAGE_REMOTE=${PORTAGE_REMOTE}" >> $GITHUB_ENV
-          echo "PORTAGE_REF=${PORTAGE_REF}" >> $GITHUB_ENV
           echo "IMAGE_FORMATS=${IMAGE_FORMATS}" >> $GITHUB_ENV
 
           # Artifact root for images and torcx tarball as seen from within the container
@@ -124,29 +92,6 @@ jobs:
           #  this with its IP address.
           echo "TORCX_TESTS_PACKAGE_URL=http://localhost:12345" >> $GITHUB_ENV
 
-      - name: Checkout submodules
-        shell: bash
-        run: |
-          if [ "${COREOS_REMOTE}" != "" -a  "${COREOS_REF}" != "" ]
-          then
-            REMOTE="${COREOS_REMOTE}"
-            REPO_PATH="sdk_container/src/third_party/coreos-overlay"
-            [[ "$REMOTE" == "https:*" ]] || REMOTE="https://github.com/${COREOS_REMOTE}"
-            git -C "$REPO_PATH" remote add test "$REMOTE"
-            git -C "$REPO_PATH" fetch test
-            git -C "$REPO_PATH" checkout "test/${COREOS_REF}"
-          fi
-
-          if [ "${PORTAGE_REMOTE}" != "" -a  "${PORTAGE_REF}" != "" ]
-          then
-            REMOTE="${PORTAGE_REMOTE}"
-            REPO_PATH="sdk_container/src/third_party/portage-stable"
-            [[ "$REMOTE" == "https:*" ]] || REMOTE="https://github.com/${PORTAGE_REMOTE}"
-            git -C "$REPO_PATH" remote add test "$REMOTE"
-            git -C "$REPO_PATH" fetch test
-            git -C "$REPO_PATH" checkout "test/${PORTAGE_REF}"
-          fi 
-
       - name: Build packages
         shell: bash
         run: |

diff --git a/.github/workflows/run-kola-tests.yaml b/.github/workflows/run-kola-tests.yaml
@@ -58,7 +58,6 @@ jobs:
         with:
           path: scripts
           fetch-depth: 0
-          submodules: true
 
       # Hack alert: actions/checkout will check out the (disjunct) merge commit of a PR
       #  instead of its head commit. That commit is not connected to any branch.
@@ -356,4 +355,4 @@ jobs:
         if: ${{ github.event_name == 'pull_request' }}
         uses: mshick/add-pr-comment@v2
         with:
-          message-path: "test-results.md"
+          message-path: "scripts/test-results.md"
diff --git a/README.md b/README.md
@@ -11,12 +11,10 @@ The SDK can be used to
 
 [flatcar-docs]: https://www.flatcar.org/docs/latest/reference/developer-guides/sdk-modifying-flatcar/
 
-# Using the scripts repository: submodules and tags
+# Using the scripts repository
 
 The repository is meant to be the entry point for Flatcar builds and development.
-For building packages, there are 2 additional repositories, [coreos-overlay](https://github.com/flatcar/coreos-overlay) and [portage-stable](https://github.com/flatcar/portage-stable), which contain all packages' `ebuild` (build configuration) files.
-These repositories are included in `scripts` via git submodules and are used by the SDK container wrapper scripts detailed on further below.
-The submodules reside in:
+Ebuilds for all packages reside in one of 2 subdirectories - [coreos-overlay](sdk_container/src/third_party/coreos-overlay) and [portage-stable](sdk_container/src/third_party/portage-stable/):
 ```
 scripts
    +--sdk_container
@@ -26,10 +24,10 @@ scripts
                              +------portage-stable
 ```
 
-When working with the scripts repo always make sure to initialise and to update these submodules; otherwise builds will break because build configuration is missing:
-```bash
-$ git clone --recurse-submodules https://github.com/flatcar/scripts.git
-```
+`portage-stable` is kept in alignment with upstream Gentoo and should not contain any modifications (with only minor, well-justified exceptions).
+Consider it a small sub-set of Gentoo.
+
+`coreos-overlay` contains significantly modified or even entirely self-written ebuilds.
 
 The `scripts` repository makes ample use of tags to mark releases.
 Sometimes, local and origin tags can diverge (e.g. when re-tagging something locally to test a build).
@@ -54,7 +52,7 @@ While work on a native ARM64 native SDK is ongoing, it's unfortunately not ready
 
 The container can be run in one of two ways - "standalone", or integrated with the [scripts](https://github.com/flatcar/scripts) repo:
 * Standalone mode will use no host volumes and will allow you to play with the SDK in a sandboxed throw-away environment. In standalone mode, you interface with Docker directly to use the SDK container.
-* Integrated mode will closely integrate with the scripts repo directory and bind-mount it as well as the portage-stable and coreos-overlay gitmodules into the container. Integrated mode uses wrapper scripts to interact with the SDK container. This is the recommended way for developing patches for Flatcar.
+* Integrated mode will closely integrate with the scripts repo directory and bind-mount it as well as the portage-stable and coreos-overlay directories into the container. Integrated mode uses wrapper scripts to interact with the SDK container. This is the recommended way for developing patches for Flatcar.
 
 ## Standalone mode
 
@@ -67,7 +65,7 @@ In standalone mode, the SDK is just another Docker container. Interaction with t
 * Start the image in interactive (tty) mode: `docker run -ti ghcr.io/flatcar/flatcar-sdk-all:3033.0.0`
   You are now inside the SDK container (the hostname will likely differ):
   `sdk@f236fda982a4 ~/trunk/src/scripts $`
-* Initialise the SDK in self-contained mode. This needs to be done once per container and will check out the scripts, coreos-overlay, and portage-stable repositories into the container.
+* Initialise the SDK in self-contained mode. This needs to be done once per container and will check out the scripts repository into the container.
   `sdk@f236fda982a4 ../sdk_init_selfcontained.sh`
 
 You can now work with the SDK container.
@@ -84,16 +82,14 @@ To start a container in privileged mode with `/dev` available use:
 
 This is the preferred mode of working with the SDK.
 Interaction with the container happens via wrapper scripts from the scripts repository.
-Both the host's scripts repo as well as its submodules (portage-stable and coreos-overlay) are made available in the container, allowing for work on these repos directly.
+Both the host's scripts repo as well as the ebuild paths (portage-stable and coreos-overlay) are made available in the container, allowing for work on these directly.
 The wrapper scripts will re-use existing containers instead of creating new ones to preserve your work in the container, enabling consistency.
 
 To clone the scripts repo and pick a version:
 * Clone the scripts repo: `git clone https://github.com/flatcar/scripts.git`
 * Optionally, check out a release tag to base your work on
   * list releases (e.g. all Alpha releases): `git tag -l alpha-*`
   * check out the release version, e.g. `3033.0.0`: `git checkout 3033.0.0`
-* Make sure to initialise and fetch git submodules - Flatcar's ebuilds are in 2 separate repositories, connected to `scripts` via submodules.
-  * `git submodule init; git submodule update`
 
 To use the SDK container:
 * Fetch image and start the SDK container: `./run_sdk_container -t`

diff --git a/bootstrap_sdk b/bootstrap_sdk
@@ -23,9 +23,9 @@
 #      built in stage 1. See "stage_repo()" documentation further below for more.
 #     This stage uses:
 #     - portage-stable from the SDK's /var/lib/gentoo/repos/gentoo
-#        or a git ref via --stage1_portage_ref command line option
+#        or a custom path via --stage1_portage_path command line option
 #     - coreos-overlay  from the SDK's /var/lib/gentoo/repos/coreos-overlay
-#        or a git ref via --stage1_overlay_ref command line option
+#        or a custom path via --stage1_overlay_path command line option
 #     Command line option refs need caution though, since
 #     stage1 must not contain updated ebuilds (see build_stage1 below).
 #
@@ -59,10 +59,10 @@ TYPE="flatcar-sdk"
 . "${BUILD_LIBRARY_DIR}/release_util.sh" || exit 1
 
 
-DEFINE_string stage1_portage_ref "" \
-  "Custom portage repo git ref to use in stage 1 (USE WITH CAUTION)"
-DEFINE_string stage1_overlay_ref "" \
-  "Custom overlay repo git ref to use in stage 1 (USE WITH CAUTION)"
+DEFINE_string stage1_portage_path "" \
+  "Path to custom portage ebuilds tree to use in stage 1 (DANGEROUS; USE WITH CAUTION)"
+DEFINE_string stage1_overlay_path "" \
+  "Path to custom overlay ebuilds tree to use in stage 1 (DANGEROUS; USE WITH CAUTION)"
 
 
 ## Define the stage4 config template
@@ -113,50 +113,54 @@ chmod 1777 "${ROOT_OVERLAY}/tmp"
 cp "${BUILD_LIBRARY_DIR}/toolchain_util.sh" "${ROOT_OVERLAY}/tmp"
 
 
-# Stage 1 uses "known-good" ebuild repos (both coreos-overlay and portage-stable)
+# Stage 1 uses "known-good" ebuilds (from both coreos-overlay and portage-stable)
 #  to build a minimal toolchain (USE="-*") for stage 2.
 #
 # No package updates must happen in stage 1, so we use the portage-stable and
-#  coreos-overlay repos included with the current SDK (from the SDK chroot's
+#  coreos-overlay paths included with the current SDK (from the SDK chroot's
 #  /var/lib/gentoo/repos/). "Current SDK" refers to the SDK we entered with
 #  'cork enter', i.e. the SDK we run ./bootstrap_sdk in.
 #
 # Using ebuilds from the above mentioned sources will ensure that stage 1 builds
 #  a minimal stage 2 from known-good ebuild versions - the same ebuild versions
 #  that were used to build the very SDK we run ./bootstrap_sdk in.
 #
-# This is needed because stage 1 lacks proper isolation and will link all packages built for
+# DANGER ZONE
+#
+# Stage 1 lacks proper isolation and will link all packages built for
 #  stage 2 against its own seed libraries ("/" in the catalyst chroot) instead of against libraries
 #  installed into the FS root of the stage 2 seed ("/tmp/stage1root" in the catalyst chroot).
 #  This is why we must prevent any updated package ebuilds to "leak" into stage 1, hence we use
 #  "known good" ebuild repo versions outlined above.
 #
-# In special circumstances it may be required to circumvent this and use git
-#  refs of either (or both) portage and overlay. The command line options
-#  --stage1-portage-ref and --stage1-overlay-ref may be used to specify
-#  a git ref known to work for stage1. In that case the stage1 seed (i.e. the seed SDK)
+# In special circumstances it may be required to circumvent this and use custom paths
+#  for either (or both) portage and overlay. The command line options
+#  --stage1-portage-path and --stage1-overlay-path may be used to specify
+#  a repo path known to work for stage1. In that case the stage1 seed (i.e. the seed SDK)
 #  will be updated prior to starting to build stage 2.
+#  NOTE that this should never be used to introduce library updates in stage 1. All binaries
+#  produced in stage 1 are linked against libraries in the seed tarball, NOT libraries produced
+#  by stage one. Therefore, these binaries will cease to work in stage 2 when linked against
+#  outdated "seed tarball" libraries which have been updated to newer versions in stage 1.
 
 stage_repo() {
     local repo="$1"
-    local gitref="$2"
+    local path="$2"
     local dest="$3"
     local gitname="$repo"
 
     if [ "$gitname" = "gentoo" ] ; then
         gitname="portage-stable"
     fi
 
-    if [ -z "$gitref" ]; then
+    if [ -z "$path" ]; then
         cp -R "/var/gentoo/repos/${repo}" "$dest"
         info "Using local SDK's ebuild repo '$repo' ('$gitname') in stage 1."
     else
-        info "Using git ref '$gitref' for ebuild repo '$repo' ('$gitname') in stage 1."
         mkdir "$dest/$repo"
-        (   cd "$dest/$repo" \
-            && git clone "https://github.com/flatcar/$gitname.git" . \
-            && git fetch --all \
-            && git checkout "$gitref" )
+        cp -R "${path}/"* "$dest/${repo}/"
+        info "Using custom path '$path' for ebuild repo '$repo' ('$gitname') in stage 1."
+        info "This may break stage 2. YOU HAVE BEEN WARNED. You break it, you keep it."
     fi
     (
         set -euo pipefail
@@ -186,9 +190,9 @@ build_stage1() {
     mkdir "$stage1_repos"
 
     # prepare ebuild repos for stage 1, either from the local SDK (default)
-    #  or from git refs specified via command line flags
-    stage_repo "gentoo" "${FLAGS_stage1_portage_ref}" "$stage1_repos"
-    stage_repo "coreos-overlay" "${FLAGS_stage1_overlay_ref}" "$stage1_repos"
+    #  or from custom paths specified via command line flags
+    stage_repo "gentoo" "${FLAGS_stage1_portage_path}" "$stage1_repos"
+    stage_repo "coreos-overlay" "${FLAGS_stage1_overlay_path}" "$stage1_repos"
 
     # Create a snapshot of "known-good" portage-stable repo copy for use in stage 1
     #  This requires us to create a custom catalyst config to point it to the
@@ -209,8 +213,8 @@ build_stage1() {
            -e "s,^portage_overlay:.*,portage_overlay: $stage1_repos/coreos-overlay," \
         "$TEMPDIR/stage1.spec"
 
-    # If we are to use a git ref for either ebuild repo we want to update the stage1 seed SDK
-    if [ -n "${FLAGS_stage1_portage_ref}" -o -n "${FLAGS_stage1_overlay_ref}" ] ; then
+    # If we are to use a custom path for either ebuild repo we want to update the stage1 seed SDK
+    if [ -n "${FLAGS_stage1_portage_path}" -o -n "${FLAGS_stage1_overlay_path}" ] ; then
         sed -i 's/^update_seed: no/update_seed: yes/' "$TEMPDIR/stage1.spec"
         echo "update_seed_command: --update --deep --newuse --complete-graph --rebuild-if-new-ver --rebuild-exclude cross-*-cros-linux-gnu/* sys-devel/gcc " \
             >>"$TEMPDIR/stage1.spec"

diff --git a/build_library/build_image_util.sh b/build_library/build_image_util.sh
@@ -400,8 +400,7 @@ get_metadata() {
         # SRC_URI is empty for the special github.com/flatcar projects
         if [ -z "${val}" ]; then
             # The grep invocation gives errors when the ebuild file is not present.
-            # This can happen if a "scripts" branch does not match the "coreos-overlay" branch
-            # or when the binary packages from ./build_packages are outdated.
+            # This can happen when the binary packages from ./build_packages are outdated.
             val="$(grep "CROS_WORKON_PROJECT=" "${ebuild_path}" | cut -d '"' -f 2)"
             if [ -n "${val}" ]; then
                 val="https://github.com/${val}"
@@ -541,8 +540,6 @@ You can read it with "less licenses.json.bz2" or convert it to a text format wit
 bzcat licenses.json.bz2 | jq -r '.[] | "\(.project):\nDescription: \(.description)\nLicenses: \(.licenses)\nHomepage: \(.homepage)\nSource code: \(.source)\nFiles:\n\(.files)\n"'
 The license texts are available under /usr/share/licenses/common/ and can be read with "less NAME.gz".
 Build system files and patches used to build these projects are located at:
-https://github.com/flatcar/coreos-overlay/
-https://github.com/flatcar/portage-stable/
 https://github.com/flatcar/scripts/
 Information on how to build Flatcar Container Linux can be found under:
 https://www.flatcar.org/docs/latest/reference/developer-guides/sdk-modifying-flatcar/

diff --git a/build_library/dev_container_util.sh b/build_library/dev_container_util.sh
@@ -48,13 +48,9 @@ main-repo = portage-stable
 
 [coreos]
 location = /var/lib/portage/coreos-overlay
-sync-type = git
-sync-uri = https://github.com/flatcar/coreos-overlay.git
 
 [portage-stable]
 location = /var/lib/portage/portage-stable
-sync-type = git
-sync-uri = https://github.com/flatcar/portage-stable.git
 EOF
 
     # Now set the correct profile, we do not use the eselect tool - it

diff --git a/ci-automation/README.md b/ci-automation/README.md
@@ -39,7 +39,7 @@ image_build amd64
 ### SDK bootstrap build
 
 1. SDK Bootstrap (`sdk.sh`): Use a seed SDK tarball and seed SDK container image to build a new SDK tarball.
-   The resulting SDK tarball will use packages and versions pinned in the coreos-overlay and portage-stable submodules.
+   The resulting SDK tarball will ship packages and versions from the updated coreos-overlay and portage-stable ebuild directories.
    This step updates the versionfile, recording the SDK container version just built.
    It will generate and push a new version tag to the scripts repo.
 2. SDK container build (`sdk_container.sh`) : use SDK tarball to build an SDK container image.