Build metal-dev + aarch64 fails with file/dir not found (index.json)

[aRustyDev]

I'm trying to get familiar with the build process of bottlerocket, but running what seems to be the commands for a new contributor is yielding an error that feels like something has changed internally, but it hasn't made its way to the docs yet? This kind of boils down to an index.json not existing, it seems like that file is supposed to be generated during the build process, but I'm not sure if its an error on my part or if the build process docs aren't telling the whole story?

I'm hoping to try contributing to this project, and am trying to take good notes on what the actual build process steps are so I can try making an PR for documentation updates with info a total newb like myself would appreciate. So any and all feedback/context on this is welcome!

PS: I have tried building this in a few different ways so far, including directly on my macos, from an ubuntu Vm, this ubuntu LXC, and even a docker container. They have all failed, but with slightly different errors so far. I figured I should only try creating one issue for now and picked this one since it seems the easiest to replicate.

Platform I'm building on:

# About my System
$ uname -a
Linux ubuntu 6.10.7-orbstack-00280-gd3b7ec68d3d4 #2 SMP Mon Sep  2 03:51:24 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

$ lsb_release -a
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 24.04.1 LTS
Release:	24.04
Codename:	noble

$ hostnamectl
 Static hostname: ubuntu
       Icon name: computer-container
         Chassis: container ☐
      Machine ID: a6fefb6f61f244778be96c03fef9430e
         Boot ID: 88b06cfa567940818cdba5df5ef6190c
  Virtualization: lxc
Operating System: Ubuntu 24.04.1 LTS
          Kernel: Linux 6.10.7-orbstack-00280-gd3b7ec68d3d4
    Architecture: x86-64

What I expected to happen:
cargo make to complete successfully.

What actually happened:

# Trying to build the metal-dev for ARM
$ VARIANT=metal-dev
$ ARCH=aarch64
$ CORES=6
$ BR_PATH=/path/to/bottlerocket # just a convenience ENV to help with debugging
$ sudo cargo make -e BUILDSYS_VARIANT=$VARIANT -e BUILDSYS_ARCH=$ARCH -e BUILDSYS_JOBS=$CORES

[cargo-make] INFO - cargo make 0.37.16
[cargo-make] INFO - Calling cargo metadata to extract project info
[cargo-make] INFO - Cargo metadata done
[cargo-make] INFO - Build File: Makefile.toml
[cargo-make] INFO - Task: default
[cargo-make] INFO - Profile: development
[cargo-make] INFO - Running Task: install-twoliter
Found Twoliter v0.4.5 installed.
Skipping installation.
[cargo-make] INFO - Execute Command: "/path/to/bottlerocket/tools/twoliter/twoliter" "--log-level=info" "fetch" "--project-path=/path/to/bottlerocket/Twoliter.toml" "--arch=aarch64"
[2024-09-15T16:34:49Z WARN  twoliter::project] A Release.toml file was found. Release.toml is deprecated. Please remove it from your project.
[2024-09-15T16:34:49Z INFO  twoliter::lock] Resolving project references to check against lock file
public.ecr.aws/bottlerocket/bottlerocket-core-kit@sha256:502b175071511c643eb727d81b1538e237db663839f95cf91951bff56e3b5efa: Pulling from bottlerocket/bottlerocket-core-kit
35be9e349dd5: Pulling fs layer
...
33770f63d6bd: Waiting
...
264ad05621e6: Pull complete
...
Digest: sha256:502b175071511c643eb727d81b1538e237db663839f95cf91951bff56e3b5efa
Status: Downloaded newer image for public.ecr.aws/bottlerocket/bottlerocket-core-kit@sha256:502b175071511c643eb727d81b1538e237db663839f95cf91951bff56e3b5efa
public.ecr.aws/bottlerocket/bottlerocket-core-kit@sha256:502b175071511c643eb727d81b1538e237db663839f95cf91951bff56e3b5efa
public.ecr.aws/bottlerocket/bottlerocket-core-kit@sha256:be8fe3bdd66abed30aac76155246b99505a5e3cd8099be25b858c993c563be2b: Pulling from bottlerocket/bottlerocket-core-kit
fd14c67319ca: Pulling fs layer
...
1606b6590baf: Waiting
...
fee32fc19fdb: Pull complete
...
Digest: sha256:be8fe3bdd66abed30aac76155246b99505a5e3cd8099be25b858c993c563be2b
Status: Downloaded newer image for public.ecr.aws/bottlerocket/bottlerocket-core-kit@sha256:be8fe3bdd66abed30aac76155246b99505a5e3cd8099be25b858c993c563be2b
public.ecr.aws/bottlerocket/bottlerocket-core-kit@sha256:be8fe3bdd66abed30aac76155246b99505a5e3cd8099be25b858c993c563be2b
[2024-09-15T16:37:32Z INFO  twoliter::lock] Extracting kit dependencies. dependencies=["bottlerocket-core-kit-2.5.0@bottlerocket (public.ecr.aws/bottlerocket/bottlerocket-core-kit:v2.5.0)"]
[2024-09-15T16:37:32Z INFO  twoliter::lock] Extracting kit 'bottlerocket-core-kit-2.5.0@bottlerocket (public.ecr.aws/bottlerocket/bottlerocket-core-kit:v2.5.0)' to '/path/to/bottlerocket/build/external-kits'
public.ecr.aws/bottlerocket/bottlerocket-core-kit@sha256:502b175071511c643eb727d81b1538e237db663839f95cf91951bff56e3b5efa: Pulling from bottlerocket/bottlerocket-core-kit
Digest: sha256:502b175071511c643eb727d81b1538e237db663839f95cf91951bff56e3b5efa
Status: Image is up to date for public.ecr.aws/bottlerocket/bottlerocket-core-kit@sha256:502b175071511c643eb727d81b1538e237db663839f95cf91951bff56e3b5efa
public.ecr.aws/bottlerocket/bottlerocket-core-kit@sha256:502b175071511c643eb727d81b1538e237db663839f95cf91951bff56e3b5efa
Error: Unable to read from '/path/to/bottlerocket/bottlerocket/build/external-kits/cache/sha256-502b175071511c643eb727d81b1538e237db663839f95cf91951bff56e3b5efa/index.json'

Caused by:
    No such file or directory (os error 2)
Error while executing command, exit code: 1

Me looking for the missing file (just in case lol)

# Looking for 'index.json'
ls $BR_PATH/bottlerocket/build/external-kits/cache/sha256-502b175071511c643eb727d81b1538e237db663839f95cf91951bff56e3b5efa/
<SHA256-NAMED-FILE>     <SHA256-NAMED-FILE>
...                     ...
<SHA256-NAMED-FILE>     <SHA256-NAMED-FILE>
<SHA256-NAMED-FILE>     2748f06e1ee6e8e749ec4defce2451b1556f1a64401bcb8512886afbebf4c36d.json
<SHA256-NAMED-FILE>      manifest.json
<SHA256-NAMED-FILE>

Could be crazy, but does this look like what should be in index.json?

# Checking contents of 2748f06e1ee6e8e749ec4defce2451b1556f1a64401bcb8512886afbebf4c36d.json
$ jq '.' $BR_PATH/build/external-kits/cache/sha256-502b175071511c643eb727d81b1538e237db663839f95cf91951bff56e3b5efa/2748f06e1ee6e8e749ec4defce2451b1556f1a64401bcb8512886afbebf4c36d.json
{
  "architecture": "arm64",
  "config": {
    "Env": [],
    "WorkingDir": "/",
    "OnBuild": null,
    "Labels": {
      "dev.bottlerocket.kit.v1": "eyJraXQiOltdLCJuYW1lIjoiYm90dGxlcm9ja2V0LWNvcmUta2l0Iiwic2RrIjp7ImRpZ2VzdCI6IkNJeC9HNzRXK0llNGdkSjQwRC9OL1VieVE5SnF3Vk9kLzlJSDRydTN6WWs9IiwibmFtZSI6ImJvdHRsZXJvY2tldC1zZGsiLCJzb3VyY2UiOiJwdWJsaWMuZWNyLmF3cy9ib3R0bGVyb2NrZXQvYm90dGxlcm9ja2V0LXNkazp2MC40NS4wIiwidmVuZG9yIjoiYm90dGxlcm9ja2V0IiwidmVyc2lvbiI6IjAuNDUuMCJ9LCJ2ZXJzaW9uIjoiMi41LjAifQo="
    }
  },
  "created": "2024-09-11T19:39:45.780104584Z",
  "history": [],
  "os": "linux",
  "rootfs": {
    "type": "layers",
    "diff_ids": [ # size == 122 layers
      "sha256:<SHA256>",
      ...
      "sha256:<SHA256>", # 120 times
      ...
      "sha256:<SHA256>"
    ]
  }
}

How to reproduce the problem:

1. Use an ubuntu 24.04.1 LTS image
2. Install dependencies

sudo apt update && apt upgrade -y
sudo apt install -y --no-install-recommends docker.io docker-buildx
sudo apt install -y build-essential openssl libssl-dev pkg-config liblz4-tool
sudo apt install -y rustup git curl yq

3. install rust stable & cargo-make

sudo rustup install stable
sudo rustup default stable
sudo cargo install cargo-make

4. Set build ENVs

VARIANT=metal-dev
ARCH=aarch64
CORES=6
LOGFILE=issue-4195.$(grep PRETTY_NAME /etc/os-release | cut -f 2 -d= | tr ' ' '-').log

5. Clone the repo

git clone https://github.com/bottlerocket-os/bottlerocket
cd bottlerocket
BR_PATH=$(pwd) # just a convenience ENV to help with debugging 

6. Run the build

CARGO_PROFILE_DEV_BUILD_OVERRIDE_DEBUG=true \
RUST_BACKTRACE=full \
sudo cargo make -e BUILDSYS_VARIANT=$VARIANT -e BUILDSYS_ARCH=$ARCH -e BUILDSYS_JOBS=$CORES 2>&1 | tee $LOGFILE

[yeazelm]

Hello @aRustyDev, thanks for cutting this issue! You are right! We have not updated our docs with the additional dependency needed to build with kits. We added it to the bottlerocket-core-kit BUILDING doc but have not done the change for this repo: https://github.com/bottlerocket-os/bottlerocket-core-kit/blob/develop/BUILDING.md#docker. We added the need for either containerd-snapshotter as feature enabled for Docker or adding crane into the path (which is automatically detected and used). Doing one of the two options should get this working. I'll also get a PR out to fix the docs shortly! Hopefully this helps!

As a side note, if you add your user to the docker group then you can avoid running cargo as root (which we recommend!). I would highly recommend you try to run the cargo calls as a normal user.

[aRustyDev]

@yeazelm So does that mean that if I am trying to build the metal-dev image, I could use the bottlerocket-core-kit repo and just run make ARCH=aarch64, or would that build something else? I'm trying to get the simplest image possible for now so I can get more familiar with it and host it locally on my machine.

[yeazelm]

You shouldn't need to build the bottlerocket-core-kit but it is vended with with the assumption you are using containerd-snapshotter with Docker. The kit is the source of the built packages that make up a Bottlerocket image, so if you don't need to customize anything, you can download the latest kit and build an image from there. That error is related to Docker not understanding how to read the kit. You'll need to delete the build/ directory from the bottlerocket-os/bottlerocket checkout and possibly prune the recently downloaded images in Docker to re-pull with either the Docker daemon using the containerd-snapshotter. If you don't want to change your Docker daemon configuration, you have install crane to download them (which doesn't require special Docker configuration). To configure the containerd snapshotter (even just for pulling images), you can specify the daemon configuration with:

{
  "features": {
    "buildkit": true,
    "containerd-snapshotter": true
  }
}

Nonetheless, folks developing on Bottlerocket have found crane to work pretty well and that should make the download of kits pretty fast!

If want to play with changing how packages are built/configured that are included in the image and want to customize Bottlerocket images, then you might need to build the kit, but if you just want to build an image, you only need the bottlerocket-os/bottlerocket repo checked out. Hopefully that helps!