feat: Parallel native/wasm bb builds. Better messaging around using c…

…i cache. (#4766)

Ignore badly named branch name. Public ECR is a bit of a no go as it has
a bunch of restrictions that is a problem:
* Only us-east-1.
* Missing various API's, like ability to retag remote images without
pulling.
* No lifecycle policies.

It was actually the second point that made me give up on public ecr
right now.

* Once upon a time, we published our deployment containers to a private
ECR repo in eu-west-2. Removes all references to the old private ECR
repo from dockerfiles and build stuff, now just using the
`aztecprotocol` dockerhub org.
* Makes `./bootstrap.sh` require an explicit command, one of
`full|fast|clean`. Add warnings when trying to use cache but can't, or
not using cache but can. By being explicit users are made aware of the
option of a `fast` bootstrap.
* barretenberg now builds native/wasm/wasm-threads together in parallel,
with pretty line coloring to distinguish output.
* New amis for build-instances (Ubuntu 22 with upgraded aws-cli).

.circleci/config.yml

@@ -86,6 +86,7 @@ jobs:
             build-system/scripts/generate_circleci_config.py > .circleci/generated_config.yml
       - continuation/continue:
           configuration_path: .circleci/generated_config.yml
+
   # Noir
   noir-x86_64:
     docker:
@@ -207,7 +208,7 @@ jobs:
           name: "Build"
           command: cond_spot_run_build barretenberg-x86_64-linux-clang-fuzzing 128
           aztec_manifest_key: barretenberg-x86_64-linux-clang-fuzzing
-          
+
   barretenberg-x86_64-linux-clang-assert:
     docker:
       - image: aztecprotocol/alpine-build-image

avm-transpiler/bootstrap.sh

@@ -17,6 +17,6 @@ if [ -n "$CMD" ]; then
 fi
 
 # Attempt to just pull artefacts from CI and exit on success.
-./bootstrap_cache.sh && exit
+[ -n "${USE_CACHE:-}" ] && ./bootstrap_cache.sh && exit
 
 ./scripts/bootstrap_native.sh

avm-transpiler/bootstrap_cache.sh

@@ -1,8 +1,6 @@
 #!/usr/bin/env bash
 set -eu
 
-[ -z "${NO_CACHE:-}" ] && type docker &> /dev/null && [ -f ~/.aws/credentials ] || exit 1
-
 cd "$(dirname "$0")"
 source ../build-system/scripts/setup_env '' '' mainframe_$USER > /dev/null
 

barretenberg/acir_tests/Dockerfile.bb

@@ -1,5 +1,5 @@
-FROM 278380418400.dkr.ecr.eu-west-2.amazonaws.com/barretenberg-x86_64-linux-clang-assert
-FROM 278380418400.dkr.ecr.eu-west-2.amazonaws.com/noir-compile-acir-tests as noir-acir-tests
+FROM aztecprotocol/barretenberg-x86_64-linux-clang-assert
+FROM aztecprotocol/noir-compile-acir-tests as noir-acir-tests
 
 FROM node:18.19.0-alpine
 RUN apk update && apk add git bash curl jq coreutils
@@ -10,7 +10,7 @@ COPY . .
 # Run every acir test through native bb build prove_then_verify flow for UltraPlonk.
 # This ensures we test independent pk construction through real/garbage witness data paths.
 RUN FLOW=prove_then_verify ./run_acir_tests.sh
-# This flow is essentially the GoblinUltraHonk equivalent to the UltraPlonk "prove and verify". (This functionality is 
+# This flow is essentially the GoblinUltraHonk equivalent to the UltraPlonk "prove and verify". (This functionality is
 # accessed via the goblin "accumulate" mechanism).
 RUN FLOW=accumulate_and_verify_goblin ./run_acir_tests.sh
 # This is a "full" Goblin flow. It constructs and verifies four proofs: GoblinUltraHonk, ECCVM, Translator, and merge

barretenberg/acir_tests/Dockerfile.bb.js

@@ -1,5 +1,5 @@
-FROM 278380418400.dkr.ecr.eu-west-2.amazonaws.com/bb.js
-FROM 278380418400.dkr.ecr.eu-west-2.amazonaws.com/noir-compile-acir-tests as noir-acir-tests
+FROM aztecprotocol/bb.js
+FROM aztecprotocol/noir-compile-acir-tests as noir-acir-tests
 
 FROM node:18.19.0
 COPY --from=0 /usr/src/barretenberg/ts-build /usr/src/barretenberg/ts

barretenberg/acir_tests/Dockerfile.bb.sol

@@ -1,6 +1,6 @@
-FROM 278380418400.dkr.ecr.eu-west-2.amazonaws.com/barretenberg-x86_64-linux-clang-assert
-FROM 278380418400.dkr.ecr.eu-west-2.amazonaws.com/barretenberg-x86_64-linux-clang-sol
-FROM 278380418400.dkr.ecr.eu-west-2.amazonaws.com/noir-compile-acir-tests as noir-acir-tests
+FROM aztecprotocol/barretenberg-x86_64-linux-clang-assert
+FROM aztecprotocol/barretenberg-x86_64-linux-clang-sol
+FROM aztecprotocol/noir-compile-acir-tests as noir-acir-tests
 
 FROM node:18.19.0-alpine
 RUN apk update && apk add git bash curl jq

barretenberg/acir_tests/Dockerfile.noir_acir_tests

@@ -2,7 +2,7 @@
 # However, the noir subrepo has no concept of the aztec build pipeline, so the Dockerfile wouldn't make sense there.
 # So, it lives here.
 # This chains off the nargo build, and creates a container with a compiled set of acir tests.
-FROM 278380418400.dkr.ecr.eu-west-2.amazonaws.com/noir
+FROM aztecprotocol/noir
 RUN apt update && apt install -y jq && rm -rf /var/lib/apt/lists/* && apt-get clean
 ENV PATH="/usr/src/noir/target/release:${PATH}"
 WORKDIR /usr/src/noir/test_programs

barretenberg/cpp/bootstrap.sh

@@ -31,8 +31,11 @@ fi
 # Download ignition transcripts.
 (cd ./srs_db && ./download_ignition.sh 0)
 
+# Install wasi-sdk.
+./scripts/install-wasi-sdk.sh
+
 # Attempt to just pull artefacts from CI and exit on success.
-./bootstrap_cache.sh && exit
+[ -n "${USE_CACHE:-}" ] && ./bootstrap_cache.sh && exit
 
 # Pick native toolchain file.
 ARCH=$(uname -m)
@@ -54,23 +57,34 @@ echo "# Building with preset: $PRESET"
 echo "# When running cmake directly, remember to use: --build --preset $PRESET"
 echo "#################################"
 
-# Build native.
-cmake --preset $PRESET -DCMAKE_BUILD_TYPE=RelWithAssert
-cmake --build --preset $PRESET --target bb
+function build_native {
+  cmake --preset $PRESET -DCMAKE_BUILD_TYPE=RelWithAssert
+  cmake --build --preset $PRESET --target bb
+}
+
+function build_wasm {
+  cmake --preset wasm
+  cmake --build --preset wasm
+}
+
+function build_wasm_threads {
+  cmake --preset wasm-threads
+  cmake --build --preset wasm-threads
+}
+
+g="\033[32m"  # Green
+b="\033[34m"  # Blue
+p="\033[35m"  # Purple
+r="\033[0m"   # Reset
+
+(build_native > >(awk -v g="$g" -v r="$r" '$0=g"native: "r $0')) &
+(build_wasm > >(awk -v b="$b" -v r="$r" '$0=b"wasm: "r $0')) &
+(build_wasm_threads > >(awk -v p="$p" -v r="$r" '$0=p"wasm_threads: "r $0')) &
+
+wait
 
 if [ ! -d ./srs_db/grumpkin ]; then
   # The Grumpkin SRS is generated manually at the moment, only up to a large enough size for tests
   # If tests require more points, the parameter can be increased here.
-  (cd ./build && cmake --build . --parallel --target grumpkin_srs_gen && ./bin/grumpkin_srs_gen 8192)
+  cd ./build && cmake --build . --parallel --target grumpkin_srs_gen && ./bin/grumpkin_srs_gen 8192
 fi
-
-# Install wasi-sdk.
-./scripts/install-wasi-sdk.sh
-
-# Build WASM.
-cmake --preset wasm
-cmake --build --preset wasm
-
-# Build WASM with new threading.
-cmake --preset wasm-threads
-cmake --build --preset wasm-threads

barretenberg/cpp/bootstrap_cache.sh

@@ -1,8 +1,6 @@
 #!/usr/bin/env bash
 set -eu
 
-[ -z "${NO_CACHE:-}" ] && type docker &> /dev/null && [ -f ~/.aws/credentials ] || exit 1
-
 cd "$(dirname "$0")"
 source ../../build-system/scripts/setup_env '' '' mainframe_$USER > /dev/null
 

barretenberg/cpp/dockerfiles/Dockerfile.bench

@@ -1,4 +1,4 @@
-FROM 278380418400.dkr.ecr.eu-west-2.amazonaws.com/barretenberg-x86_64-linux-clang
+FROM aztecprotocol/barretenberg-x86_64-linux-clang
 WORKDIR /usr/src/barretenberg/cpp
 RUN apk update && apk add curl libstdc++ jq
 RUN ./scripts/ci/ultra_honk_bench.sh

barretenberg/ts/Dockerfile

@@ -1,4 +1,4 @@
-FROM 278380418400.dkr.ecr.eu-west-2.amazonaws.com/barretenberg-wasm-linux-clang
+FROM aztecprotocol/barretenberg-wasm-linux-clang
 
 FROM node:18.19.0-alpine
 COPY --from=0 /usr/src/barretenberg /usr/src/barretenberg

barretenberg/ts/bootstrap.sh

@@ -19,7 +19,7 @@ if [ -n "$CMD" ]; then
 fi
 
 # Attempt to just pull artefacts from CI and exit on success.
-./bootstrap_cache.sh && exit
+[ -n "${USE_CACHE:-}" ] && ./bootstrap_cache.sh && exit
 
 yarn install --immutable
 echo "Building with command 'yarn $BUILD_CMD'..."

barretenberg/ts/bootstrap_cache.sh

@@ -1,8 +1,6 @@
 #!/usr/bin/env bash
 set -eu
 
-[ -z "${NO_CACHE:-}" ] && type docker &> /dev/null && [ -f ~/.aws/credentials ] || exit 1
-
 cd "$(dirname "$0")"
 source ../../build-system/scripts/setup_env '' '' mainframe_$USER > /dev/null
 

bootstrap.sh

@@ -12,39 +12,47 @@ cd "$(dirname "$0")"
 
 CMD=${1:-}
 
-if [ -n "$CMD" ]; then
-  if [ "$CMD" = "clean" ]; then
-    echo "WARNING: This will erase *all* untracked files, including hooks and submodules."
-    echo -n "Continue? [y/n] "
-    read user_input
-    if [ "$user_input" != "y" ] && [ "$user_input" != "Y" ]; then
-      exit 1
-    fi
+YELLOW="\033[93m"
+BOLD="\033[1m"
+RESET="\033[0m"
 
-    # Remove hooks and submodules.
-    rm -rf .git/hooks/*
-    rm -rf .git/modules/*
-    for SUBMODULE in $(git config --file .gitmodules --get-regexp path | awk '{print $2}'); do
-      rm -rf $SUBMODULE
-    done
+source ./build-system/scripts/setup_env '' '' '' > /dev/null
 
-    # Remove all untracked files, directories, nested repos, and .gitignore files.
-    git clean -ffdx
+if [ "$CMD" = "clean" ]; then
+  echo "WARNING: This will erase *all* untracked files, including hooks and submodules."
+  echo -n "Continue? [y/n] "
+  read user_input
+  if [ "$user_input" != "y" ] && [ "$user_input" != "Y" ]; then
+    exit 1
+  fi
+
+  # Remove hooks and submodules.
+  rm -rf .git/hooks/*
+  rm -rf .git/modules/*
+  for SUBMODULE in $(git config --file .gitmodules --get-regexp path | awk '{print $2}'); do
+    rm -rf $SUBMODULE
+  done
 
-    exit 0
-  elif [ "$CMD" = "full" ]; then
-    export NO_CACHE=1
-  else
-    echo "Unknown command: $CMD"
+  # Remove all untracked files, directories, nested repos, and .gitignore files.
+  git clean -ffdx
+
+  exit 0
+elif [ "$CMD" = "full" ]; then
+  if can_use_ci_cache; then
+    echo -e "${BOLD}${YELLOW}WARNING: Performing a full bootstrap. Consider leveraging './bootstrap.sh fast' to use CI cache.${RESET}"
+    echo
+  fi
+elif [ "$CMD" = "fast" ]; then
+  export USE_CACHE=1
+  if ! can_use_ci_cache; then
+    echo -e "${BOLD}${YELLOW}WARNING: Either docker or aws credentials are missing. Install docker and request credentials. Note this is for internal aztec devs only.${RESET}"
     exit 1
   fi
+else
+  echo "usage: $0 <full|fast|clean>"
+  exit 1
 fi
 
-# if [ ! -f ~/.nvm/nvm.sh ]; then
-#   echo "Nvm not found at ~/.nvm"
-#   exit 1
-# fi
-
 # Install pre-commit git hooks.
 HOOKS_DIR=$(git rev-parse --git-path hooks)
 echo "(cd barretenberg/cpp && ./format.sh staged)" >$HOOKS_DIR/pre-commit

boxes/Dockerfile

@@ -1,8 +1,8 @@
 # Builds the boxes (they were copied into yarn-project-base so the cli can unbox).
 # Produces a container that can be run to test a specific box. See docker-compose.yml.
-FROM 278380418400.dkr.ecr.eu-west-2.amazonaws.com/aztec AS aztec
-FROM 278380418400.dkr.ecr.eu-west-2.amazonaws.com/noir as noir
-FROM 278380418400.dkr.ecr.eu-west-2.amazonaws.com/noir-projects as noir-projects
+FROM aztecprotocol/aztec AS aztec
+FROM aztecprotocol/noir as noir
+FROM aztecprotocol/noir-projects as noir-projects
 
 # We need yarn. Start fresh container.
 FROM node:18.19.0

build-system/scripts/build

@@ -104,14 +104,14 @@ for PARENT_REPO in $(query_manifest dependencies $REPOSITORY); do
       # Finally attempt to locate x86_64 image tag, as sometimes we build arch specific images from x86_64 images.
       PARENT_IMAGE_TAG=$(calculate_image_tag $PARENT_REPO x86_64)
       if ! image_exists $PARENT_REPO $PARENT_IMAGE_TAG; then
-        echo "Failed to locate multiarch image, arch specific image, or x86_64 image. Aborting."
+        echo "Failed to locate multiarch image, arch specific image, or x86_64 image for $PARENT_REPO. Aborting."
         exit 1
       fi
     fi
   fi
 
   # Substitute references to parent repo, with the relevent built image uri.
-  DEPLOY_URI=$ECR_DEPLOY_URL/$PARENT_REPO
+  DEPLOY_URI=aztecprotocol/$PARENT_REPO
   PARENT_IMAGE_URI=$ECR_URL/$PARENT_REPO:$PARENT_IMAGE_TAG
   awk '{if ($1 == "FROM" && $2 == "'$DEPLOY_URI'") $2 = "'$PARENT_IMAGE_URI'"; print $0}' $DOCKERFILE > _temp && mv _temp $DOCKERFILE
 done

build-system/scripts/build_local

@@ -64,8 +64,7 @@ for E in "${PROJECTS[@]}"; do
     fi
   fi
 
-  # TODO: Move everything to aztecprotocol/*
-  DEPLOY_IMAGE_URI=$ECR_DEPLOY_URL/$REPO:latest
+  DEPLOY_IMAGE_URI=aztecprotocol/$REPO:latest
   CACHE_IMAGE_URI=$(calculate_image_uri $REPO)
   ARR=(${CACHE_IMAGE_URI//:/ })
   REPO_URI=${ARR[0]}
@@ -112,9 +111,6 @@ for E in "${PROJECTS[@]}"; do
   # Retag with deployment uri.
   docker tag $CACHE_IMAGE_URI $DEPLOY_IMAGE_URI
 
-  # Retag for aztecprotocol dockerhub.
-  docker tag $DEPLOY_IMAGE_URI aztecprotocol/$REPO:latest
-
   remove_old_images $REPO
 
   echo -e "${BOLD}Tagged${RESET}: aztecprotocol/$REPO:latest"

build-system/scripts/can_use_ci_cache

@@ -0,0 +1,3 @@
+#!/usr/bin/env bash
+set -euo pipefail
+type docker &> /dev/null && docker ps 2>&1 > /dev/null && [ -f ~/.aws/credentials ] && ecr_login > /dev/null || exit 1

build-system/scripts/cond_run_compose

@@ -27,7 +27,6 @@ if ! check_rebuild $SUCCESS_TAG $REPOSITORY; then
     echo "Pulling $REPO..."
     REPO_IMAGE_URI=$(calculate_image_uri $REPO)
     retry docker pull $REPO_IMAGE_URI
-    docker tag $REPO_IMAGE_URI $ECR_DEPLOY_URL/$REPO
     docker tag $REPO_IMAGE_URI aztecprotocol/$REPO
   done
 

build-system/scripts/generate_circleci_config.py

@@ -20,7 +20,7 @@ def get_manifest_job_names():
 
 def is_already_built_circleci_job(circleci_job, already_built_manifest_jobs):
     """
-    This function checks if a given CircleCI job is associated with a specific already-built manifest job. 
+    This function checks if a given CircleCI job is associated with a specific already-built manifest job.
     It does so by checking the job's steps for an 'aztec_manifest_key' that contain references to manifest names.
     We want to see at least one such key, and for all such keys to be in 'already_built_manifest_jobs'.
     """
@@ -43,7 +43,7 @@ def is_already_built_circleci_job(circleci_job, already_built_manifest_jobs):
                 return False
         matching_steps += 1
     # All steps have matched - but make sure that's actually more than one step
-    return matching_steps > 0 
+    return matching_steps > 0
 
 def get_already_built_circleci_job_names(circleci_jobs):
     already_built_manifest_jobs = list(get_already_built_manifest_job_names())
@@ -75,11 +75,11 @@ def get_already_built_manifest_job_names():
 def remove_jobs_from_workflow(jobs, to_remove):
     """
     Removes jobs from a given CircleCI JSON workflow.
-    
+
     Parameters:
         jobs (dict): The JSON object representing the CircleCI workflow jobs dependencies portion.
         to_remove (list): The list of jobs to be removed from the workflow.
-    
+
     Returns:
         dict: The new JSON object with specified jobs removed.
     """
@@ -97,7 +97,7 @@ def remove_jobs_from_workflow(jobs, to_remove):
 
 if __name__ == '__main__':
     # The CircleCI workflow as a JSON string (Replace this with your actual workflow)
-    
+
     # Convert the JSON string to a Python dictionary
     workflow_dict = yaml.safe_load(open('.circleci/config.yml'))