Docker: Cannot build and run on ARM (M1 Mac)

[thilgen]

Overview

I am able to successfully build and execute Salto with Docker on an X86 system.

However, using the same steps and files on an ARM system (M1 Mac), the Docker image builds but fails to run.

I used the standard Salto Dockerfile - with ENTRYPOINT commented out - for each of these scenarios

FROM node:18.9.0-buster-slim
WORKDIR /app

RUN \
    apt update && \
    apt install -y openssl build-essential libgssapi-krb5-2 git ccache python3

COPY . .

RUN echo "Updating yarn" && \
    corepack enable

RUN echo "Running yarn install" && \
    yarn --immutable

RUN \
    echo "Running yarn build" && \
    yarn pre-build && \
    yarn build-ts

# ENTRYPOINT ["/usr/bin/env", "node", "./packages/cli/bin/salto"]

Note: I see the same behavior if I leave in ENTRYPOINT and do interactive operations from the host system; but I wanted to get as close to showing the issue as possible which meant executing things inside the image so that I could reduce the impact that the host may have contributed to the execution error.

On a clean AWS EC2 instance (x86-64) - 🟩 SUCCESS 🟩

[Amazon Linux: al2023-ami-2023.5.20240708.0-kernel-6.1-x86_64]

EC2> sudo yum -y upgrade
EC2> sudo yum -y install git
EC2> sudo yum -y install docker
EC2> sudo systemctl start docker

EC2> cd ~
EC2> git clone git@github.com:salto-io/salto.git
EC2> cd salto
EC2> sudo docker build -t salto-cli-x86 .

EC2> sudo docker run -it salto-cli-x86 /bin/bash

EC2-DOCKER> mkdir ~/my-workspace
EC2-DOCKER> cd ~/my-workspace
EC2-DOCKER> node /app/packages/cli/dist/src/main.js init --env-name production
Initiated empty workspace
Use `salto account add <service-name>` to add accounts to the environment

EC2-DOCKER> 

On MacBook Pro 14.5 (M1 Pro / arm-64) - 🟥 FAILED 🟥

MAC> mkdir ~/Desktop/salto-docker-build-test
MAC> cd ~/Desktop/salto-docker-build-test

MAC> git clone git@github.com:salto-io/salto.git
MAC> cd salto
MAC> docker build -t salto-cli-arm .

MAC> docker run -it salto-cli-arm /bin/bash

MAC-DOCKER> mkdir ~/my-workspace
MAC-DOCKER> cd ~/my-workspace
MAC-DOCKER> node /app/packages/cli/dist/src/main.js init --env-name production

/app/packages/core/src/local-workspace/remote_map/remote_map.ts:339
  const newDb = getRemoteDbImpl()(loc)
                                 ^
TypeError: getRemoteDbImpl(...) is not a function
    at getOpenDBConnection (/app/packages/core/src/local-workspace/remote_map/remote_map.ts:339:34)
    at createDBConnections (/app/packages/core/src/local-workspace/remote_map/remote_map.ts:638:31)
    at /app/node_modules/async-lock/lib/index.js:125:12
    at AsyncLock._promiseTry (/app/node_modules/async-lock/lib/index.js:249:31)
    at exec (/app/node_modules/async-lock/lib/index.js:124:9)
    at AsyncLock.acquire (/app/node_modules/async-lock/lib/index.js:140:3)
    at withCreatorLock (/app/packages/core/src/local-workspace/remote_map/remote_map.ts:329:21)
    at /app/packages/core/src/local-workspace/remote_map/remote_map.ts:673:11
MAC-DOCKER> 

Experiment: Run salto-cli-x86 on the MacBook Pro (arm-64) - 🟧 MOSTLY SUCCESS 🟧

EC2> sudo docker save -o ~/salto-cli-x86.tar salto-cli-x86
EC2> sudo chown ec2-user ~/salto-cli-x86.tar

MAC> scp -rp ec2-user@domain:/home/ec2-user/salto-cli-x86.tar salto-cli-x86.tar
MAC> docker load -i salto-cli-x86.tar

MAC> docker run -it salto-cli-x86 /bin/bash
WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested
MAC-DOCKER> mkdir ~/my-workspace
MAC-DOCKER> cd ~/my-workspace
MAC-DOCKER> node /app/packages/cli/dist/src/main.js init --env-name production
Initiated empty workspace
Use `salto account add <service-name>` to add accounts to the environment

MAC-DOCKER> 

Experiment: Run salto-cli-arm on the AWS EC2 instance (x86-64) - 🟥 FAILED 🟥

MAC> docker save -o salto-cli-arm.tar salto-cli-arm
MAC> scp -rp salto-cli-arm.tar ec2-user@domain:/home/ec2-user/salto-cli-arm.tar 

EC2> sudo docker load -i /home/ec2-user/salto-cli-arm.tar
EC2> sudo docker run -it salto-cli-arm /bin/bash
WARNING: The requested image's platform (linux/arm64/v8) does not match the detected host platform (linux/amd64/v4) and no specific platform was requested
exec /usr/local/bin/docker-entrypoint.sh: exec format error
EC2> 

[thilgen]

Thanks to the suggestion from @gilhoffer - I had half the answer right in front of me - that the x86 arch sort of works on M1-Arm - with a small tweak.

The way to get this to work on my M1 Mac was to both build and execute the image specifying the linux/amd64 architecture - so that Docker and the Mac would use Rosetta to emulate the x86 behavior.

MAC> mkdir ~/Desktop/salto-docker-build-test
MAC> cd ~/Desktop/salto-docker-build-test

MAC> git clone git@github.com:salto-io/salto.git
MAC> cd salto
MAC> docker build -t salto-cli-x86-on-arm --platform linux/amd64 .

MAC> docker run -it --platform linux/amd64 salto-cli-x86-on-arm /bin/bash

MAC-DOCKER> mkdir ~/my-workspace
MAC-DOCKER> cd ~/my-workspace
MAC-DOCKER> node /app/packages/cli/dist/src/main.js init --env-name production
Initiated empty workspace
Use `salto account add <service-name>` to add accounts to the environment

MAC-DOCKER> 

[thilgen]

Is this worth adding a note to the Docker instructions in (https://github.com/salto-io/salto/blob/main/README.md) that M1 users can try this approach? Or would it be preferred to just not mention it?

FWIW - I do think it would be good to state somewhere on that page that only x86 is officially supported (that would have saved me some time trying to figure out what was going on here).