Add helix image for armv6 raspbian #1083

sbomer · 2024-05-30T23:55:21Z

The raspbian 10 helix images were removed in #927, so removing both raspbian/10 Dockerfiles and replacing them with raspbian 12 that has recent enough glibc.

src/raspbian/12/helix/arm32v6/Dockerfile

- Use --platform for base image - Specify bookworm in build-rootfs - Set osVersion in manifest.json - Fix raspbian version in manifest.json

eng/common/templates/stages/dotnet/build-test-publish-repo.yml

src/raspbian/manifest.json

akoeplinger · 2024-06-18T07:46:23Z

src/raspbian/12/helix/arm32v6/Dockerfile

@@ -1,4 +1,4 @@
-FROM mcr.microsoft.com/dotnet-buildtools/prereqs:raspbian-10-crossdeps-local as builder
+FROM --platform=linux/arm/v7 mcr.microsoft.com/dotnet-buildtools/prereqs:raspbian-12-crossdeps-local as builder


why are we passing --platform here?

Should it be linux/arm/v6?

It shouldn't be necessary at all. That would only be necessary if you were pulling a multi-arch tag which isn't the case here.

Without --platform I am getting the following in ci:

Dockerfile:1 -------------------- 1 | >>> FROM mcr.microsoft.com/dotnet-buildtools/prereqs:raspbian-12-crossdeps-local as builder 2 | 3 | ARG ROOTFS_DIR=/crossrootfs -------------------- ERROR: failed to solve: mcr.microsoft.com/dotnet-buildtools/prereqs:raspbian-12-crossdeps-local: mcr.microsoft.com/dotnet-buildtools/prereqs:raspbian-12-crossdeps-local: not found at Microsoft.DotNet.ImageBuilder.ExecuteHelper.Execute(ProcessStartInfo info, Func`2 executor, Boolean isDryRun, String errorMessage, String executeMessageOverride) in /image-builder/src/ExecuteHelper.cs:line 101 at Microsoft.DotNet.ImageBuilder.ExecuteHelper.ExecuteWithRetry(ProcessStartInfo info, Action`1 processStartedCallback, Boolean isDryRun, String errorMessage, String executeMessageOverride) in /image-builder/src/ExecuteHelper.cs:line 60 at Microsoft.DotNet.ImageBuilder.ExecuteHelper.ExecuteWithRetry(String fileName, String args, Boolean isDryRun, String errorMessage, String executeMessageOverride) in /image-builder/src/ExecuteHelper.cs:line 45 at Microsoft.DotNet.ImageBuilder.DockerService.BuildImage(String dockerfilePath, String buildContextPath, String platform, IEnumerable`1 tags, IDictionary`2 buildArgs, Boolean isRetryEnabled, Boolean isDryRun) in /image-builder/src/DockerService.cs:line 60 at Microsoft.DotNet.ImageBuilder.DockerServiceCache.BuildImage(String dockerfilePath, String buildContextPath, String platform, IEnumerable`1 tags, IDictionary`2 buildArgs, Boolean isRetryEnabled, Boolean isDryRun) in /image-builder/src/DockerServiceCache.cs:line 37 at Microsoft.DotNet.ImageBuilder.Commands.BuildCommand.BuildImage(PlatformInfo platform, IEnumerable`1 allTags) in /image-builder/src/Commands/BuildCommand.cs:line 483

IIRC I could build this locally with --platform. I think v7 is right because the raspbian-12-crossdeps image says v7 in manifest.json. Let's see if this works in ci.

If we are going to replace "armv6" image and new image doesn't provide armv6, then it is not the right image? I think uname -a etc. should show armv6l. v7 and v6 are very different ISAs; which is why we have ARM and ARMV6 as separate architectures.

Without --platform I am getting the following in ci:

Dockerfile:1 -------------------- 1 | >>> FROM mcr.microsoft.com/dotnet-buildtools/prereqs:raspbian-12-crossdeps-local as builder 2 | 3 | ARG ROOTFS_DIR=/crossrootfs -------------------- ERROR: failed to solve: mcr.microsoft.com/dotnet-buildtools/prereqs:raspbian-12-crossdeps-local: mcr.microsoft.com/dotnet-buildtools/prereqs:raspbian-12-crossdeps-local: not found at Microsoft.DotNet.ImageBuilder.ExecuteHelper.Execute(ProcessStartInfo info, Func`2 executor, Boolean isDryRun, String errorMessage, String executeMessageOverride) in /image-builder/src/ExecuteHelper.cs:line 101 at Microsoft.DotNet.ImageBuilder.ExecuteHelper.ExecuteWithRetry(ProcessStartInfo info, Action`1 processStartedCallback, Boolean isDryRun, String errorMessage, String executeMessageOverride) in /image-builder/src/ExecuteHelper.cs:line 60 at Microsoft.DotNet.ImageBuilder.ExecuteHelper.ExecuteWithRetry(String fileName, String args, Boolean isDryRun, String errorMessage, String executeMessageOverride) in /image-builder/src/ExecuteHelper.cs:line 45 at Microsoft.DotNet.ImageBuilder.DockerService.BuildImage(String dockerfilePath, String buildContextPath, String platform, IEnumerable`1 tags, IDictionary`2 buildArgs, Boolean isRetryEnabled, Boolean isDryRun) in /image-builder/src/DockerService.cs:line 60 at Microsoft.DotNet.ImageBuilder.DockerServiceCache.BuildImage(String dockerfilePath, String buildContextPath, String platform, IEnumerable`1 tags, IDictionary`2 buildArgs, Boolean isRetryEnabled, Boolean isDryRun) in /image-builder/src/DockerServiceCache.cs:line 37 at Microsoft.DotNet.ImageBuilder.Commands.BuildCommand.BuildImage(PlatformInfo platform, IEnumerable`1 allTags) in /image-builder/src/Commands/BuildCommand.cs:line 483

This doesn't make sense to me. This happens on the CI in AzDO? I can't find a failed build with that error. I'd like to see the build log to understand what's happening. Getting a not found error would mean the image tag doesn't exist at all, regardless of its platform.

If we are going to replace "armv6" image and new image doesn't provide armv6, then it is not the right image?

I am trying to follow the setup introduced in https://github.com/dotnet/dotnet-buildtools-prereqs-docker/pull/550/files#diff-d5da67421f1a1e08a8982140c63e4c621c9332ea33774a4db797a3dc334261b9 where the crossdeps image uses armv7, and it is used to build a rootfs for an armv6 helix image. The intention of adding --platform is to specify the armv7 variant for the crossdeps image, even though we are targeting armv6 in the final stage of this dockerfile. That's the idea, but not sure if our infrastructure supports this.

@mthalman see the build failures without --platform here.

Locally, if I do docker buildx build --platform=linux/arm/v6 . and I leave out --platform=linux/arm/v7 from the base image, I get this:

BUILDKIT_PROGRESS=plain docker buildx build --platform=linux/arm/v6 . #0 building with "default" instance using docker driver #1 [internal] load build definition from Dockerfile #1 DONE 0.0s #1 [internal] load build definition from Dockerfile #1 transferring dockerfile: 2.71kB done #1 DONE 0.0s #2 [internal] load metadata for mcr.microsoft.com/dotnet-buildtools/prereqs:raspbian-12-crossdeps-local #2 ERROR: mcr.microsoft.com/dotnet-buildtools/prereqs:raspbian-12-crossdeps-local: not found ------ > [internal] load metadata for mcr.microsoft.com/dotnet-buildtools/prereqs:raspbian-12-crossdeps-local: ------ Dockerfile:3 -------------------- 1 | ARG ROOTFS_DIR=/crossrootfs 2 | 3 | >>> FROM mcr.microsoft.com/dotnet-buildtools/prereqs:raspbian-12-crossdeps-local as builder 4 | 5 | ARG ROOTFS_DIR -------------------- ERROR: failed to solve: mcr.microsoft.com/dotnet-buildtools/prereqs:raspbian-12-crossdeps-local: failed to resolve source metadata for mcr.microsoft.com/dotnet-buildtools/prereqs:raspbian-12-crossdeps-local: mcr.microsoft.com/dotnet-buildtools/prereqs:raspbian-12-crossdeps-local: not found

Which looks similar to the CI failure. If I keep --platform=linux/arm/v7 in the dockerfile, it finds the image (which I built using --platform=linux/arm/v7 and tagged locally).

Ok, I get why --platform is needed now. That's fine. I'll leave it to the other reviewers to determine whether consuming a v7 image from a v6 image is the right thing to do.

This reverts commit 7b1bad6.

This reverts commit e59bdc7.

- Pass --platform=linux/arm/v7 - Specify lldb version

Required to build cryptography wheel

MichaelSimons · 2024-08-15T21:47:36Z

This PR will need to be updated to respond to the following infra changes that were recently made:

MichaelSimons · 2024-08-23T19:28:38Z

@sbomer - Friendly ping. Is this something you intend to complete? This PR is pretty old now.

mthalman · 2024-10-31T16:08:36Z

@sbomer - Friendly ping. Is this something you intend to complete? This PR is pretty old now.

@sbomer - Any update here? Note that I've deleted all of the Raspbian 10 Dockerfiles because they were based on Ubuntu 18.04 which is EOL.

sbomer · 2024-10-31T17:40:58Z

I was hitting some blocking issues last time I looked at this. I'll at least update the PR to see if this is close to working. I don't have much time to work on this, so if it's not straightforward to fix I'll close this out for now.

akoeplinger · 2024-10-31T18:04:55Z

src/raspbian/12/helix/arm32v6/Dockerfile

@@ -9,7 +9,7 @@ RUN git config --global user.email builder@dotnet-buildtools-prereqs-docker && \
    git clone --depth 1 --single-branch https://github.com/dotnet/arcade /scripts

 # Build the rootfs
-RUN /scripts/eng/common/cross/build-rootfs.sh armv6 raspbian --skipunmount
+RUN /scripts/eng/common/cross/build-rootfs.sh armv6 bookworm lldb15 --skipunmount


@am11 FYI we can probably remove the code in https://github.com/dotnet/arcade/blob/d1bc0779143e410ec82696f9865c4eb407db8d67/eng/common/cross/build-rootfs.sh#L170-L183 which sets the __UbuntuRepo, __CodeName etc since that is handled by the bookworm argument

akoeplinger · 2024-10-31T18:06:47Z

src/raspbian/12/helix/arm32v6/Dockerfile

        python3-dev \
        python3-pip \
+        rustc \


https://github.com/dotnet/dotnet-buildtools-prereqs-docker/pull/1145/files installs rust from rustup instead because apparently the rustc in bookworm is too old?

akoeplinger · 2024-10-31T18:07:02Z

src/raspbian/12/helix/arm32v6/Dockerfile

@@ -60,7 +62,7 @@ ENV LANG=en_US.utf8
 RUN ln -sf /usr/bin/python3 /usr/bin/python && \
    pip download --no-deps helix-scripts --index-url https://dnceng.pkgs.visualstudio.com/public/_packaging/helix-client-prod/pypi/simple && \
    export CRYPTOGRAPHY_DONT_BUILD_RUST=1 && \
-    pip install ./helix_scripts-*-py3-none-any.whl
+    pip install ./helix_scripts-*-py3-none-any.whl --break-system-packages


please add a comment why we need --break-system-packages

Alternatively, could we use the multistage docker + python venv approach we've done in all the other images and avoid using --break-system-packages (#1142)?

Add helix image for armv6 raspbian

20ecbbb

sbomer requested a review from jkoritzinsky May 30, 2024 23:55

sbomer requested a review from a team as a code owner May 30, 2024 23:55

dotnet-issue-labeler bot added area-dockerfiles untriaged labels May 30, 2024

sbomer commented May 30, 2024

View reviewed changes

src/raspbian/12/helix/arm32v6/Dockerfile Outdated Show resolved Hide resolved

jkoritzinsky approved these changes May 30, 2024

View reviewed changes

sbomer added 3 commits May 31, 2024 16:47

Some fixes

7afb569

- Use --platform for base image - Specify bookworm in build-rootfs - Set osVersion in manifest.json - Fix raspbian version in manifest.json

Increase timeout

8667ab0

Propagate linuxArmBuildJobTimeout

360f9ce

mthalman reviewed Jun 10, 2024

View reviewed changes

eng/common/templates/stages/dotnet/build-test-publish-repo.yml Outdated Show resolved Hide resolved

sbomer mentioned this pull request Jun 11, 2024

Building rootfs fails for ubuntu-22.04-cross-riscv64 #1088

Closed

Merge branch 'main' into raspbianHelix

7d24c9c

akoeplinger reviewed Jun 18, 2024

View reviewed changes

sbomer added 3 commits June 18, 2024 16:08

Merge branch 'main' into raspbianHelix

232c599

Remove --platform

7b1bad6

Revert "Remove --platform"

e59bdc7

This reverts commit 7b1bad6.

sbomer mentioned this pull request Jul 3, 2024

Improve build error on systems that have a 32-bit time_t dotnet/runtime#104368

Merged

sbomer added 8 commits July 3, 2024 20:30

Revert "Revert "Remove --platform""

95a10f5

This reverts commit e59bdc7.

Merge remote-tracking branch 'origin/main' into raspbianHelix

832419e

Fix rootfs build

e5366a6

- Pass --platform=linux/arm/v7 - Specify lldb version

Avoid multiple pip installs and use venv

131b3d4

Pass --break-system-packages

c28e26d

Install rust

2981dfd

Required to build cryptography wheel

Increase timeout

989dbf0

Install pkg-config, increase timeout

f4ed118

am11 mentioned this pull request Jul 16, 2024

Disable armv6 in runtime-community pipeline dotnet/runtime#104951

Merged

This was referenced Oct 31, 2024

Update usage of EOL Raspbian 10 Helix container image dotnet/runtime#109420

Open

Raspbian Dockerfiles should include a package upgrade by default #1238

Closed

Merge remote-tracking branch 'origin/main' into raspbianHelix

3a2003b

sbomer requested review from a team as code owners October 31, 2024 17:45

Update tag

f410e6a

akoeplinger reviewed Oct 31, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add helix image for armv6 raspbian #1083

Add helix image for armv6 raspbian #1083

sbomer commented May 30, 2024

akoeplinger Jun 18, 2024

am11 Jun 18, 2024

mthalman Jun 18, 2024

sbomer Jun 18, 2024

am11 Jun 18, 2024

mthalman Jun 19, 2024

sbomer Jul 3, 2024

sbomer Jul 3, 2024

mthalman Jul 8, 2024

MichaelSimons commented Aug 15, 2024

MichaelSimons commented Aug 23, 2024

mthalman commented Oct 31, 2024

sbomer commented Oct 31, 2024 •

edited

Loading

akoeplinger Oct 31, 2024 •

edited

Loading

akoeplinger Oct 31, 2024

akoeplinger Oct 31, 2024

jkoritzinsky Oct 31, 2024

		@@ -1,4 +1,4 @@
		FROM mcr.microsoft.com/dotnet-buildtools/prereqs:raspbian-10-crossdeps-local as builder
		FROM --platform=linux/arm/v7 mcr.microsoft.com/dotnet-buildtools/prereqs:raspbian-12-crossdeps-local as builder

Add helix image for armv6 raspbian #1083

Are you sure you want to change the base?

Add helix image for armv6 raspbian #1083

Conversation

sbomer commented May 30, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

MichaelSimons commented Aug 15, 2024

MichaelSimons commented Aug 23, 2024

mthalman commented Oct 31, 2024

sbomer commented Oct 31, 2024 • edited Loading

akoeplinger Oct 31, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sbomer commented Oct 31, 2024 •

edited

Loading

akoeplinger Oct 31, 2024 •

edited

Loading