[Bug]: vscode devcontainers "features" feature fails on Windows and MacOS

[JustinGrote]

Issue Description

Dev containers features fails on the podman buildx command, it looks like the temp file doesn't get copied correctly by the builder.

Steps to reproduce the issue

Steps to reproduce the issue

1. Create a devcontainer with a feature - https://code.visualstudio.com/blogs/2022/09/15/dev-container-features
2. Open the devcontainer in local vscode

Simple Example:

// For format details, see https://aka.ms/devcontainer.json. For config options, see the
// README at: https://github.com/devcontainers/templates/tree/main/src/dotnet
{
  "name": "PowerShell Binary Module",
  // Or use a Dockerfile or Docker Compose file. More info: https://containers.dev/guide/dockerfile
  "image": "mcr.microsoft.com/devcontainers/dotnet:0-7.0",
  "features": {
     "ghcr.io/devcontainers/features/powershell:1": {
       "version": "latest"
     }
  },
  "containerUser": "vscode"
}

Describe the results you received

[4484 ms] Start: Run: podman buildx build --load --build-context dev_containers_feature_content_source=C:\Users\jgrote\AppData\Local\Temp\devcontainercli\container-features\0.25.2-1675271780761 --build-arg _DEV_CONTAINERS_BASE_IMAGE=mcr.microsoft.com/devcontainers/dotnet:0-7.0 --build-arg _DEV_CONTAINERS_IMAGE_USER=root --build-arg _DEV_CONTAINERS_FEATURE_CONTENT_SOURCE=dev_container_feature_content_temp --target dev_containers_target_stage -t vsc-yourfirstpsmoduleincsharp-a2b143a26a2888997d906577ede0f015-features -f C:\Users\jgrote\AppData\Local\Temp\devcontainercli\container-features\0.25.2-1675271780761\Dockerfile.extended c:\Users\jgrote\AppData\Roaming\Code - Insiders\User\globalStorage\ms-vscode-remote.remote-containers\data\empty-folder
time="2023-02-01T09:16:22-08:00" level=error msg="While reading directory C:\\Us
ers\\jgrote\\AppData\\Roaming\\Code - Insiders\\User\\globalStorage\\ms-vscode-r
emote.remote-containers\\data\\empty-folder: EOF"
Error: stat /var/tmp/libpod_builder4232675966/build/C:/Users/jgrote/AppData/Loca
l/Temp/devcontainercli/container-features/0.25.2-1675271780761/Dockerfile.extend
ed: no such file or directory

[5085 ms] Error: Command failed: podman buildx build --load --build-context dev_containers_feature_content_source=C:\Users\jgrote\AppData\Local\Temp\devcontainercli\container-features\0.25.2-1675271780761 --build-arg _DEV_CONTAINERS_BASE_IMAGE=mcr.microsoft.com/devcontainers/dotnet:0-7.0 --build-arg _DEV_CONTAINERS_IMAGE_USER=root --build-arg _DEV_CONTAINERS_FEATURE_CONTENT_SOURCE=dev_container_feature_content_temp --target dev_containers_target_stage -t vsc-yourfirstpsmoduleincsharp-a2b143a26a2888997d906577ede0f015-features -f C:\Users\jgrote\AppData\Local\Temp\devcontainercli\container-features\0.25.2-1675271780761\Dockerfile.extended c:\Users\jgrote\AppData\Roaming\Code - Insiders\User\globalStorage\ms-vscode-remote.remote-containers\data\empty-folder
[5086 ms]     at Aoe (c:\Users\jgrote\.vscode-insiders\extensions\ms-vscode-remote.remote-containers-0.266.1\dist\spec-node\devContainersSpecCLI.js:1893:1355)
[5086 ms]     at uT (c:\Users\jgrote\.vscode-insiders\extensions\ms-vscode-remote.remote-containers-0.266.1\dist\spec-node\devContainersSpecCLI.js:1893:1291)
[5086 ms]     at process.processTicksAndRejections (node:internal/process/task_queues:96:5)
[5086 ms]     at async Poe (c:\Users\jgrote\.vscode-insiders\extensions\ms-vscode-remote.remote-containers-0.266.1\dist\spec-node\devContainersSpecCLI.js:1899:2128)
[5086 ms]     at async Zf (c:\Users\jgrote\.vscode-insiders\extensions\ms-vscode-remote.remote-containers-0.266.1\dist\spec-node\devContainersSpecCLI.js:1899:3278)
[5086 ms]     at async aue (c:\Users\jgrote\.vscode-insiders\extensions\ms-vscode-remote.remote-containers-0.266.1\dist\spec-node\devContainersSpecCLI.js:2020:15276)
[5087 ms]     at async oue (c:\Users\jgrote\.vscode-insiders\extensions\ms-vscode-remote.remote-containers-0.266.1\dist\spec-node\devContainersSpecCLI.js:2020:15030)
[5097 ms] Exit code 1
[5101 ms] Command failed: C:\Users\jgrote\AppData\Local\Programs\Microsoft VS Code Insiders\Code - Insiders.exe --ms-enable-electron-run-as-node c:\Users\jgrote\.vscode-insiders\extensions\ms-vscode-remote.remote-containers-0.266.1\dist\spec-node\devContainersSpecCLI.js up --user-data-folder c:\Users\jgrote\AppData\Roaming\Code - Insiders\User\globalStorage\ms-vscode-remote.remote-containers\data --docker-path podman --docker-compose-path podman-compose --workspace-folder c:\Users\jgrote\Projects\YourFirstPSModuleInCSharp --workspace-mount-consistency cached --id-label devcontainer.local_folder=c:\Users\jgrote\Projects\YourFirstPSModuleInCSharp --log-level debug --log-format json --config c:\Users\jgrote\Projects\YourFirstPSModuleInCSharp\.devcontainer\devcontainer.json --default-user-env-probe loginInteractiveShell --mount type=volume,source=vscode,target=/vscode,external=true --skip-post-create --update-remote-user-uid-default on --mount-workspace-git-root true

Describe the results you expected

Describe the results you expected

podman info output

If you are unable to run podman info for any reason, please provide the podman version, operating system and its version and the architecture you are running.

Podman in a container

No

Privileged Or Rootless

Rootless

Upstream Latest Release

Yes

Additional environment details

Additional environment details

Additional information

Additional information like issue happens only occasionally or issue happens with a particular architecture or on a particular setting

[vrothberg]

Thanks for reaching out, @JustinGrote!

@flouthoc PTAL

[3belut]

I have the same issue, also probably the same as microsoft/vscode-remote-release#7408.

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[mikegoatly]

See also here: microsoft/vscode-remote-release#8103

I'm running into this as well using the latest version of podman on Windows (currently 4.4.1). It looks to me that the absolute path not being forwarded correctly to podman. For example, in the OP the following is being passed to podman:

-f C:\Users\jgrote\AppData\Local\Temp\devcontainercli\container-features\0.25.2-1675271780761\Dockerfile.extended

But the path that's being reported as not found in podman is:

/var/tmp/libpod_builder4232675966/build/C:/Users/jgrote/AppData/Local/Temp/devcontainercli/container-features/0.25.2-1675271780761/Dockerfile.extended

I would have thought it should be mapped to:

/mnt/c/Users/jgrote/AppData/Local/Temp/devcontainercli/container-features/0.25.2-1675271780761/Dockerfile.extended

[G3N7]

Same thing happening in 4.4.4

[Jacques-Peeters]

Same thing happening in 4.4.2 and 4.4.4 on Mac M1.

Managed to make it work by uninstalling, manually installing 4.3.1 .

[acidghost]

This is happening on 4.5.0 too, on macOS M1. Even with a fresh VM.

[dmorand17]

I'm experiencing the same behavior on 4.5.0 on M1 Mac

[ReeSilva]

The same on Windows and Podman 4.5.0

[aacebedo]

Having the same problem here

[JustinGrote]

Hey everyone please thumbs up the top issue comment if you are having it, do not "me too" comment, all you do is ping notifications to everyone.

As of now there is no workaround as podman does not handle the paths provided by features on Windows/Mac, so if you need to use devcontainers with features, docker seems to be the only option ATM or use Linux as your host

EDIT: Apparently as of 4.6.0-rc1 there is a potential fix: #17313 (comment)

[aacebedo]

Just to add that this is specific to MacOS and Windows podman. The same files on linux works without any problem.

[nobodyman1]

I had the same issue using Podman v4.5.1.
I updated to Podman v4.6.0-rc1 and set dev.containers.mountWaylandSocket: false.
Now it´s working!

[MRDGH2821]

Along with this, I have some additional errors

[2023-07-22T04:36:19.899Z] Start: Run: podman buildx build --load --build-context dev_containers_feature_content_source=C:\Users\mihir\AppData\Local\Temp\devcontainercli\container-features\0.46.0-1690000563663 --build-arg _DEV_CONTAINERS_BASE_IMAGE=mcr.microsoft.com/devcontainers/typescript-node:18 --build-arg _DEV_CONTAINERS_IMAGE_USER=root --build-arg _DEV_CONTAINERS_FEATURE_CONTENT_SOURCE=dev_container_feature_content_temp --target dev_containers_target_stage -t vsc-discord-ban-utils-bot-1aa5cea51a6a29accd4e5bf90982d33d7326a6face5746c5a4de512e457efaa4-features -f C:\Users\mihir\AppData\Local\Temp\devcontainercli\container-features\0.46.0-1690000563663\Dockerfile.extended c:\Users\mihir\AppData\Roaming\Code\User\globalStorage\ms-vscode-remote.remote-containers\data\empty-folder
[2023-07-22T04:36:19.981Z] 
[2023-07-22T04:36:20.163Z] time="2023-07-22T10:06:20+05:30" level=error msg="While reading directory C:\\Us
ers\\mihir\\AppData\\Roaming\\Code\\User\\globalStorage\\ms-vscode-remote.remote
-containers\\data\\empty-folder: EOF"
[2023-07-22T04:36:20.257Z] [1/2] STEP 1/4: FROM mcr.microsoft.com/devcontainers/typescript-node:18 AS dev_c
ontainers_feature_content_normalize
[2023-07-22T04:36:20.288Z] [1/2] STEP 2/4: USER root
--> Using cache f02a69516a9b42331906b7d959d7ee3ef9d900eb2d63711a6b2e39db0bd56698
--> f02a69516a9b
[1/2] STEP 3/4: COPY --from=dev_containers_feature_content_source devcontainer-f
eatures.builtin.env /tmp/build-features/
[2023-07-22T04:36:20.320Z] Error: building at STEP "COPY --from=dev_containers_feature_content_source devco
ntainer-features.builtin.env /tmp/build-features/": checking on sources under "/
home/user/C:\\Users\\mihir\\AppData\\Local\\Temp\\devcontainercli\\container-fea
tures\\0.46.0-1690000563663": error in copier subprocess: changing to intended-n
ew-root directory "/home/user/C:\\Users\\mihir\\AppData\\Local\\Temp\\devcontain
ercli\\container-features\\0.46.0-1690000563663": chdir /home/user/C:\Users\mihi
r\AppData\Local\Temp\devcontainercli\container-features\0.46.0-1690000563663: no
 such file or directory

Complete log
remoteContainers-2023-07-22T04-35-58.871Z.log

I posted here too - https://stackoverflow.com/questions/76739312/devcontainer-build-error-when-using-podman-with-vscode

[MihaMarkic]

Confirming that it works for me on macOS M2 as well with podman 4.6.0 - no need to fiddle with wayland settings (that is Windows specific I think)

[fsdrw08]

still have same problem in windows 11, with podman v4.6.1

[todaynowork]

still have the same issue in Mac Apple M1, with podman v4.6.2

[todaynowork]

still have the same issue in Mac Apple M1, with podman v4.7.0

[hansmbakker]

I have this issue with Windows 11 and podman 4.7.2

[nkowne63]

I have this issue with Windows 11 and podman 4.8.2

[azenkwed]

I have this issue with Windows 11 and podman 4.8.0

[JustinGrote]

Just a bump that this is the big reason I don't use podman and continue to use docker on Windows...

[rhatdan]

@n1hility Thoughts?

[GarmannF]

I have the same issue, it's a show stopper for using podman for me.

[lmdkfs]

still have the same issue in Mac Apple Intel , with podman 4.8.3

[martin-dziura-ot]

Still having an issue on Windows with podman 4.9.3

But I want to add a finding, that might be helpful: My username contains a white space (which is common in Windows) and this might cause the problem. The failing command looks like this:

Error: Command failed: podman buildx build --load --build-context dev_containers_feature_content_source=C:\Users\MYNAME~1\AppData\Local\Temp\devcontainercli\container-features\xxx --build-arg _DEV_CONTAINERS_BASE_IMAGE=mcr.microsoft.com/devcontainers/base:bullseye --build-arg _DEV_CONTAINERS_IMAGE_USER=root --build-arg _DEV_CONTAINERS_FEATURE_CONTENT_SOURCE=dev_container_feature_content_temp --target dev_containers_target_stage -t vsc-xxx-features -f C:\Users\MYNAME~1\AppData\Local\Temp\devcontainercli\container-features\xxx\Dockerfile.extended c:\Users\Myname Surname\AppData\Roaming\Code\User\globalStorage\ms-vscode-remote.remote-containers\data\empty-folder

The call gets interpreted as a command with two args, one being c:\Users\Myname, the other being Surname\AppData\Roaming\Code\User\globalStorage\ms-vscode-remote.remote-containers\data\empty-folder. Manually calling this leads to Error: accepts at most 1 arg(s), received 2.

Simply adding quotes to the arg in that call would solve the issue. Does anyone of the contributors know where to fix that?

[rajyraman]

I had the same issue using Podman v4.5.1. I updated to Podman v4.6.0-rc1 and set dev.containers.mountWaylandSocket: false. Now it´s working!

Podman 5.0.2. Does not work on Windows. I had to add dotnet installation on to the Dockerfile.

[usma0118]

+1 facing with podman v5.0.2 on mac

[virzak]

This is the most voted on issue in this repo. Could it please get some attention?

[rhatdan]

@l0rd Thoughts?

[l0rd]

Looking at it. The VS Code dev containers extension documentation talks about some limitations when using podman, but that may be outdated as it refers to podman v1.9. @JustinGrote or @virzak can you please share the configuration settings you are using (podman and vscode side)?

update: I have identified the cause of the problem. The issue is with the option --build-context <label>=<path> of podman build. The context <path> is not re-mapped as a path in the guest OS. It is mentioned in the documentation. I am looking at the source code to see how hard it is to fix.

[l0rd]

The additional build contexts specified with --build-context are not included in the build context tarball (and are not copied to the remote machine). To address that we should add the additionalBuildContexts to the tarContent too, and update the additional contexts paths.

@mheon is that something that we plan to work on in the context of the buildx APIs? Otherwise I can give it a try during this sprint.

Also, this issue is related (the same?) to #18691 and #18840

[mheon]

I don't think this is tracked by the buildx work - another card would be best

[futuralogic]

Good news.

Latest podman 5.3.0-rc1 fixes this. Was mentioned in one of the comments that it would likely release in 5.3.x
https://github.com/containers/podman/releases/tag/v5.3.0-rc1

Verified vscode, Windows 11, devcontainer with a feature now builds correctly.

[Dark-Schneider-666]

Good news.

Latest podman 5.3.0-rc1 fixes this. Was mentioned in one of the comments that it would likely release in 5.3.x https://github.com/containers/podman/releases/tag/v5.3.0-rc1

Verified vscode, Windows 11, devcontainer with a feature now builds correctly.

Maybe on Windows. Tested on Mac with Podman 5.3.0 final release and podman-compose version 1.2.0 and the same "command failed". Tested setting in vscode:

• podman and podman-compose.
• podman and docker-compose.

Podman with Docker compatibility enabled.

[mheon]

Please file a new issue, as you are likely encountering a separate problem.

[Dark-Schneider-666]

Well I made some testing and expose so if could help for something. All this is for Mac with Apple Silicon.

Currently what seems to work with vscode is offical Docker Desktop distribution. Uninstalled and installing Colima and got the very same failure than with Podman (Desktop).
But, when I started with Dev Containers development on vscode was using Colima, just installing Colima and docker and everything ran nicely. What changed is that now it required me to install the buildx cli-plugin not required time ago, and then I got the same failure 'command failed...exit code 1'. Or sometimes it builds (i.e. the Rails dev container) but cannot open the folder on the dev container properly, it opens an empty workspace environment while using Docker Desktop opens correctly the folder.

Then, maybe the official docker distribution included with Docker Desktop (that installs its own executables) have some tweaks? And vscode plugins are developed and tested using only this distribution?.

Maybe could be at virtualization layer? In both Podman and Colima (using community docker distribution binaries) can configure only the virtualization layer (QEMU or Apple, and if use Rosseta), but on Docker Desktop I have another option that is file sharing implementation, with 3 options: VirtioFS, gRPC FUSE, osxfs (Legacy). With VirtioFS selected as default. Different implementation using VirtioFS?

Found this related: rust-lang/docker-rust#161

So if the issue is fixed on Windows doesn't surprise me as it uses another virtualization layer, and currently be a Mac with Apple Silicon issue only.

Hope this info helps.