[debian bookworm] Python Feature fails to install with 'error: externally-managed-environment'

[joshspicer]

ref: #576

Test run: https://github.com/devcontainers/features/actions/runs/5293497423/jobs/9581722477#step:4:1608

[2023-06-16T19:24:31.534Z] #18 36.34 Processing triggers for libc-bin (2.36-9) ...
#18 36.39 Installing Python tools...
#18 36.42 Updating pip...

[2023-06-16T19:24:31.810Z] #18 36.77 error: externally-managed-environment
#18 36.77 
#18 36.77 × This environment is externally managed
#18 36.77 ╰─> To install Python packages system-wide, try apt install
#18 36.77     python3-xyz, where xyz is the package you are trying to
#18 36.77     install.
#18 36.77     
#18 36.77     If you wish to install a non-Debian-packaged Python package,
#18 36.77     create a virtual environment using python3 -m venv path/to/venv.
#18 36.77     Then use path/to/venv/bin/python and path/to/venv/bin/pip. Make
#18 36.77     sure you have python3-full installed.
#18 36.77     
#18 36.77     If you wish to install a non-Debian packaged Python application,
#18 36.77     it may be easiest to use pipx install xyz, which will manage a
#18 36.77     virtual environment for you. Make sure you have pipx installed.
#18 36.77     
#18 36.77     See /usr/share/doc/python3.11/README.venv for more information.
[2023-06-16T19:24:31.810Z] 
#18 36.77 
#18 36.77 note: If you believe this is a mistake, please contact your Python installation or OS distribution provider. You can override this, at the risk of breaking your Python installation or OS, by passing --break-system-packages.
#18 36.77 hint: See PEP 668 for the detailed specification.

[joshspicer]

Looks like the azure-cli (which attempts to install python) is effected the same way: https://github.com/devcontainers/features/actions/runs/5294360726/jobs/9583631989

[joshspicer]

Good info on this change: https://pythonspeed.com/articles/externally-managed-environment-pep-668/#:~:text=If%20you%20wish%20to%20install,you%20have%20python3%2Dfull%20installed.

[vermegi]

Affirmative on azure-cli install being effected by this change on Debian bookworm as well: Azure/azure-cli#26737 and #589

Reverting to -bullseye, this is (for now) working fine.

[jongio]

Add "version": "latest"

"ghcr.io/devcontainers/features/azure-cli:1": {
			"version": "latest"
		}

[alfredodeza]

From what I can tell in the logs, specifically in this section:

[2023-06-16T19:23:54.049Z] #17 11.11 error: externally-managed-environment
#17 11.11 
#17 11.11 × This environment is externally managed
#17 11.11 ╰─> To install Python packages system-wide, try apt install
#17 11.11     python3-xyz, where xyz is the package you are trying to
#17 11.11     install.

That happens when installing a package directly with pip onto system-installed Python. It looks like there is a suggestion right afterwards:

You can override this, at the risk of breaking your Python installation or OS, by passing --break-system-packages

This type of situation is usually avoided in development environments where developers will install libraries and packages in a virtualenv (which is also suggested).

I tried looking around the scripts that are running this test but couldn't figure out what exactly is doing these installs, but happy to offer some more help here if needed

[mering]

Azure/azure-cli#26737 seems to use pipx as a resolution

Other alternatives are listed here.

Another alternative would be to use the system packages if version == "os-installed".

[SebastianBalle]

I have similar issues when I try to install docker-outside-docker. I get the error:

#13 32.44 error: externally-managed-environment
#13 32.44 
#13 32.44 × This environment is externally managed
#13 32.44 ╰─> To install Python packages system-wide, try apt install
#13 32.44     python3-xyz, where xyz is the package you are trying to
#13 32.44     install.
#13 32.44     
#13 32.44     If you wish to install a non-Debian-packaged Python package,
#13 32.44     create a virtual environment using python3 -m venv path/to/venv.
#13 32.44     Then use path/to/venv/bin/python and path/to/venv/bin/pip. Make
#13 32.44     sure you have python3-full installed.
#13 32.44     
#13 32.44     If you wish to install a non-Debian packaged Python application,
#13 32.44     it may be easiest to use pipx install xyz, which will manage a
#13 32.44     virtual environment for you. Make sure you have pipx installed.
#13 32.44     
#13 32.44     See /usr/share/doc/python3.11/README.venv for more information.

Even if I install python3-full before running the script I still get the same error. I use the following base image: mcr.microsoft.com/vscode/devcontainers/base:1.0.4-bookworm.

Update 20231031

Using the mcr.microsoft.com/vscode/devcontainers/base:1.0.4-bookworm with an arm64 architecture will fail in the installation unless the default value of docker compose is changed from v1 to v2.
In general the default version for docker compose should be v2 as v1 is no longer supported. The default value should either be modified by the maintainers or overwritten by the user.

[jtbandes]

@SebastianBalle Do you know how to override this version to make the installation work on arm64? I tried editing devcontainer.json to look like this based on these docs:

  "features": {
    "ghcr.io/devcontainers/features/docker-in-docker:2": {
      "dockerDashComposeVersion": "v2"
    },

But that doesn't seem to work.

[SebastianBalle]

@jtbandes

We store the script hardcoded in our internal repository. We simply changed the default value as follow:

DOCKER_DASH_COMPOSE_VERSION=${7:-"v2"} # v1 or v2

In our devcontainer.json we have "dockerComposeFile": "docker-compose.yml", in which we have specified the Dockerfile that contains the above mentioned default value.

[samruddhikhandale]

In general the default version for docker compose should be v2 as v1 is no longer supported. The default value should either be modified by the maintainers or overwritten by the user.

Opened #745 for tracking v1 deprecation.

I should be able to get to #745 soon. Thanks for your patience.

[samruddhikhandale]

As Docker v1 is deprecated, and Docker v2 works fine with debain:bookworm, we can simply close this issue once #745 is fixed.

Reasoning: As docker v1 is out of support by Docker, similarly, we can deprecate it's support and ask users to switch to v2.

[Clockwork-Muse]

As Docker v1 is deprecated, and Docker v2 works fine with debain:bookworm, we can simply close this issue once #745 is fixed.

That's not sufficient. The python feature still fails to install on its own, regardless of whether or not any docker feature is installed. That is, the python feature is broken as well.

[Clockwork-Muse]

Side note: Attempting to get a "partially working" feature install (mostly by disabling tool install) results in a broken setup. In particular, pipx is installed, but attempts to reference global locations when called, meaning it fails due to permissions issues. At the moment, installing python/pipx via the apt-packages feature is working for me.

[0xMihir]

Yeah, I'm getting the same error with bookworm and v2:

{
	"name": "Rust",

	"image": "mcr.microsoft.com/devcontainers/rust:bookworm",
	"features": {
		"ghcr.io/devcontainers-contrib/features/apt-get-packages:2": {
			"packages": "software-properties-common, libzstd-dev"
		},
		"ghcr.io/devcontainers/features/docker-in-docker:2": {
			"dockerDashComposeVersion": "v2"
		}
	},
	"overrideFeatureInstallOrder": [
		"ghcr.io/devcontainers-contrib/features/apt-get-packages:2",
		"ghcr.io/devcontainers/features/docker-in-docker:2"
	],
	"appPort": "9042:9042",
	"remoteUser": "root",
	"onCreateCommand": "",
	"privileged": true,
	"customizations": {
		"vscode": {
			"extensions": [
				"1YiB.rust-bundle",
				"panicbit.cargo"
			]
		}
	},
}