Django container complains "error walking file system" when venv is installed

[jmoppel]

What happened?

After creating a fresh project, and then docker-compose build and up, The Django Docker container infinitely complains "error walking file system".

What should've happened instead?

The Django container should start without any errors.

Additional details

This happens if you have a Python venv environment installed anywhere inside the project directory. The problem is, inside the docker container, venv symlinks to files that don't exist in the container (e.g. /usr/bin/python3.8). I have a venv to test changes to requirements locally instead of waiting for a docker-compose build to fail (which takes a lot longer).

Steps to reproduce

Inside a new or existing cookiecutter project directory

docker-compose -f local.yml build
docker-compose -f local.yml up

Stop the containers (ctrl-c). Now, create a new python venv

python -m venv venv

Now bring up your containers and the Django Container will constantly complain "error walking file system":

docker-compose -f local.yml up

I assume this is watchgod complaining. Let me know if this is correct or not.

If you're open to a fix for this issue, I'll open a pull request to update the Dockerfile and fix this problem.

[luzfcb]

@jmoppel I was unable to reproduce the issue.
What operating system are you using?

If you want, could you please record the steps you made in the terminal using https://asciinema.org/ and after sending the recording link? (Be careful to avoid showing things that could compromise security, such as passwords, tokens, etc.)

Install asciinema

pip3 install asciinema

Create an account to able to manage your asciinema records

asciinema auth

Start record the steps

asciinema rec -t "cookiecutter-django issue 3083"

Note:
if you just want to learn how to use asciinema but without uploading the recording, I recommend forcing the recording to a local file by entering a file-name for the recording

asciinema rec -t "cookiecutter-django issue 3083" myrecord.cast

to watch the recording, just run

asciinema play myrecord.cast

to upload to your account on https://asciinema.org/

asciinema upload myrecord.cast

[jmoppel]

https://asciinema.org/a/L0Rzbuo6EgWi5EZpqgiOL6Vf7

One minor correction to the video. Before creating the video, I did a docker-compose build and then a docker-compose up. In the video I stated it was the other way around, which is not correct.

For anyone who might find this issue in the future, the exact error message in the Docker console is:
django | error walking file system: FileNotFoundError [Errno 2] No such file or directory: '/app/venv/bin/python3.8'

I have created a Docker file that corrects this problem on my machine.

[bryanus1]

I have the same error when install a venv, docker show this log: error walking file system: FileNotFoundError [Errno 2] No such file or directory: '/app/venv/bin/python'

[jmoppel]

@bryanus1 The fix I implemented to was to include the following lines in my local Django Dockerfile just after all the apt-get stuff.

#  Prevent 'error walking file system' when Python venv exists on host
RUN ln /usr/local/bin/python3.8 /usr/bin/python
RUN ln /usr/local/bin/python3.8 /usr/bin/python3
RUN ln /usr/local/bin/python3.8 /usr/bin/python3.8

There's probably a cleaner way to implement than what I've done, so it doesn't break for different versions of python. However, this is what we're using at the moment. I'm not certain if @luzfcb would like a pull request for this, since he hasn't been able to replicate the issue.

[bryanus1]

Add lines just after apt-get and error persists.

[luzfcb]

@bryanus1 try to add venv to the .dockerignore file.
anyway, I still can't reproduce this error

[NyasakiAT]

@bryanus1 try to add venv to the .dockerignore file.
anyway, I still can't reproduce this error

Well I'm having the same issue.
All the previously mentioned solutions didn't worked, adding the venv to the '.dockerignore' didn't resolved it either

[bryanus1]

I realized that i can work without problem but .dockerignore doesn't works well

[munjata]

when removed the host virtual env (.venv) I created, its worked.

[luzfcb]

I added venv to the .dockerignore 18a9404

[jmoppel]

Add lines just after apt-get and error persists.

@bryanus1 Did you rebuild your containers (which is required for the symlinks to be created in your container)? Run the following before trying to bring up your containers.

docker-compose -f local.yml build --no-cache

[bryanus1]

Add lines just after apt-get and error persists.

@bryanus1 Did you rebuild your containers (which is required for the symlinks to be created in your container)? Run the following before trying to bring up your containers.

docker-compose -f local.yml build --no-cache

Rebuild images with this command but no works!!!

[moige01]

Same here. Adding venv to .dockerignore does not work... Nor rebuilding without cache. Also, renaming the folder to whatever else reproduce the same error.

[jmoppel]

@bryanus1 Make a note of where the symlink venv/bin/python points to. Shell into your container and see if that path exists in your container. If it doesn't, that's your problem. As I described in my initial post, the crux of the issue I had is that the symlink to the python executable in venv/bin/is broken because the path it's pointing to exists on your host machine, but not in the container.

[bryanus1]

@jmoppel venv/bin/python has symlink to python in my local machine and not into container, that's is problem. How I resolve it?

[ruslaniv]

I had the same problem, here is how I solved it.

In a docker container there is no need to create a virtual environment, since docker is akin to a virtual environment, so in a container we can pip install all packages globally.
So when we spin up a container with a docker compose up, mapping an app folder from the host to a folder in a container, it is important to omit mapping the venv folder from the host, otherwise python gets confused, since now there is a symlink for app/venv/bin/python inside a container pointing to a python interpreter location from the host operating system.
So a way to omit mapping venv from the host to a container is add this into docker-compose:

 volumes:
      - .:/app
      - app/venv/ # This line hides mapping of venv folder in the container

[bryanus1]

I had the same problem, here is how I solved it.

In a docker container there is no need to create a virtual environment, since docker is akin to a virtual environment, so in a container we can pip install all packages globally.
So when we spin up a container with a docker compose up, mapping an app folder from the host to a folder in a container, it is important to omit mapping the venv folder from the host, otherwise python gets confused, since now there is a symlink for app/venv/bin/python inside a container pointing to a python interpreter location from the host operating system.
So a way to omit mapping venv from the host to a container is add this into docker-compose:

 volumes:
      - .:/app
      - app/venv/ # This line hides mapping of venv folder in the container

I test your solution and works without problems, only make a change that volume must be a absolutly path, change
volumes: - .:/app - /app/venv/ add a / before app

[jmoppel]

It looks like this is a known issue with Watchgod that has an open pull request to fix the issue.

samuelcolvin/watchfiles#80

[oesah]

In my case, @bryanus1 's solution worked:

    ...
    volumes:
      - .:/app:z
      - /app/.venv