Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

support for volumes #13

Closed
chpio opened this issue May 7, 2019 · 71 comments
Closed

support for volumes #13

chpio opened this issue May 7, 2019 · 71 comments
Labels
enhancement New feature or request fixed-pending-confirmation it's supposed to be fixed, waiting user to confirm

Comments

@chpio
Copy link

chpio commented May 7, 2019

  • shared volumes (defined in the root section of the docker-compose file)
    • and support for mounting these volumes in services
  • "just" volumes, a path to mount a value at that place (is this even needed with 1. in place?)

https://docs.docker.com/compose/compose-file/#volumes

@muayyad-alsadi
Copy link
Collaborator

"just" volumes is already supported, but shared volumes needs some work. I'll see what I can do.

@chpio
Copy link
Author

chpio commented May 7, 2019

"just" volumes is already supported

im getting an error for

services:
  web:
    # ...
    volumes:
      - "/var/www/storage"

Error: incorrect volume format "/var/www/storage", should be host-dir:ctr-dir[:option]

@muayyad-alsadi
Copy link
Collaborator

understood, so the podman commands would be

$ podman volume create myvol
$ v=`podman volume inspect myvol | jq -r '.[0].mountPoint'`
$ podman run -v $v:/var/www/storage  ...

@muayyad-alsadi muayyad-alsadi added the enhancement New feature or request label May 9, 2019
@muayyad-alsadi
Copy link
Collaborator

muayyad-alsadi commented May 13, 2019

the long syntax has 3 types: volume, bind or tmpfs

- type: volume
  source: mydata # this is a name
  target: /data
  read_only: false
  # type-specific options
  tmpfs: {size: 123, mode: 1777}
  volume: {nocopy: true} 
  bind: {propagation: Z, z, shared, slave, private, rshared, rslave, or rprivate(default)}

there are more than one shorthand syntax for compose

  # Just specify a path and let the Engine create a volume
  - /var/lib/mysql

  # Specify an absolute path mapping
  - /opt/data:/var/lib/mysql

  # Path on the host, relative to the Compose file
  - ./cache:/tmp/cache

  # User-relative path
  - ~/configs:/etc/configs/:ro

  # Named volume
  - datavolume:/var/lib/mysql

those can be passed to podman using: (which is what currently used)

  • --tmpfs /tmp:rw,size=787448k,mode=1777
  • -v HOST-DIR:CONTAINER-DIR:OPTIONS

where options can be

  • [rw|ro]
  • [z|Z]
  • [[r]shared|[r]slave|[r]private]

or unified --mount like this

  • --mount type=tmpfs,tmpfs-size=512M,destination=/path/in/container
  • --mount type=bind,source=/path/on/host,destination=/path/in/container

@muayyad-alsadi
Copy link
Collaborator

@chpio please test, this is now fully implemented

5606c69

and

fc61e01

@chpio
Copy link
Author

chpio commented Jun 11, 2019

thank you!

those commits seem to have broken something:

on commit fc61e01 (882cef7 seem to work fine)

$ podman-compose up
podman stop -t=1 foo_web_1
Error: no container with name or ID foo_web_1 found: no such container
125
podman rm foo_web_1
Error: no container with name or ID foo_web_1 found: no such container
1
podman pod rm foo
Error: no pod with name or ID foo found: no such pod
125
podman pod create --name=foo --share net -p 8080:80
c5c2a81eac58619f51540ea97394cee227d648c94664c1c3b87809a82e5ce5cf
0
Traceback (most recent call last):
  File "/home/thomas/bin/podman-compose", line 710, in <module>
    main()
  File "/home/thomas/bin/podman-compose", line 705, in main
    podman_path=args.podman_path
  File "/home/thomas/bin/podman-compose", line 665, in run_compose
    no_cleanup, dry_run, podman_path, shared_vols)
  File "/home/thomas/bin/podman-compose", line 555, in up
    args = container_to_args(cnt, dirname, podman_path, shared_vols)
  File "/home/thomas/bin/podman-compose", line 434, in container_to_args
    i, podman_path, cnt['_dirname'],
KeyError: '_dirname'

My dockerfile:

version: '3'
services:
  web:
    build: containers/web
    ports:
      - "8080:80"
    volumes:
      - ./dist:/var/www/html

@muayyad-alsadi
Copy link
Collaborator

I've just pushed a29cf06

please test

@chpio
Copy link
Author

chpio commented Jun 11, 2019

im getting this now

Traceback (most recent call last):
  File "/home/thomas/bin/podman-compose", line 740, in <module>
    main()
  File "/home/thomas/bin/podman-compose", line 735, in main
    podman_path=args.podman_path
  File "/home/thomas/bin/podman-compose", line 693, in run_compose
    no_cleanup, dry_run, podman_path, shared_vols)
  File "/home/thomas/bin/podman-compose", line 570, in up
    args = container_to_args(cnt, dirname, podman_path, shared_vols)
  File "/home/thomas/bin/podman-compose", line 436, in container_to_args
    shared_vols
  File "/home/thomas/bin/podman-compose", line 362, in mount_desc_to_args
    if is_str(mount_desc): mount_desc=parse_short_mount(mount_desc, basedir)
  File "/home/thomas/bin/podman-compose", line 78, in parse_short_mount
    for opt in mount_opt.split(','):
AttributeError: 'NoneType' object has no attribute 'split'

@muayyad-alsadi
Copy link
Collaborator

muayyad-alsadi commented Jun 11, 2019

I've handled the case, please try now

@chpio
Copy link
Author

chpio commented Jun 11, 2019

Traceback (most recent call last):
  File "/home/thomas/bin/podman-compose", line 741, in <module>
    main()
  File "/home/thomas/bin/podman-compose", line 736, in main
    podman_path=args.podman_path
  File "/home/thomas/bin/podman-compose", line 694, in run_compose
    no_cleanup, dry_run, podman_path, shared_vols)
  File "/home/thomas/bin/podman-compose", line 571, in up
    args = container_to_args(cnt, dirname, podman_path, shared_vols)
  File "/home/thomas/bin/podman-compose", line 437, in container_to_args
    shared_vols
  File "/home/thomas/bin/podman-compose", line 365, in mount_desc_to_args
    mount_type = mount_desc.get("type")
AttributeError: 'NoneType' object has no attribute 'get'

@muayyad-alsadi
Copy link
Collaborator

I've sorry for bothering you with me. I've pushed a fix. I'll create a compose file and test all the cases ASAP

@muayyad-alsadi
Copy link
Collaborator

I've fixed all of issues and added tests

as you can see here and here

@chpio
Copy link
Author

chpio commented Jun 11, 2019

thx, that dockerfile from above is working now, but i haven't tested volumes as a whole ("external" volumes syntax).

@muayyad-alsadi
Copy link
Collaborator

thank you.

I'll keep this ticket open until I test all cases.

@muayyad-alsadi muayyad-alsadi added the fixed-pending-confirmation it's supposed to be fixed, waiting user to confirm label Jul 3, 2019
@cognition9144
Copy link

cognition9144 commented Aug 14, 2019

Volumes currently works well. But the way podman-compose claims volumes might require some polish.

It allocates the volume and uses --mount <type, source, dest> to mount that path, instead of -v <volume_name>. But if we then execute podman prune, those volumes are considered as orphans and will be removed.

@muayyad-alsadi
Copy link
Collaborator

how do you suggesting passing them to podman run?

@cognition9144
Copy link

how do you suggesting passing them to podman run?

podman run -m data:/app/data

Although it's undocumented, it works the same way as Docker's named volumes.

@muayyad-alsadi
Copy link
Collaborator

starting from which podman version they have this option?

@muayyad-alsadi
Copy link
Collaborator

I've podman 1.2.0 and it shows

$ podman run --help
...
  -m, --memory string                            Memory limit (format: <number>[<unit>], where unit = b, k, m or g)
...

@cognition9144
Copy link

I've podman 1.2.0 and it shows

$ podman run --help
...
  -m, --memory string                            Memory limit (format: <number>[<unit>], where unit = b, k, m or g)
...

Sorry, my mistake. They don't have -m, but they have -v, --mount, and --volume.
sudo podman run --rm -v test-data:/data ubuntu is definitely valid.

➜  ~ sudo podman run --rm -v test-data:/data ubuntu   
➜  ~ sudo podman volume ls
DRIVER   VOLUME NAME
local    test-data

@muayyad-alsadi
Copy link
Collaborator

muayyad-alsadi commented Aug 14, 2019

I've reviewed the code and found that I'm already using --mount and we can easily pass it as-is if supported by podman. we just need to know when podman run --mount type=volume:source=vol-name:destination=/path/to/target is supported

       type=volume,source=my-volume,destination=/path/in/container,volume-label="color=red",volume-label="shape=round"

@cognition9144
Copy link

I've reviewed the code and found that I'm already using --mount and we can easily pass it as-is if supported by podman. we just need to know when podman run --mount type=volume:source=vol-name:destination=/path/to/target is supported

       type=volume,source=my-volume,destination=/path/in/container,volume-label="color=red",volume-label="shape=round"

I don't know when exactly it is supported. And according to the discussion in containers/podman#3811 they fogot to add documentation when they add that functionality.

I just know that podman 1.4.4 supports it.

@muayyad-alsadi
Copy link
Collaborator

we need to know how to pass :z for volume types.

#41 (comment)

@mpnordland
Copy link

mpnordland commented Oct 4, 2019

Since this issue seems to be tracking volumes support, I'll add this here.
At this line:

src = json.loads(out)[0]["mountPoint"]

In podman v1.6.0 the key is spelled "Mountpoint". This causes the following stacktrace:

Traceback (most recent call last):
  File "/home/micah/dev/boilerplate/venv/bin/podman-compose", line 10, in <module>
    sys.exit(main())
  File "/home/micah/dev/boilerplate/venv/lib/python3.7/site-packages/podman_compose.py", line 1093, in main
    podman_compose.run()
  File "/home/micah/dev/boilerplate/venv/lib/python3.7/site-packages/podman_compose.py", line 625, in run
    cmd(self, args)
  File "/home/micah/dev/boilerplate/venv/lib/python3.7/site-packages/podman_compose.py", line 782, in wrapped
    return func(*args, **kw)
  File "/home/micah/dev/boilerplate/venv/lib/python3.7/site-packages/podman_compose.py", line 898, in compose_up
    detached=args.detach, podman_command=podman_command)
  File "/home/micah/dev/boilerplate/venv/lib/python3.7/site-packages/podman_compose.py", line 457, in container_to_args
    mount_args = mount_desc_to_args(compose, volume, cnt['_service'], cnt['name'])
  File "/home/micah/dev/boilerplate/venv/lib/python3.7/site-packages/podman_compose.py", line 386, in mount_desc_to_args
    mount_desc = mount_dict_vol_to_bind(compose, fix_mount_dict(mount_desc, proj_name, srv_name))
  File "/home/micah/dev/boilerplate/venv/lib/python3.7/site-packages/podman_compose.py", line 366, in mount_dict_vol_to_bind
    src = json.loads(out)[0]["mountPoint"]
KeyError: 'mountPoint'

@dinokov
Copy link

dinokov commented Oct 27, 2019

I have the same problem.
Every time I put top-level volumes key in docker-compose.yml file I get the same error after running :

podman-compose up

...
podman volume inspect webdevops-1604_mysql || podman volume create webdevops-1604_mysql
Traceback (most recent call last):
  File "/home/damir/.local/bin/podman-compose", line 10, in <module>
    sys.exit(main())
  File "/home/damir/.local/lib/python3.7/site-packages/podman_compose.py", line 1093, in main
    podman_compose.run()
  File "/home/damir/.local/lib/python3.7/site-packages/podman_compose.py", line 625, in run
    cmd(self, args)
  File "/home/damir/.local/lib/python3.7/site-packages/podman_compose.py", line 782, in wrapped
    return func(*args, **kw)
  File "/home/damir/.local/lib/python3.7/site-packages/podman_compose.py", line 898, in compose_up
    detached=args.detach, podman_command=podman_command)
  File "/home/damir/.local/lib/python3.7/site-packages/podman_compose.py", line 457, in container_to_args
    mount_args = mount_desc_to_args(compose, volume, cnt['_service'], cnt['name'])
  File "/home/damir/.local/lib/python3.7/site-packages/podman_compose.py", line 386, in mount_desc_to_args
    mount_desc = mount_dict_vol_to_bind(compose, fix_mount_dict(mount_desc, proj_name, srv_name))
  File "/home/damir/.local/lib/python3.7/site-packages/podman_compose.py", line 366, in mount_dict_vol_to_bind
    src = json.loads(out)[0]["mountPoint"]
KeyError: 'mountPoint'

If I comment out last lines like this:

     #volumes:
       #phpmyadmin:
       #db_data: 

then the error is no more !
I am using podman version 1.6.1

@muayyad-alsadi
Copy link
Collaborator

but this is already handled here

https://github.com/containers/podman-compose/blob/devel/podman_compose.py#L384

    try:
        src = json.loads(out)[0]["mountPoint"]
    except KeyError:
        src = json.loads(out)[0]["Mountpoint"]

please use devel branch as in README.

@dinokov
Copy link

dinokov commented Oct 28, 2019

I installed devel branch and can use volumes now .

Thank you very much !

@muayyad-alsadi
Copy link
Collaborator

yes, please don't use 0.1.5, use the latest devel branch instead

pip3 install https://github.com/containers/podman-compose/archive/devel.tar.gz

@x80486
Copy link

x80486 commented Sep 20, 2020

Not trying to be pushy or anything like that, but wouldn't be better to make a release with all the changes you folks have made since 0.1.5? I think there is a lot of work already and releasing more frequently will give you more chances to make podman-compose better because more users are going to use/test it 🥳

@tinywrkb
Copy link

For my needs, the new volumes support working great. I tested named volumes, bind mounts (short syntax) with relative and absolute paths.
I suspect like myself, other users might find podman-compose barely usable without volumes support, so it would be nice to see a new release being tagged off master.

@Marek77
Copy link

Marek77 commented May 6, 2021

Hello, we have the same issue ("mountPoint" vs "Mountpoint") with pip3-installed podman-compose (not devel). Can we please get this fix merged and released ?

@thegarlynch
Copy link

thegarlynch commented May 31, 2021

Still no release ? if you don't want it, atleast just made tagged release for devel branch so some linux distro can pinpoint to it

@Jasstkn
Copy link

Jasstkn commented Oct 26, 2021

Hi. Same issue here :(

KeyError: 'mountPoint'

@djnotes
Copy link

djnotes commented Oct 26, 2021

There is not much point in using podman-compose anymore, while Podman supports Docker-Compose. You can use Docker Compose with Podman backend now.

@unixfox
Copy link

unixfox commented Oct 26, 2021

There is not much point in using podman-compose anymore, while Podman supports Docker-Compose. You can use Docker Compose with Podman backend now.

How so? Could you link to us to an article about that?

@djnotes
Copy link

djnotes commented Oct 26, 2021 via email

@Jasstkn
Copy link

Jasstkn commented Nov 3, 2021

There is not much point in using podman-compose anymore, while Podman supports Docker-Compose. You can use Docker Compose with Podman backend now.

I've tried to make it work on Mac but with no lack. :(

@ChristophP
Copy link

I've tried to make it work on Mac but with no lack. :(

Same here. Does anyone know if there is a way to expose the podman backend from inside the VM to the host MacOS system?

@stellarpower
Copy link

There is not much point in using podman-compose anymore, while Podman supports Docker-Compose. You can use Docker Compose with Podman backend now.

I didn't know we had full support. Ideally I should just move to K8s yaml as my last foray into Ansible wasn't a lot neater.

I am experienceing a bit of trouble with a sort-of hack I used in docker to get bind-mounted named volumes:

version: 3.3
services:
  foo:
    image: ..
    volumes: foo_Config:/config
volumes:
    foo_Config:
      driver: local
      driver_opts:
        device:    $VolumeRoot/foo/Config
        o:         bind
        type:      none

Not sure if it belongs here, or a separate issue (or if it could be in Podman itself - would need to work out the correct equivalent chain of CLI commands first). It took quite some time to find out how to do this in Docker, as everything was either named volume or bind-mount, not both as I wanted, to share between containers but back with ZFS.

Is this still worth following up on reporting this, or is development on podman-compose likely to draw to a close if docker-compose can be used instead? Struggling to find much time for GH outside of work these days, but I have a couple more things I've found in podman-compose, and I try to report a least, otherwise nothing can ever be fixed.

@muayyad-alsadi
Copy link
Collaborator

I'll close this, because latest version of podman-compose solved it.
please if you have an issue reported separately with full details.

we have an issue reported for macos, I don't have a mac, I'm using fedora. so I can't help with that.

@burredis
Copy link

burredis commented Jan 21, 2022

Hello everybody

I'm using macOS and this issue happened to me.

Return:

Error: statfs /foo/bar: no such file or directory
exit code: 125

Versions:

using podman version: 3.4.4
podman-composer version  1.0.3
podman --version 
podman version 3.4.4

Does someone have the solution?

Regards,

@muayyad-alsadi
Copy link
Collaborator

I was told that there is a bug in on MacOS when passing ./somthing/otherthing that does not happen when using somthing/otherthing

I don't have mac.

@alex198710
Copy link

Hello @burredis,
I have the exact same problem and the same podman versions.
Did you find any solution?
Thanks in advance,
Regards,

@burredis
Copy link

burredis commented Feb 16, 2022

Hey alex198710,
The solution that I found is to use colima. It works all that I need.
Can I start docker and all commands and features.

Regards

@alex198710
Copy link

Thanks @burredis,
I also found https://rancherdesktop.io/
I will check both.
Regards

@toanvo288
Copy link

toanvo288 commented Feb 16, 2022

Hi there, I using MacOS version Monterey. The volumes tag in compose file does not work at all.
Noted: I have used PostgreSQL image from docker.io
using podman version: 3.4.4
podman-composer version 1.0.3

volumes:
      - ./tests/Sql/InitialDbReleaseVersion.sql:/docker-entrypoint-initdb.d/InitialDbReleaseVersion.sql
Error: statfs /tests/Sql/InitialDbReleaseVersion.sql: no such file or directory
exit code: 125   

@muayyad-alsadi
Copy link
Collaborator

please try podman v4 on mac
there are reports regarding volumes on mac and v4 solved it.

@matheussouza9
Copy link

@toanvo288 I'm stuck on the same problem. I found this article that mounts a host dir to the qemu vm using ssh reverse proxy combined with sshfs, but, in the last step, I'm getting the following error:

[core@localhost ~]$ sshfs -p 10000 matheus@127.0.0.1:/Users /mnt/Users
connect_to 127.0.0.1 port 22: failed.
read: Connection reset by peer

@matheussouza9
Copy link

@muayyad-alsadi unfortunately Podman v4 is not available on brew yet. Homebrew/homebrew-core#95318 (comment)

@viinhpham
Copy link

viinhpham commented Mar 14, 2022

@toanvo288 I'm stuck on the same problem. I found this article that mounts a host dir to the qemu vm using ssh reverse proxy combined with sshfs, but, in the last step, I'm getting the following error:

[core@localhost ~]$ sshfs -p 10000 matheus@127.0.0.1:/Users /mnt/Users
connect_to 127.0.0.1 port 22: failed.
read: Connection reset by peer

@matheussouza9
I resolved this by:
sshfs your-username@your-current-local-ip:absolute-path-to-your-local-project-folder your-mounted-forder -o allow_other,uid=1000,gid=1000`

*Note: Don't use 127.0.0.1, You can use your host LAN ip address

@viinhpham
Copy link

Hi there, I using MacOS version Monterey. The volumes tag in compose file does not work at all. Noted: I have used PostgreSQL image from docker.io using podman version: 3.4.4 podman-composer version 1.0.3

volumes:
      - ./tests/Sql/InitialDbReleaseVersion.sql:/docker-entrypoint-initdb.d/InitialDbReleaseVersion.sql
Error: statfs /tests/Sql/InitialDbReleaseVersion.sql: no such file or directory
exit code: 125   

@toanvo288

  • You will need to mount your Mac folder in to podman machine folder by logging in podman machine and run the ssh reverse proxy command:

sshfs your-username@your-current-local-ip:absolute-path-to-your-local-project-folder your-podman-mounted-forder -o allow_other,uid=1000,gid=1000`

@toanvo288
Copy link

@viinhpham and @matheussouza9 : thanks all. I'll check and verify it again. Hope it works with Podman version 4 without a workaround.

@lmsurpre
Copy link

Hope it works with Podman version 4 without a workaround.

it doesn't

@drgsn
Copy link

drgsn commented Mar 2, 2023

not sure if this happens for windows too, but for mac podman machine starts in a detached state, so it won't be able to access anything from your storage. You need to attach the $HOME to it in order to be able to access volumes

  podman machine stop podman-machine-default
  podman machine rm podman-machine-default
  podman machine init -v $HOME:$HOME
  podman machine start

code stolen from https://stackoverflow.com/questions/69298356/how-to-mount-a-volume-from-a-local-machine-on-podman

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request fixed-pending-confirmation it's supposed to be fixed, waiting user to confirm
Projects
None yet
Development

Successfully merging a pull request may close this issue.