Support native source folder for volume mount in remote model #8016

jeffwubj · 2020-10-14T14:27:23Z

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind feature

Description

Currently, looks like in podman's remote client case, bind volume will use source folders on podman server's host, could we support (or give some configurations to choose) using source folders on podman client's host?

podman client and podman server may run in different hosts, or one runs in the host, the other runs in a VM. Have the ability to mount folders on the client side might be able to open more use cases...

afbjorklund · 2020-10-14T16:25:00Z

I think this also happens, when using docker to connect to a remote server (rather than using the Docker Desktop product) ?
In Docker Toolbox (docker-machine), they worked around this issue by sharing the entire home folder (!) using vboxsf...

You should be able to create network volumes using podman volume, but I am a little uncertain about the syntax to use.
It takes a --driver parameter, which only talks about the default "local" driver that uses regular directories on the host.

github-actions · 2020-11-14T00:13:54Z

A friendly reminder that this issue had no activity for 30 days.

rhatdan · 2020-11-16T17:48:35Z

I am not sure how we would make this work, other then to export the current directory to the server running podman via nfs, samba or some other network protocol and then volume mount that directory into the container.

I am not sure is sshfs has this type ability, to share your homedir with a remote machine?

afbjorklund · 2020-11-16T18:19:48Z

I am not sure is sshfs has this type ability, to share your homedir with a remote machine?

The way we used sshfs in docker-machine, was to share the remote machine with a local mount.

https://docs.docker.com/machine/reference/mount/

That is, the mount was going in the other direction. Used sftp on the remote and fuse on the local.

Other than that, it was just VirtualBox shared folders...

nlfiedler · 2020-11-25T02:57:13Z

May I suggest that the issue is that podman is validating the mount points on the local side, rather than on the remote side? I'm running on macOS and deploying to a Linux host, on which the mount points exist. Maybe if podman client were to not validate the paths when connecting to a remote service, then it would just work?

mheon · 2020-11-25T04:14:49Z

That sounds like a separate issue? This one is about emulating a Docker behavior where folders from an OS X host are able to be mounted into containers on the Linux VM running containers. What you're describing is definitely separate (but definitely a bug, and I encourage you to file an issue about it).

…

On Tue, Nov 24, 2020 at 9:57 PM Nathan Fiedler ***@***.***> wrote: May I suggest that the issue is that podman is validating the mount points on the local side, rather than on the remote side? I'm running on macOS and deploying to a Linux host, on which the mount points exist. Maybe if podman client were to not validate the paths when connecting to a remote service, then it would just work? — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#8016 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AB3AOCCW773SGH72W3WGX23SRRXBVANCNFSM4SQV632Q> .

nlfiedler · 2020-11-25T05:05:19Z

Ah, you're right, I was seeing what I wanted to see. I've filed another issue, thanks.

github-actions · 2020-12-26T00:23:39Z

A friendly reminder that this issue had no activity for 30 days.

rhatdan · 2020-12-26T11:25:29Z

This issue is not stale, but no one is currently working on it.

github-actions · 2021-01-26T00:21:44Z

A friendly reminder that this issue had no activity for 30 days.

rhatdan · 2021-01-26T15:54:57Z

@baude This would be your stretch goal for Podman on a MAC.

github-actions · 2021-02-26T00:16:36Z

A friendly reminder that this issue had no activity for 30 days.

github-actions · 2021-03-29T00:07:17Z

A friendly reminder that this issue had no activity for 30 days.

rhatdan · 2021-03-30T18:04:02Z

Podman machine work is on going, but this is still a stretch goal. Would need someone from community to start working on it.

github-actions · 2021-04-30T00:03:49Z

A friendly reminder that this issue had no activity for 30 days.

afbjorklund · 2022-03-13T20:33:42Z

Does anyone have the same issue ?

I think that people who are using the WSL virtual machine (on Windows) are seeing the same thing too...

The files that are actually on the VM are fast, but the files that are accessed through /mnt (9p) are slow.

So it is "normal", for this Wizard of Oz setup.

ocramuni · 2022-03-14T15:21:55Z

Docker is using a different approach: they are using the Virtualization Framework and Shared directories - VirtIO FS. Why is it not applicable to podman on Mac ? I'm sure that it requires a huge rewrite/rethinking of how podman/qemu/etc are integrated, but in the long time maybe you (Podman developers) will have to spend less time to fit VM/podman with new Apple hardware or new version of OS and we'll have better performance on running containers

afbjorklund · 2022-03-14T15:25:50Z

The virtualization.framework is already supported by QEMU "hvf". Supporting virtiofs (in Rust) is planned for a future version of Podman, it was mentioned on the community meeting. EDIT: seems like it was only hypervisor.framework so far, even for ARM

Please note that Docker Desktop is not Open Source...

willcohen · 2022-03-14T15:32:12Z

While Virtualization.framework is higher level than what QEMU uses, the "hvf" being referenced by @afbjorklund it uses is https://developer.apple.com/documentation/hypervisor, which AFAIK is what the Virtualization framework uses underneath.

ocramuni · 2022-03-14T15:36:58Z

ok, thank you for the info. So we have to wait for the Rust virtiofs support to see I/O performance improvement.

afbjorklund · 2022-03-14T15:46:04Z

The first step would be to have any remote file system support... So virtfs will be an improvement to Podman.

fogfish · 2022-03-14T15:49:43Z

Small remark about the usage of time tar ... to bechmark I/O. The approach measures only wall clock but do not give strong signals about the overhead I/O vs CPU. I would recommend to use dedicate tools to measure I/O e.g. fio.

afbjorklund · 2022-03-14T17:58:59Z

@fogfish I think the example involved both, since it was tar and gzip

fogfish · 2022-03-14T20:33:07Z

The first step would be to have any remote file system support...

@afbjorklund what kind of remote fs your are looking in this context? I can repeat my measurements on it.

ocramuni · 2022-03-14T20:38:15Z

I have run the same tar benchmark starting podman machine VM (Fedora CoreOS) with native Apple Virtualization Framework:

Fedora CoreOS VM using native Apple Virtualization Framework, with /Users host directory mounted as virtiofs

[root@localhost ~]# mount -t virtiofs /Users /mnt/Users/
[root@localhost ~]# cd /var/mnt/Users/xxxxxx/
[root@localhost xxxxx]$ time tar xzf linux-5.17-rc7.tar.gz

real	0m30.699s
user	0m6.862s
sys	0m10.025s

9p mounted filesystem on qemu

[root@localhost ~]# cd /var/mnt/Users/xxxxxx/
[root@localhost xxxxxx]# time tar xzf /var/mnt/Users/xxxxxx/linux-5.17-rc7.tar.gz
real	5m47.921s
user	0m5.617s
sys	0m12.413s

I hope to get the same performance with the Rust virtiofs qemu support

afbjorklund · 2022-03-14T20:38:42Z

The ones we talked about are the native fs (xfs?), sshfs (fuse), virtfs (9p) and virtiofs.

If you are bored, you can also include SMB and NFS ?
But "virtfs" is the one, currently in podman...

For sshfs, we just used sshocker
For virtiofs, you are on your own :)

baude · 2022-03-15T01:47:25Z

can we please transfer this conversation to a github discussion? i'm not sure we are acting in the spirit of a github issue anymore?

afbjorklund · 2022-03-15T06:19:52Z

The discussion hasn't been about the issue (remote mount) in a long time, but I still prefer that over the previous situation...

A friendly reminder that this issue had no activity for 30 days.

Moving it to chat is fine.

fogfish · 2022-03-17T10:20:27Z

Let's use this discussion then to continue collaboration:
#13537

wnm3 · 2022-03-21T16:48:22Z

So installed v4.0.2 today on MacOS v12.3 and it seems mounting directories is still FUBAR claiming existing directories don't exist. Is it possible to run podman on MacOS and mount directories? If so what do we need to do. IBM has suggested we move to podman in lieu of docker desktop but this is a show stopper.

baude · 2022-03-21T19:59:43Z

@wnm3 yes it works... did you read the blog announcement for it? What did you try? You gotta provide more information and perhaps I can help.

fulldecent · 2022-03-21T20:48:20Z

This issue causes VS Code to not work with podman

mafischer · 2022-03-21T23:02:48Z

I can provide an example which occurs with VS Code using Remote-Containers:

podman run --sig-proxy=false -a STDOUT -a STDERR --mount type=bind,source=/Users/michael/Dev/micro-service-base-project,target=/workspaces/micro-service-base-project,consistency=cached --mount type=volume,src=vscode,dst=/vscode -l vsch.local.folder=/Users/michael/Dev/micro-service-base-project -l vsch.quality=stable --entrypoint /bin/sh vsc-micro-service-base-project-116577fed6f5e6281cc6aa55fccfc89a -c echo Container started
Error: statfs /Users/michael/Dev/micro-service-base-project: no such file or directory

afbjorklund · 2022-03-22T06:38:13Z

You need to mount it using a flag to podman machine init, like so: -v /Users:/Users.

Unlike Docker Desktop there are no default mounts (yet?), so everything is explicit.

srcshelton · 2022-03-22T08:46:05Z

@wnm3 yes it works... did you read the blog announcement for it? What did you try? You gotta provide more information and perhaps I can help.

This blog post: https://podman.io/blogs/2022/03/15/podman4.0.2brew.html ?

Is there further documentation with more detail available for macOS(/Windows?) mounts specifically?

From further replies above it sounds as if podman machine init must be run with --volume arguments for every possible mountpoint that any container may subsequently wish to use - is this correct?

typekpb · 2022-03-22T08:57:24Z

@srcshelton , correct, works for me with mount on machine as well, see: https://medium.com/@butkovic/favoring-podman-over-docker-desktop-33368e031ba0

baude · 2022-03-22T11:44:23Z

we intend to have a default volume mount in 4.1, but for now you have to mount it. Consider something like -v $HOME:$HOME ?

wnm3 · 2022-03-22T14:14:34Z

Thanks @typekpb the article you referenced is nicely organized and provided the missing step for me when initializing the machine to mount data that could later be referenced by the -v command when running a container.

wnm3 · 2022-03-22T14:44:07Z

Confirming that following the steps in the article https://medium.com/@butkovic/favoring-podman-over-docker-desktop-33368e031ba0 works perfectly.

mafischer · 2022-03-22T15:17:54Z

I am also able to get it working with some quirks. I am having issues with a few things.

VS code is creating a volume every time it opens a remote container. It works the first time, but when it tries subsequent times, it fails Error: volume with name vscode already exists: volume already exists

As a work around I am trying to delete the volume:

% podman volume rm vscode
Error: volume vscode is being used by the following container(s): b45c42acf5d7826564560c5df1a5d7098d5ec1bad9e12859bb3a0ee176e695c6: volume is being used
% podman container ls
CONTAINER ID  IMAGE       COMMAND     CREATED     STATUS      PORTS       NAMES

I am unable to delete the volume because it is being "used" by a container which doesn't exist according to the output of podman container ls.

I am able to delete the volume when I run this command:

% podman system prune --force --volumes                             
Deleted Containers
b45c42acf5d7826564560c5df1a5d7098d5ec1bad9e12859bb3a0ee176e695c6
Deleted Volumes
vscode

jsanta · 2022-03-23T14:45:37Z

Tried creating my podman machine with the -v parameter to mount a folder, specifically /opt
Worked on a first try, restarted my mac and stopped working. Had to create again my containers without any volumes.
I'm getting:

Mounting volume... /opt:/var/mnt/opt
2022/03/23 11:21:34 tcpproxy: for incoming conn [::1]:49929, error dialing "192.168.127.2:22": connect tcp 192.168.127.2:22: no route
2022/03/23 11:21:34 tcpproxy: for incoming conn [::1]:49928, error dialing "192.168.127.2:22": connect tcp 192.168.127.2:22: no route
Error: exit status 255

Then to check:

› podman machine ssh                         
Connecting to vm podman-machine-default. To close connection, use `~.` or `exit`
Warning: Permanently added '[localhost]:63883' (ECDSA) to the list of known hosts.
Fedora CoreOS 35.20220305.dev.0
Tracker: https://github.com/coreos/fedora-coreos-tracker
Discuss: https://discussion.fedoraproject.org/tag/coreos

[root@localhost ~]# ls /mnt/opt/
[root@localhost ~]#

baude · 2022-03-23T14:49:11Z

@jsanta can you please file a new issue for this one? can you also walk me through all the commands issued? I appreciate it.

baude · 2022-03-23T14:55:36Z

there is a github discussion on the matter at -> #13537

please file new issues for bugs regarding volume mounts.

openshift-ci-robot added the kind/feature Categorizes issue or PR as related to a new feature. label Oct 14, 2020

Luap99 mentioned this issue Nov 11, 2020

Error: statfs /tmp/secret: no such file or directory #8302

Closed

github-actions bot added the stale-issue label Nov 14, 2020

rhatdan removed the stale-issue label Nov 23, 2020

github-actions bot added the stale-issue label Dec 26, 2020

rhatdan removed the stale-issue label Dec 26, 2020

github-actions bot added the stale-issue label Jan 26, 2021

rhatdan removed the stale-issue label Jan 26, 2021

rhatdan assigned baude Jan 26, 2021

github-actions bot added the stale-issue label Feb 26, 2021

rhatdan removed the stale-issue label Feb 26, 2021

github-actions bot added the stale-issue label Mar 29, 2021

rhatdan removed the stale-issue label Mar 30, 2021

github-actions bot added the stale-issue label Apr 30, 2021

rhatdan removed the stale-issue label May 3, 2021

baude closed this as completed Mar 21, 2022

containers locked as resolved and limited conversation to collaborators Mar 23, 2022

Support native source folder for volume mount in remote model #8016

Support native source folder for volume mount in remote model #8016

Comments

jeffwubj commented Oct 14, 2020

afbjorklund commented Oct 14, 2020

github-actions bot commented Nov 14, 2020

rhatdan commented Nov 16, 2020

afbjorklund commented Nov 16, 2020

nlfiedler commented Nov 25, 2020

mheon commented Nov 25, 2020 via email

nlfiedler commented Nov 25, 2020

github-actions bot commented Dec 26, 2020

rhatdan commented Dec 26, 2020

github-actions bot commented Jan 26, 2021

rhatdan commented Jan 26, 2021

github-actions bot commented Feb 26, 2021

github-actions bot commented Mar 29, 2021

rhatdan commented Mar 30, 2021

github-actions bot commented Apr 30, 2021

afbjorklund commented Mar 13, 2022 • edited Loading

ocramuni commented Mar 14, 2022

afbjorklund commented Mar 14, 2022 • edited Loading

willcohen commented Mar 14, 2022 • edited Loading

ocramuni commented Mar 14, 2022

afbjorklund commented Mar 14, 2022 • edited Loading

fogfish commented Mar 14, 2022

afbjorklund commented Mar 14, 2022

fogfish commented Mar 14, 2022

ocramuni commented Mar 14, 2022

afbjorklund commented Mar 14, 2022 • edited Loading

baude commented Mar 15, 2022

afbjorklund commented Mar 15, 2022

fogfish commented Mar 17, 2022

wnm3 commented Mar 21, 2022

baude commented Mar 21, 2022

fulldecent commented Mar 21, 2022

mafischer commented Mar 21, 2022

afbjorklund commented Mar 22, 2022 • edited Loading

srcshelton commented Mar 22, 2022

typekpb commented Mar 22, 2022

baude commented Mar 22, 2022

wnm3 commented Mar 22, 2022

wnm3 commented Mar 22, 2022

mafischer commented Mar 22, 2022 • edited Loading

jsanta commented Mar 23, 2022

baude commented Mar 23, 2022

baude commented Mar 23, 2022

afbjorklund commented Mar 13, 2022 •

edited

Loading

afbjorklund commented Mar 14, 2022 •

edited

Loading

willcohen commented Mar 14, 2022 •

edited

Loading

afbjorklund commented Mar 14, 2022 •

edited

Loading

afbjorklund commented Mar 14, 2022 •

edited

Loading

afbjorklund commented Mar 22, 2022 •

edited

Loading

mafischer commented Mar 22, 2022 •

edited

Loading