With remote service, podman run --mount complains that path does not exist on client.

[nlfiedler]

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

With a remote connection defined, invoke podman run with the --mount option and it complains that the source path does not exist. True, but I am using a connection to remote host, so I would expect it to understand that the path in question is on the remote host, not the client.

Steps to reproduce the issue:

1. podman system connection add --identity ~/.ssh/id_rsa microvm nfiedler@192.168.1.128
2. podman run --name=minio -d -e MINIO_ACCESS_KEY=minio -e MINIO_SECRET_KEY='keyboard cat' --publish '9000:9000' --mount type=bind,source=/mnt/tank/packs,destination=/data minio/minio --compat server /data

Describe the results you received:

Error: error checking path "/mnt/tank/packs": stat /mnt/tank/packs: no such file or directory

Describe the results you expected:

No error reported, and the container would be running on the remote host.

Additional information you deem important (e.g. issue happens only occasionally):

Happens every time.

Output of podman version:

Client:
Version:      2.1.1
API Version:  2.0.0
Go Version:   go1.15.2
Built:        Fri Sep 25 18:07:22 2020
OS/Arch:      darwin/amd64

Server:
Version:      2.1.1
API Version:  2.0.0
Go Version:   go1.13.15
Built:        Sun Oct 25 17:00:00 2020
OS/Arch:      linux/amd64

Output of podman info --debug:

host:
  arch: amd64
  buildahVersion: 1.16.1
  cgroupManager: cgroupfs
  cgroupVersion: v1
  conmon:
    package: conmon-2.0.21-1.1.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.0.21, commit: unknown'
  cpus: 4
  distribution:
    distribution: '"opensuse-microos"'
    version: "20201121"
  eventLogger: journald
  hostname: yubaba
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 100
      size: 1
    - container_id: 1
      host_id: 165536
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1001
      size: 1
    - container_id: 1
      host_id: 165536
      size: 65536
  kernel: 5.9.8-2-default
  linkmode: dynamic
  memFree: 2565263360
  memTotal: 4113084416
  ociRuntime:
    name: runc
    package: runc-1.0.0~rc92-1.1.x86_64
    path: /usr/bin/runc
    version: |-
      runc version 1.0.0-rc92
      spec: 1.0.2-dev
  os: linux
  rootless: true
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.0.0-1.3.x86_64
    version: |-
      slirp4netns version 1.0.0
      commit: unknown
      libslirp: 4.3.1
  swapFree: 0
  swapTotal: 0
  uptime: 4h 37m 46.46s (Approximately 0.17 days)
registries:
  search:
  - registry.opensuse.org
  - docker.io
store:
  configFile: /home/nfiedler/.config/containers/storage.conf
  containerStore:
    number: 1
    paused: 0
    running: 1
    stopped: 0
  graphDriverName: overlay
  graphOptions:
    overlay.mount_program:
      Executable: /usr/bin/fuse-overlayfs
      Package: fuse-overlayfs-1.1.2-31.6.x86_64
      Version: |-
        fuse-overlayfs: version 1.1.0
        FUSE library version 3.10.0
        using FUSE kernel interface version 7.31
  graphRoot: /home/nfiedler/.local/share/containers/storage
  graphStatus:
    Backing Filesystem: btrfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  imageStore:
    number: 2
  runRoot: /run/user/1001/containers
  volumePath: /home/nfiedler/.local/share/containers/storage/volumes
version:
  APIVersion: 2.0.0
  Built: 1603670400
  BuiltTime: Mon Oct 26 00:00:00 2020
  GitCommit: ""
  GoVersion: go1.13.15
  OsArch: linux/amd64
  Version: 2.1.1

Package info (e.g. output of rpm -q podman or apt list podman):

$ brew info podman
podman: stable 2.1.1 (bottled)
Tool for managing OCI containers and pods
https://podman.io/
/usr/local/Cellar/podman/2.1.1 (143 files, 37.1MB) *
  Poured from bottle on 2020-11-24 at 16:53:56
From: https://github.com/Homebrew/homebrew-core/blob/HEAD/Formula/podman.rb
License: Apache-2.0
==> Dependencies
Build: go ✘, go-md2man ✘
==> Caveats
Bash completion has been installed to:
  /usr/local/etc/bash_completion.d

zsh completions have been installed to:
  /usr/local/share/zsh/site-functions
==> Analytics
install: 1,667 (30 days), 7,943 (90 days), 7,998 (365 days)
install-on-request: 1,641 (30 days), 7,552 (90 days), 7,607 (365 days)
build-error: 0 (30 days)

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide?

Recent versions: No. Looked at the changelog and this issue is not included among the 2.2.x changes.
Troubleshooting: Yes, nothing there about this.

Additional environment details (AWS, VirtualBox, physical, etc.):

MacBook Pro 15 inch 2018, podman service running in MicroOS from 3 days ago (no lsb_release or /etc/release?) in VirtualBox 6.1.16. I can run the same command from the shell in MicroOS and it succeeds, as expected.

[Luap99]

Does it also happens if you use --volume instead?

[rhatdan]

Podman should not be checking for the existence of the source directory on podman remote in either the --volume or the --mount command.

Care to open a PR to fix this issue?