improve error message for graph driver mismatch

[asottile]

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind feature

Description

little itty-bitty feature -- a hint as to what command to run would be helpful in this error output:

$ podman run --rm -ti ubuntu:focal echo hi
ERRO[0000] User-selected graph driver "overlay" overwritten by graph driver "vfs" from database - delete libpod local files to resolve 
Error: vfs driver does not support overlay.mount_program options

Steps to reproduce the issue:

1. run any container using vfs (for example podman run --rm -ti ubuntu:focal echo hi
2. configure the storage engine to use overlay

   $ cat ~/.config/containers/storage.conf 
   [storage]
   driver = "overlay"
   [storage.options]
   mount_program = "/usr/bin/fuse-overlayfs"

3. run any container (for example podman run --rm -ti ubuntu:focal echo hi)

Describe the results you received:

$ podman run --rm -ti ubuntu:focal echo hi
ERRO[0000] User-selected graph driver "overlay" overwritten by graph driver "vfs" from database - delete libpod local files to resolve 
Error: vfs driver does not support overlay.mount_program options

Describe the results you expected:

A friendlier suggestion with the actions needed to solve the problem, for example:

$ podman run --rm -ti ubuntu:focal echo hi
ERRO[0000] User-selected graph driver "overlay" overwritten by graph driver "vfs" from database - delete libpod local files to resolve 
  hint: rm ~/.local/share/containers/storage/libpod/bolt_state.db
Error: vfs driver does not support overlay.mount_program options

Additional information you deem important (e.g. issue happens only occasionally):

Output of podman version:

$ podman version
Version:      2.0.4
API Version:  1
Go Version:   go1.14.2
Built:        Wed Dec 31 16:00:00 1969
OS/Arch:      linux/amd64

Output of podman info --debug:

$ podman info --debug
host:
  arch: amd64
  buildahVersion: 1.15.0
  cgroupVersion: v1
  conmon:
    package: 'conmon: /usr/libexec/podman/conmon'
    path: /usr/libexec/podman/conmon
    version: 'conmon version 2.0.20, commit: '
  cpus: 5
  distribution:
    distribution: ubuntu
    version: "20.04"
  eventLogger: file
  hostname: babibox
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 5.4.0-42-generic
  linkmode: dynamic
  memFree: 4134412288
  memTotal: 8348299264
  ociRuntime:
    name: runc
    package: 'containerd.io: /usr/bin/runc'
    path: /usr/bin/runc
    version: |-
      runc version 1.0.0-rc10
      commit: dc9208a3303feef5b3839f4323d9beb36df0a9dd
      spec: 1.0.1-dev
  os: linux
  remoteSocket:
    path: /run/user/1000/podman/podman.sock
  rootless: true
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: 'slirp4netns: /usr/bin/slirp4netns'
    version: |-
      slirp4netns version 1.1.4
      commit: unknown
      libslirp: 4.2.0
      SLIRP_CONFIG_VERSION_MAX: 2
  swapFree: 1964396544
  swapTotal: 1964396544
  uptime: 10h 57m 37.16s (Approximately 0.42 days)
registries:
  search:
  - docker.io
  - quay.io
store:
  configFile: /home/asottile/.config/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions:
    overlay.mount_program:
      Executable: /usr/bin/fuse-overlayfs
      Package: 'fuse-overlayfs: /usr/bin/fuse-overlayfs'
      Version: |-
        fusermount3 version: 3.9.0
        fuse-overlayfs: version 1.1.0
        FUSE library version 3.9.0
        using FUSE kernel interface version 7.31
  graphRoot: /home/asottile/.local/share/containers/storage
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  imageStore:
    number: 1
  runRoot: /run/user/1000/containers
  volumePath: /home/asottile/.local/share/containers/storage/volumes
version:
  APIVersion: 1
  Built: 0
  BuiltTime: Wed Dec 31 16:00:00 1969
  GitCommit: ""
  GoVersion: go1.14.2
  OsArch: linux/amd64
  Version: 2.0.4

Package info (e.g. output of rpm -q podman or apt list podman):

$ apt list podman
Listing... Done
podman/unknown,now 2.0.4~1 amd64 [installed]
podman/unknown 2.0.4~1 arm64
podman/unknown 2.0.4~1 armhf
podman/unknown 2.0.4~1 s390x

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide?

Yes

Additional environment details (AWS, VirtualBox, physical, etc.):

on both virtualbox and aws

[cyruseuros]

The error message is even worse when runroot is changed. It doesn't even tell you what's wrong, just that it failed to chown /run/user/<uid>/overlay. Deleting bolt_state.db fixes it, but you don't even get a hint that the database is overriding your config file.

[rhatdan]

@mheon PTAL

If either of you are interested in opening a PR to fix this, it would be appreciated.

I would not recommend removing the boltdb, but doing a podman system reset to remove all storage.

[asottile]

podman system reset fails with the same error so it isn't helpful here sadly

[rhatdan]

@mheon If we delete the database, will the database be regenerated, or will we have potentially lost data?

[rhatdan]

Podman system reset should work in this situation. I believe. Could you open a separate issue on this?

[mheon]

We will not re-generate the DB, so you will lose any currently-created containers. There is no safe way to migrate a container to a new storage path at present.

[asottile]

while making the other issue, I also noticed that podman system reset blows away ~/.config/containers/storage.conf -- 🤔

[asottile]

opened #7446 for the "unable to reset" issue 👍

[asottile]

also #7447 for the "reset deletes storage.conf"

[rhatdan]

Ok so we need to blow away all storage in this case.

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[rugk]

This issue is still not solved. The error message does still not give any information on how to actually solve the problem.

See also #8705

[rhatdan]

What would you like to see the error message say?

[rugk]

delete libpod local files to resolve

As an ordinary user I do not know at all what this is and what it means. So as the OP has suggested, tell me how to do that and where that file is.
For practical reasons, and in order to explain a) what the reason behind this error message is and b) what implications it has if I delete the libpod files (i.e. "all your containers will be lost!!!!"), I'd suggest to link to an FAQ entry or so.
(Also, better reopen this issue.)

[rhatdan]

The issue was a leftover storage.conf that was created in a previously (In my opinion broken version of Podman) This storage.conf file is no longer created and you are the first person who has reported an issue, that I know of, so this is not exactly a huge problem, and I don't see it as easy to fix.

[rugk]

But should not error messages anyway still always be as useful as possible? As it seems, I was not the first one to wonder about this message, the OP probably was.
Maybe rather than a fix, an explanation of the error/why it happens, may be an alternative.

[nicki-krizek]

I ran into this issue with v2.2.1, although I'm not sure why, since I didn't modify the default config.

The "delete libpod local files to resolve" isn't very helpful (only to find this issue...). podman system reset had the same error. It took me quite a while to figure out the files I needed to delete were located in ~/.local/share/containers/.

[pdfrod]

I also started receiving the error message ERRO[0000] User-selected graph driver "overlay" overwritten by graph driver "vfs" from database - delete libpod local files to resolve today, despite not having touched Podman configuration for months. Plus I've always used "overlay", so I really don't get why I'm getting error messages about "vfs" all of the sudden.

I'm not sure how to proceed here as I'd prefer not to lose my containers and data.

[mheon]

That error message indicates that you most likely have not been using overlay, even if you thought you were - the Podman DB indicates your existing containers were created with vfs. Did you upgrade Podman recently? I can recall some Podman 3.0 issues around storage options, and this could be related.

[rhatdan]

Also if you installed fuse-overlayfs, it would shift from a default of vfs to overlay.

[pdfrod]

Yes, I guess that's what happened. I thought I had overlay configured, but apparently I forgot to do it the last time I reinstalled my box. So the storage defaulted to 'vfs'. And now that I've upgraded to Podman 3.0.1, I guess the default changed to overlay.

[pdfrod]

I confirm that setting vfs in my configuration fixed the issue for me. But yeah, I felt a bit clueless with that error message. Something more along the lines of: Your storage driver is configured to "overlay" but your existing containers / data were created with "vfs"... would be more helpful.

[rhatdan]

PRs welcomed.