Docker issue: /ipfs and /ipns are empty within container despite node having pinned files

[pirate]

I'm not sure if there's a bug with the Docker setup in the README, or if there are missing steps that should be documented.

Despite having a test file uploaded and pinned in my IPFS web dashboard, and the daemon set to mount them automatically, the folders in /ipns (and /ipfs) are empty within Docker and do not contain my test data.

If I try to mount them as volumes on the host, not only are they empty but they don't even contain the empty 12D3K... and local dirs that were visible from within the container.

I have the following docker-compose.yml setup as instructed by the README and this document: https://github.com/ipfs/go-ipfs/blob/master/docs/fuse.md#mounting-ipfs:

version: '2.4'

services:
    ipfs:
        image: ipfs/go-ipfs:latest
        command: daemon --migrate=true --mount
        #privileged: true    # not needed in theory, adding this makes no difference though
        cap_add:
            - SYS_ADMIN
        security_opt:
            - apparmor:unconfined
        devices:
            - /dev/fuse:/dev/fuse
        ports:
            - 8080:8080   # public gateway
            - 4001:4001   # public swarm UDP
            - 5001:5001   # private API
        volumes:
            - /etc/fuse.conf:/etc/fuse.conf:ro
            - ./data/ipfs:/data/ipfs
            - ./data/export:/export
            #- ./data/shares/ipfs:/ipfs  # eventually want to do this, but baby steps first
            #- ./data/shares/ipns:/ipns

/etc/fuse.conf:

user_allow_other

/data/ipfs/config:

...
  "Mounts": {
    "FuseAllowOther": true,
    "IPFS": "/ipfs",
    "IPNS": "/ipns"
  },

When I start the container with docker-compose up it reports that /ipfs and /ipns have been successfully mounted, but they remain empty within the container despite having pinned files.

$ docker-compose up
Creating network "zerviceipfs_default" with the default driver
Creating zerviceipfs_ipfs_1 ... done
Attaching to zerviceipfs_ipfs_1
ipfs_1  | Changing user to ipfs
ipfs_1  | ipfs version 0.8.0-rc2
ipfs_1  | Found IPFS fs-repo at /data/ipfs
ipfs_1  | Initializing daemon...
ipfs_1  | go-ipfs version: 0.8.0-rc2-4080333
ipfs_1  | Repo version: 11
ipfs_1  | System version: amd64/linux
ipfs_1  | Golang version: go1.14.4
ipfs_1  | 2021/02/17 19:50:48 failed to sufficiently increase receive buffer size (was: 208 kiB, wanted: 2048 kiB, got: 416 kiB). See https://github.com/lucas-clemente/quic-go/wiki/UDP-Receive-Buffer-Size for details.
ipfs_1  | Swarm listening on /ip4/127.0.0.1/tcp/4001
...
ipfs_1  | Swarm announcing /ip4/76.71.247.47/tcp/4001
ipfs_1  | API server listening on /ip4/0.0.0.0/tcp/5001
ipfs_1  | WebUI: http://0.0.0.0:5001/webui
ipfs_1  | IPFS mounted at: /ipfs
ipfs_1  | IPNS mounted at: /ipns
ipfs_1  | Gateway (readonly) server listening on /ip4/0.0.0.0/tcp/8080
ipfs_1  | Daemon is ready

$ docker-compose exec --user=root ipfs /bin/sh
 / # whoami
root
/ # ls /ip*s/**
/ipns/12D3KooWCSPrazUW7QNcpvWJh3jaWzErz52SrKC9X6uXqB4XVp4c:

/ipns/local:
/ #

I'm trying to eventually mount /ipfs and /ipns as volumes so that the host OS can write files into them, but I cant even get them working within the container as regular non-volume folders first. The eventually goal is to be able to put a folder into ~/myfiles on the host, and have it automatically uploaded by ipfs within the docker container without having to mess around with the ipfs files cli to add files manually.

Not sure if this ticket means that isn't possible yet: ipfs/roadmap#90 If it's definitely not possible then that should probably also be documented in the README so people don't waste time trying to get mounts working from within Docker. However this issue is specific to the fact that /ipfs and /ipns are empty when they should contain data.

Semi-related issues:

• FUSE mount Mutable File System (MFS) roadmap#90
• Mount MFS ipfs-desktop#618
• Write access using fuse or docker-volume-ipfs #6542
• https://discuss.ipfs.io/t/mounting-ipfs-or-ipns-as-read-write-directory/5970/3
• add an experimental version of mount to go-ipfs ipfs-inactive/package-managers#74 (comment)
• [EPIC] Towards better file system APIs ipfs-inactive/package-managers#71
• https://github.com/jfmherokiller/ipfs-mfs-fuse
• https://github.com/piedar/js-ipfs-mount

[welcome]

Thank you for submitting your first issue to this repository! A maintainer will be here shortly to triage and review.
In the meantime, please double-check that you have provided all the necessary information to make this process easy! Any information that can help save additional round trips is useful! We currently aim to give initial feedback within two business days. If this does not happen, feel free to leave a comment.
Please keep an eye on how this issue will be labeled, as labels give an overview of priorities, assignments and additional actions requested by the maintainers:

• "Priority" labels will show how urgent this is for the team.
• "Status" labels will show if this is ready to be worked on, blocked, or in progress.
• "Need" labels will indicate if additional input or analysis is required.

Finally, remember to use https://discuss.ipfs.io if you just need general support.

[fheslouin]

Same issue for me. I'm trying to understand if there is any solution for that feature

[fheslouin]

I could not find any solution, any improvements on this ?

[joe-rice]

For me, this was related to #9044. dag-pb files wouldn't show through fuse, but raw files (those added with add --raw-leaves) showed up- even from outside the container in a configuration similar to your own.

Raw pin base32 cids begin with "bafkrei" and dag-pb base32 cids begin with "bafybei". You can list your raw pins with ipfs pin ls -q | ipfs cid base32 | grep \^bafkrei.

[sachasmart-weavik]

Any updates on this? Running into the same issue here

[Erudition]

#7738 tells us that these folders are supposed to be empty, no?

[icf20]

u may want o look into this https://youtu.be/PHujBhq4J9A

[pirate]

Some new ways to do this from here: https://news.ycombinator.com/item?id=39213960

• As a filesystem via NFS NFS host and guest support djdv/go-filesystem-utils#40
• https://github.com/peergos/peergos

[djdv]

As a filesystem via NFS djdv/go-filesystem-utils#40

To clarify something, that project supports mounting IPFS and IPNS via FUSE and WinFSP already. With NFS and 9P implemented, but still pending some testing.
There's 2 pseudo systems, "PinFS" and "KeyFS" as well, which populate the root with your node's (recursive) pins, and (ipfs key) keys respectively. Support for the IPFS Files API is also pending more testing.

It all works, but I don't want to encourage people to use it (hence the rather bare repo).