Pin button not working reliably.

[MicahZoltu]

Describe the bug
I am using IPFS Desktop which has v2.11.4 UI integrated. When I add files by IPFS hash, they show up in my files list. I then click the ... on the right and choose Pin. Sometimes the little pin icon shows up, sometimes it does not.

To Reproduce
Unknown. I can follow the same steps multiple times and sometimes I'm able to pin, sometimes I'm not. I generally use CIDv1 hashes if that matters, though CIDv0 are also used on occasion. This doesn't seem to be correlated though.

Expected behavior
When I click "Pin" the little pin icon shows up.

Screenshots

Desktop (please complete the following information):

• OS: Windows 10
• Browser: N/A
• Version: v2.11.4

Additional context
Add any other context about the problem here.

[jessicaschilling]

Thanks, @MicahZoltu -- we will keep an eye out for this as well, but in the meantime if you're able to find any definitely unpinnable files that we could reproduce, that would be extremely helpful.

[MicahZoltu]

You can try bafybeiet2646ne3zlmbnsg72wd23w7mqsyxkjadv4i75f3dqgjhku726jq (Warning: 115MB, multiple files) to see if that reproduces the issue. It is one that I currently cannot get pinned with the UI despite trying repeatedly. I even restarted my computer (and by extension, restarted IPFS) and it still won't pin for me. Sometimes, restarts make it so I can pin, but not for this one it seems.

Note: I haven't tried pinning via the CLI because I don't know how to access the CLI when using IPFS desktop, so I don't know if this is purely a UI issue or may be an IPFS core issue.

[olizilla]

If you are comfortable digging around, you can pop open the chromium dev tools in IPFS Desktop, by hitting
cmd+option+i
(on macos, other key combos here https://developers.google.com/web/tools/chrome-devtools/shortcuts)

There you may see some errors about not finding i18n language files for your current locale which you can ignore. If you try to pin this unpinnable CID, do you see any additional errors appear in the console?

Note: I haven't tried pinning via the CLI because I don't know how to access the CLI when using IPFS desktop, so I don't know if this is purely a UI issue or may be an IPFS core issue.

IPFS Desktop can add and ipfs command to your system path, if you open the app menu and select
Preferences > Command Line tools

[lidel]

I believe this is an issue with content discovery, and also has some implications to the way we track pinning progress (now vs near future).
Brain dump below, let me know if I misunderstood anything 🙏

How to reproduce

Was able to reproduce pinning problem with ephemeral go-ipfs 0.7.0 alone (ensures we test default config and empty repo):

$ docker run --rm -it --net=host ipfs/go-ipfs:v0.7.0

To switch CLI to use this ephemeral container, execute in other terminal:

$ IPFS_PATH=$(mktemp -d)
$ echo "/ip4/127.0.0.1/tcp/5001" > $IPFS_PATH/api

Then, in the same terminal, try to pin mentioned CID:

$ time ipfs refs -r bafybeiet2646ne3zlmbnsg72wd23w7mqsyxkjadv4i75f3dqgjhku726jq

In my case it hangs after some initial success (failing to find deeper blocks).
I stopped waiting after 30 minutes, it was stuck at Processed 406 nodes

Due to this, I think the problem is that

1. pinning some CIDs takes long time, and rare content may never succeed due to no providers
2. webui is unable to track "ongoing pinning" so it just shows item as "not pinned"

UI/UX for "ongoing pinning"

The WebUI-specific problem here is that our UI does not account for the fact that pinning operation may take long time, and may not succeed at all (be stuck in limbo for days).

When it comes to remote pinning added in #1615 / ipfs/ipfs-gui#91 things will be easy: remote pin object is created immediately and then webui can recheck/track its status to display proper state in UI (queued|pinning|pinned|failed).

When it comes to local pins, the ipfs pin ls shows only "finished" pins, there is no "ongoing" state, and ipfs pin add is a blocking operation, so if it does not succeed fast (eg. blocks not in local datastore), will introduce confusing UX.

Normally I'd suggest we add some "background pinning" magic to webui to track "ongoing" pinning in userland, but upcoming ipfs pin local commands planned by @aschmahmann will support "ongoing" state and asynchronous mode, which will remove the problem on tracking "ongoing pinning" present in the current ipfs pin add.

Due to this, I believe there is no point in investing too much time in fixing the old ipfs pin add API in WebUI.
We already want to remove /pins from UI.

Instead, plan for leveraging ipfs pin local add when it lands.

Hiding local pinning when data is not in local store?

Just an idea for a temporary fix until we have pin local add: hide "local pin" option from context menus when entire DAG is not already in datastore.

I believe the check would be to do refs in offline mode? (is there a better way?)
When the entire DAG is not in local datastore refs -r --offline returns an error:

$ ipfs refs -r --offline bafybeice66hroljqb5lmxcjwvahkxr2els7prhwaqrp6xrfcsgw2blpt6
Error: merkledag: not found

By hiding pinning for remote content we will avoid UX issues described by @MicahZoltu in #1684 (comment), and we will be able to focus on switching to ipfs pin local add

[aschmahmann]

👍 to @lidel's comments on the UI/UX issues with when pins take a while.

I believe the check would be to do refs in offline mode? (is there a better way?)

Not sure off hand. You could do ipfs dag stat --offline, but that's not too different than your suggestion.

---

As for this pin in particular. My suspicion is that for some reason not all of the content is actually available to the network (e.g. the only provider with all the data is behind a NAT, offline, etc.).

One way to verify this is the case would be to run on the command line ipfs dag export bafybeice66hroljqb5lmxcjwvahkxr2els7prhwaqrp6xrfcsgw2blpt6 > output.car on the node that you think has all the data. If that doesn't work then you may not actually have all the data, and if it does then it'd be helpful if you could post it here for us to look at.

Note: If you're on Windows use CMD instead of PowerShell when running the above command since PowerShell has trouble with binary outputs.

[jessicaschilling]

After almost 9 hours of letting ipfs dag export --offline bafybeice66hroljqb5lmxcjwvahkxr2els7prhwaqrp6xrfcsgw2blpt6 > output.car (offline, because everything was indicating I had all the data on my local) spin, I'm getting nowhere - so thinking @aschmahmann's suspicion of not all the content being available is correct.

[MicahZoltu]

IIUC, it sounds like the issue is that the UI currently doesn't have a way of differentiating between "not pinned" and "in the process of fetching so it can be pinned". This means that when I click that pin button, a process in the background starts but the pin icon won't show up until the pin is complete. In this specific example, the pin will never complete because the data isn't available so the pin icon will never show up.

That does seem to align with my symptoms of "sometimes the pin button doesn't work, sometimes it does". The times it does is when I already have the files locally and pinning is just a matter of flagging them as keep forever, and the times it doesn't work are when I have to fetch a bunch of data and it takes a while or may never complete.

[jessicaschilling]

@MicahZoltu ... and in the case of bafybeiet2646ne3zlmbnsg72wd23w7mqsyxkjadv4i75f3dqgjhku726jq, you just don't have all the content. I wasn't able to grab it either. That one's an issue with the original provider not ensuring someone is hosting the full thing.

[jessicaschilling]

Note: There are still some unanswered questions under "UI/UX for "ongoing pinning" above. However, parking this until go-ipfs 0.9; we'll know more about local pinning behavior and what's needed at that point.

[lidel]

This should be fixed in webui v2.12.x, where Pin operation got replaced with Set pinning modal.
Remaining work around tracking pin progress is tracked in #1752