Only partially able to read file through ipns mount #5328

kvm2116 · 2018-08-01T16:53:00Z

Version information:

go-ipfs version: 0.4.18-dev-4bca53e
Repo version: 7
System version: amd64/linux
Golang version: go1.10

Type:

Bug

Description:

I created a text file (size 300MB) with random ascii characters using the following command:
base64 /dev/urandom | head -c 300000000 > file.txt
Added this text file to /ipns mount on the machine

On another machine, I tried reading a third of the file using head:
head -c 1000000 /ipns/QmezBPmm4RRBRTBYPhVHsSSAPS2Q8hRTDd3PrfTsG8yvnf/file.txt

This command prints some number of lines and then gets stuck permanently.
Using debugging log, the following line is repeated "10:42:23.157 DEBUG bitswap: 9 keys in bitswap wantlist workers.go:185"

kvm2116 · 2018-08-23T15:37:31Z

@Stebalien Any thoughts?
I am trying to debug, but without much luck so far.

Stebalien · 2018-08-23T17:40:01Z

When that happens, could you check ipfs swarm peers --streams to see if the machines are still connected and if they have open bitswap streams?

This could also be #5183.

kvm2116 · 2018-08-23T18:00:31Z

This is what the output of "ipfs swarm peers --streams" looks like when it is stuck

mkunal@node-1:~$ ipfs swarm peers --streams
/ip4/128.110.153.130/tcp/4001/ipfs/QmYgbFxWGykyTMNrotndQmoV67RpixAsus6zkRzk5bB66Q
/ipfs/bitswap/1.1.0
/ipfs/kad/1.0.0
/ipfs/kad/1.0.0

Stebalien · 2018-08-23T18:20:05Z

Damn. Assuming peer A is downloading and peer B is serving, can you run ipfs bitswap wantlist -p $PEER_A on peer B and ipfs bitswap wantlist on peer A? This is looking more like #5183.

kvm2116 · 2018-08-23T18:24:47Z

On peer B (serving):
$ipfs bitswap wantlist -p QmRHCnAmHGjZACNND1v5AJw6ndCPyrN5nT7k5umSsZ69D4
QmXFRdKuZiCcnRiTNyEcKtwL127WDjYrnu4C5pScykR5hg
QmXC8zjajA5q3RRJa9oXJxkVBN5ibihjnJvPZRayMERCjC
QmSw6qxguQ3ZSk8ufA416pVvVHwSotKppQg54VUHEoGrrv

On peer A (downloading):
$ ipfs bitswap wantlist
QmXFRdKuZiCcnRiTNyEcKtwL127WDjYrnu4C5pScykR5hg
QmXC8zjajA5q3RRJa9oXJxkVBN5ibihjnJvPZRayMERCjC
QmSw6qxguQ3ZSk8ufA416pVvVHwSotKppQg54VUHEoGrrv

On peer B (serving), I verified the blocks exist by printing the raw data using "ipfs object get QmSw6qxguQ3ZSk8ufA416pVvVHwSotKppQg54VUHEoGrrv"

Stebalien · 2018-08-23T18:36:16Z

Interesting...

Does peer B have those blocks? Does it hang when running ipfs block stat QmXFRdKuZiCcnRiTNyEcKtwL127WDjYrnu4C5pScykR5hg?

kvm2116 · 2018-08-23T18:39:02Z

No hanging on peer B.

$ipfs block stat QmXFRdKuZiCcnRiTNyEcKtwL127WDjYrnu4C5pScykR5hg
Key: QmXFRdKuZiCcnRiTNyEcKtwL127WDjYrnu4C5pScykR5hg
Size: 262158

$ ipfs block stat QmSw6qxguQ3ZSk8ufA416pVvVHwSotKppQg54VUHEoGrrv
Key: QmSw6qxguQ3ZSk8ufA416pVvVHwSotKppQg54VUHEoGrrv
Size: 262158

$ ipfs block stat QmXC8zjajA5q3RRJa9oXJxkVBN5ibihjnJvPZRayMERCjC
Key: QmXC8zjajA5q3RRJa9oXJxkVBN5ibihjnJvPZRayMERCjC
Size: 262158

On peer A (downloading), it hangs for the above 3 commands.

Stebalien · 2018-08-23T18:47:38Z

Awesome! Well, not for you but this definitely looks like #5183. Could you follow the instructions here and upload the result to github (or ipfs)? That'll help me figure out where IPFS is stuck.

kvm2116 · 2018-08-23T19:00:29Z

Here are the results:

EDIT: added the binary
issue_dump.tar.gz

Stebalien · 2018-08-23T19:08:32Z

Can I get the exact binary you used as well?

kvm2116 · 2018-08-23T19:10:06Z

My apologies. Added the binary in the tar.

issue_dump.tar.gz

Stebalien · 2018-08-23T19:11:31Z

Which peer is this and would you mind doing the other peer as well?

kvm2116 · 2018-08-23T19:17:08Z

Edit: the two files were swapped

Sender (peer B):
sender_issue_dump.tar.gz

Downloader (peer A):

downloader_issue_dump.tar.gz

Stebalien · 2018-08-23T19:19:36Z

Awesome! Thanks!

One more question. Is this the first time you've started these nodes (trying to reproduce).

kvm2116 · 2018-08-23T19:24:03Z

I am renting these machines from Cloudlab (https://www.cloudlab.us/).

Since the time I obtained them (about 8 days ago), they have been running. I didn't reboot them.

Does that answer the question?

kvm2116 · 2018-08-23T20:05:42Z

Please let me know how I can help to identify or fix the issue.

kvm2116 · 2018-08-23T21:25:46Z

Not sure if this helps though I would like to point out that using "ipfs get file.txt" works as it should. It downloads the file in its entirety without any issues.

Stebalien · 2018-08-23T22:58:24Z

Oh. Hm. That's really odd. Can you run on ipfs refs -r $file on peer A. It looks like it actually has the file in question, it just doesn't realize it...

Just to make sure file.txt is the file you'r trying to download through fuse, right?

kvm2116 · 2018-08-23T23:07:20Z

Yes, file.txt is the file trying to be downloaded.

In the output, it prints one line and then hangs.
$ ipfs refs -r /ipns/QmYgbFxWGykyTMNrotndQmoV67RpixAsus6zkRzk5bB66Q/file.txt
QmaQ7e4fEWRizAUgGe5Dbbf5oAb8RfNtJCUpYxpYmQNbUn

Ah, I think I see what you mean.

In the earlier experiment, which worked:
On peer B, I did "ipfs add file.txt" -> output_hash
On peer A, "ipfs get output_hash"

In the /ipns mount scenario, which is hanging:
On peer B, "cp file.txt /ipns/local/file.txt"
On peer A, "head -c 1000000 /ipns/peer_a/file.txt"

Stebalien · 2018-08-23T23:40:44Z

Wait, but ipfs get /ipns/QmYgbFxWGykyTMNrotndQmoV67RpixAsus6zkRzk5bB66Q/file.txt succeeds? Is this on peer A?

kvm2116 · 2018-08-23T23:45:47Z

I just tried that and it succeeds.

on peer A (the client trying to download):
ipfs get /ipns/QmYgbFxWGykyTMNrotndQmoV67RpixAsus6zkRzk5bB66Q/file.txt
Saving file(s) to file.txt
286.17 MiB / 286.17 MiB [===========================================================] 100.00% 12s

Stebalien · 2018-08-23T23:55:15Z

Does it actually download the complete file (trying to rule out a bug in ipfs get).

Stebalien · 2018-08-23T23:55:31Z

And thank you for all your help on this!

kvm2116 · 2018-08-24T00:02:00Z

Sure thing. Happy to help! Please feel free to let me know any way I can help.

I did "cat file.txt" and it printed the entire file.
I also did a diff on the downloaded file and the original file and they are identical.

Stebalien · 2018-08-24T00:13:21Z

So, to summarize, on peer A:

ipfs get /ipns/Qm... works.
ipfs refs -r /ipns/Qm... hangs.
cat /ipns/Qm... hangs.

kvm2116 · 2018-08-24T15:55:30Z

Yes.
Any ideas on what might be causing this issue? I have the entire weekend available to run more tests/possibly implement the solution.

Stebalien · 2018-08-24T21:31:56Z

No but the get versus refs difference is probably going to narrow this down a lot.

Stebalien · 2018-08-24T21:41:02Z

If you re-run these commands (#5328 (comment)), do you get the same results?

kvm2116 · 2018-08-24T21:56:57Z

No, it varies. Sometimes more entries, sometimes less.

Stebalien · 2018-08-24T22:15:22Z

What's the output of ipfs diag cmds?

Stebalien · 2018-08-24T22:15:30Z

(on both machines)

kvm2116 · 2018-08-24T22:24:05Z

I ran "cat /ipns/Qm.../file.txt". After it hangs, obtained the output of "ipfs diag cmds".

Peer A (downloader):
Command Active StartTime RunTime
repo/gc false Aug 24 15:39:03 14.671006ms
refs false Aug 24 15:39:09 5.458619543s
repo/gc false Aug 24 15:57:06 135.123628ms
pin/ls false Aug 24 15:57:11 254.572µs
refs false Aug 24 15:57:23 5.416848016s
pin/ls false Aug 24 16:21:03 231.248µs
repo/gc false Aug 24 16:21:06 144.058141ms
diag/cmds true Aug 24 16:22:42 1.515304ms

Peer B (sender):
Command Active StartTime RunTime
bitswap/wantlist false Aug 24 15:36:00 888.235µs
bitswap/wantlist false Aug 24 15:36:07 227.549µs
bitswap/wantlist false Aug 24 15:36:48 352.875µs
bitswap/wantlist false Aug 24 15:38:27 222.73µs
diag/cmds true Aug 24 16:23:21 1.188582ms

Stebalien · 2018-08-24T22:45:33Z

And does ipfs get still work? Does ipfs refs still hang?

Internally, it looks like anything that might cause ipfs refs to hang should cause ipfs get to hang.

kvm2116 · 2018-08-24T22:50:19Z

ipfs get /ipns/Qm.../file.txt works ipfs refs /ipns/Qm.../file.txt works cat /ipns/Qm.../file.txt hangs After its hanging, get the list of hashes using "ipfs bitswap want list" For any one of the hashes from the list (let's say Qmhash), "ipfs refs Qmhash" prints couple lines of hashes and then hangs. On Aug 24, 2018, at 6:45 PM, Steven Allen <notifications@github.com> wrote: And does ipfs get *still work*? Does ipfs refs *still hang*? Internally, it looks like anything that might cause ipfs refs to hang should cause ipfs get to hang. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#5328 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AToQMDc04Sts6zmseQgcR7QmRwWHFGRCks5uUIIXgaJpZM4Vq3V-> .

kvm2116 · 2018-08-24T22:54:53Z

One more data point : cat /ipns/Qm.../file.txt hangs But, if I run ipfs refs /ipns/Qm.../file.txt (which works) and then "cat /ipns/Qm.../file.txt" (this succeeds) On Aug 24, 2018, at 6:45 PM, Steven Allen <notifications@github.com> wrote: And does ipfs get *still work*? Does ipfs refs *still hang*? Internally, it looks like anything that might cause ipfs refs to hang should cause ipfs get to hang. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#5328 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AToQMDc04Sts6zmseQgcR7QmRwWHFGRCks5uUIIXgaJpZM4Vq3V-> .

kvm2116 mentioned this issue Aug 23, 2018

IPFS private network video load break #4756

Open

Only partially able to read file through ipns mount #5328

Only partially able to read file through ipns mount #5328

Comments

kvm2116 commented Aug 1, 2018 • edited Loading

Version information:

Type:

Description:

kvm2116 commented Aug 23, 2018

Stebalien commented Aug 23, 2018

kvm2116 commented Aug 23, 2018

Stebalien commented Aug 23, 2018

kvm2116 commented Aug 23, 2018 • edited Loading

Stebalien commented Aug 23, 2018

kvm2116 commented Aug 23, 2018 • edited Loading

Stebalien commented Aug 23, 2018

kvm2116 commented Aug 23, 2018 • edited Loading

Stebalien commented Aug 23, 2018

kvm2116 commented Aug 23, 2018 • edited Loading

Stebalien commented Aug 23, 2018

kvm2116 commented Aug 23, 2018 • edited Loading

Stebalien commented Aug 23, 2018 • edited Loading

kvm2116 commented Aug 23, 2018 • edited Loading

kvm2116 commented Aug 23, 2018

kvm2116 commented Aug 23, 2018

Stebalien commented Aug 23, 2018

kvm2116 commented Aug 23, 2018 • edited Loading

Stebalien commented Aug 23, 2018

kvm2116 commented Aug 23, 2018

Stebalien commented Aug 23, 2018

Stebalien commented Aug 23, 2018

kvm2116 commented Aug 24, 2018

Stebalien commented Aug 24, 2018

kvm2116 commented Aug 24, 2018

Stebalien commented Aug 24, 2018

Stebalien commented Aug 24, 2018

kvm2116 commented Aug 24, 2018

Stebalien commented Aug 24, 2018

Stebalien commented Aug 24, 2018

kvm2116 commented Aug 24, 2018

Stebalien commented Aug 24, 2018

kvm2116 commented Aug 24, 2018 via email

kvm2116 commented Aug 24, 2018 via email

kvm2116 commented Aug 1, 2018 •

edited

Loading

kvm2116 commented Aug 23, 2018 •

edited

Loading

kvm2116 commented Aug 23, 2018 •

edited

Loading

kvm2116 commented Aug 23, 2018 •

edited

Loading

kvm2116 commented Aug 23, 2018 •

edited

Loading

kvm2116 commented Aug 23, 2018 •

edited

Loading

Stebalien commented Aug 23, 2018 •

edited

Loading

kvm2116 commented Aug 23, 2018 •

edited

Loading

kvm2116 commented Aug 23, 2018 •

edited

Loading