go-ipfs sometimes closes the stream preemptively #339

daviddias · 2016-08-10T08:11:48Z

Ok, I thought I had fixed the adding dirs on the browser, but then realised that there is a bug that only happens sometimes (like 1 out of 20), where we don't receive all the hashes for the files added.

I thought this could be an js-ipfs-api problem, but further testing (and some fixes) narrowed down the problem to how go-ipfs is responding, even when requests are 100% the same, the response may differ.

The failing failing test is: https://github.com/ipfs/interface-ipfs-core/blob/feat/ipfs.files.get/src/files.js#L113-L143 (again: just fails in the browser and some of the times)

(left -> success; center -> fails)

Full request + response of a Success
Full request + response of a Failure
wireshark pcap (filter for HTTP and then look for the POST /add

How to reproduce

> git clone git@github.com:ipfs/interface-ipfs-core.git -b feat/ipfs.files.get 
> cd interface-ipfs-core
> npm i && npm run build && npm link
> cd ..
> git clone git@github.com:ipfs/js-ipfs-api.git -b feat/ipfs.files.get
> cd js-ipfs-api 
> npm i && npm link interface-ipfs-core
> npm run test:browser # repeat this one till it fails

whyrusleeping · 2016-08-10T16:46:51Z

@diasdavid @dignifiedquire could you find a way to reproduce this with a simple curl command? I guess i'm just not very good at javascript but i have trouble even getting these tests to run, consistently hitting new and exciting problems.

daviddias · 2016-08-10T17:05:35Z

After we talked and you getting the test to run, are you still getting that issue?

whyrusleeping · 2016-08-10T17:12:38Z

apparently i just needed to rerun it a few times for the tests to run. I was having issues with karma and chrome, but a few more tries and it started running the tests.

whyrusleeping · 2016-08-11T18:25:28Z

A relevant note: this has only been reproduced in the browser testing.

whyrusleeping · 2016-08-18T23:26:25Z

might be related to: ipfs/kubo#2112

Which i can only reproduce using ipfs 0.4.2 or earlier.

daviddias · 2016-08-25T04:20:10Z

Still seeing the same problem in 0.4.3.

whyrusleeping · 2016-08-25T16:12:42Z

@diasdavid thanks for checking.

daviddias · 2016-08-29T14:56:08Z

@whyrusleeping anything else I can do to help with this bug?

daviddias · 2016-09-07T20:06:06Z

Any developments?

…nt#339 is fixed

daviddias · 2016-12-05T04:03:49Z

@keks, @whyrusleeping mentioned that you found an issue on the http-api that might be the root of this problem (about closing the multipart stream), does it ring a bell?

daviddias · 2017-11-17T19:08:27Z

Spoke too soon. This is still a thing, although less frequently!

GTRI-nathan · 2018-01-17T19:33:58Z

I think I might be running into this error, but it happens every time for me. I made a quick and dirty paired down testcase here: https://github.com/GTRI-nathan/js-ipfs-api-test
Does this seem to be the same thing or should I open a new issue?

Thanks!

julientregoat · 2018-03-19T16:47:14Z

I think the same was happening to me. When I try to create a directory of files using ipfs.files.add([{path: /dir}, {path: /dir/filename.type}]), it only creates the directory, then stops at the next file.

I get the error from my daemon'expected at least one child dir'. But outside of an array of objects, it's working completely fine. Not sure if this is the same problem but it seemed similar since it would just stop and not return any errors.

Working around it with using ipfs.files.mkdir and write.

License: MIT Signed-off-by: Alan Shaw <alan@tableflip.io>

* feat: uses modular interface tests Reduces code repetition, allows test skipping and running only some tests. License: MIT Signed-off-by: Alan Shaw <alan@tableflip.io> * feat: skip unimplemented files tests and add ls* tests License: MIT Signed-off-by: Alan Shaw <alan@tableflip.io> * fix: adds skips for #339 License: MIT Signed-off-by: Alan Shaw <alan@tableflip.io> * fix: adds skips for key and miscellaneous License: MIT Signed-off-by: Alan Shaw <alan@tableflip.io> * feat: add types and util tests (skipped as currently failing) License: MIT Signed-off-by: Alan Shaw <alan@tableflip.io> * feat: add pin tests License: MIT Signed-off-by: Alan Shaw <alan@tableflip.io> * fix(pubsub): adds skips for tests on windows License: MIT Signed-off-by: Alan Shaw <alan@tableflip.io> * fix(config): adds skip for config.replace License: MIT Signed-off-by: Alan Shaw <alan@tableflip.io> * chore: re-adds bitswap tests License: MIT Signed-off-by: Alan Shaw <alan@tableflip.io> * chore: move detect-node back to dependencies License: MIT Signed-off-by: Alan Shaw <alan@tableflip.io> * chore: add skip reasons License: MIT Signed-off-by: Alan Shaw <alan@tableflip.io> * chore: update interface-ipfs-core dependency License: MIT Signed-off-by: Alan Shaw <alan@tableflip.io> * chore: add reason for pubsub in browser skips License: MIT Signed-off-by: Alan Shaw <alan@tableflip.io> * chore: add bitswap skips for offline errors License: MIT Signed-off-by: Alan Shaw <alan@tableflip.io> * fix: remove skip for test that was removed License: MIT Signed-off-by: Alan Shaw <alan@tableflip.io> * chore: update interface-ipfs-core version License: MIT Signed-off-by: Alan Shaw <alan@tableflip.io> * chore: update deps and test on CI (#802) * chore: update interface-ipfs-core * fix(dag): `dag.put()` allows for optional options (#801) * fix(dag): ensure `dag.put()` allows for optional options This is to align with API changes made in ipfs-inactive/interface-js-ipfs-core@011c417 and ipfs/js-ipfs#1415 License: MIT Signed-off-by: Pascal Precht <pascal.precht@gmail.com> * fix(dag): fixes to allow options to be optional License: MIT Signed-off-by: Alan Shaw <alan@tableflip.io> * chore: update interface-ipfs-core dependency License: MIT Signed-off-by: Alan Shaw <alan@tableflip.io> * chore: update ipfsd-ctl dependency License: MIT Signed-off-by: Alan Shaw <alan@tableflip.io> * fix: increase timeout for addFromURL with wrap-with-directory Sadly we can't guarantee ipfs.io will respond within 5s License: MIT Signed-off-by: Alan Shaw <alan@tableflip.io> * fix: skip bitswap offline tests on all platforms The offline tests create and stop a node. ipfs/kubo#4078 seems to not only be restricted to windows. License: MIT Signed-off-by: Alan Shaw <alan@tableflip.io>

* feat: uses modular interface tests Reduces code repetition, allows test skipping and running only some tests. License: MIT Signed-off-by: Alan Shaw <alan@tableflip.io> * feat: skip unimplemented files tests and add ls* tests License: MIT Signed-off-by: Alan Shaw <alan@tableflip.io> * fix: adds skips for ipfs-inactive#339 License: MIT Signed-off-by: Alan Shaw <alan@tableflip.io> * fix: adds skips for key and miscellaneous License: MIT Signed-off-by: Alan Shaw <alan@tableflip.io> * feat: add types and util tests (skipped as currently failing) License: MIT Signed-off-by: Alan Shaw <alan@tableflip.io> * feat: add pin tests License: MIT Signed-off-by: Alan Shaw <alan@tableflip.io> * fix(pubsub): adds skips for tests on windows License: MIT Signed-off-by: Alan Shaw <alan@tableflip.io> * fix(config): adds skip for config.replace License: MIT Signed-off-by: Alan Shaw <alan@tableflip.io> * chore: re-adds bitswap tests License: MIT Signed-off-by: Alan Shaw <alan@tableflip.io> * chore: move detect-node back to dependencies License: MIT Signed-off-by: Alan Shaw <alan@tableflip.io> * chore: add skip reasons License: MIT Signed-off-by: Alan Shaw <alan@tableflip.io> * chore: update interface-ipfs-core dependency License: MIT Signed-off-by: Alan Shaw <alan@tableflip.io> * chore: add reason for pubsub in browser skips License: MIT Signed-off-by: Alan Shaw <alan@tableflip.io> * chore: add bitswap skips for offline errors License: MIT Signed-off-by: Alan Shaw <alan@tableflip.io> * fix: remove skip for test that was removed License: MIT Signed-off-by: Alan Shaw <alan@tableflip.io> * chore: update interface-ipfs-core version License: MIT Signed-off-by: Alan Shaw <alan@tableflip.io> * chore: update deps and test on CI (ipfs-inactive#802) * chore: update interface-ipfs-core * fix(dag): `dag.put()` allows for optional options (ipfs-inactive#801) * fix(dag): ensure `dag.put()` allows for optional options This is to align with API changes made in ipfs-inactive/interface-js-ipfs-core@011c417 and ipfs/js-ipfs#1415 License: MIT Signed-off-by: Pascal Precht <pascal.precht@gmail.com> * fix(dag): fixes to allow options to be optional License: MIT Signed-off-by: Alan Shaw <alan@tableflip.io> * chore: update interface-ipfs-core dependency License: MIT Signed-off-by: Alan Shaw <alan@tableflip.io> * chore: update ipfsd-ctl dependency License: MIT Signed-off-by: Alan Shaw <alan@tableflip.io> * fix: increase timeout for addFromURL with wrap-with-directory Sadly we can't guarantee ipfs.io will respond within 5s License: MIT Signed-off-by: Alan Shaw <alan@tableflip.io> * fix: skip bitswap offline tests on all platforms The offline tests create and stop a node. ipfs/kubo#4078 seems to not only be restricted to windows. License: MIT Signed-off-by: Alan Shaw <alan@tableflip.io>

achingbrain · 2020-03-09T14:16:36Z

This issue is very stale so I'm going to close it. Tests were previously skipped for this PR but aren't any more. Please open a new issue against ipfs/js-ipfs if you still see it.

daviddias added the bug label Aug 10, 2016

daviddias assigned whyrusleeping Aug 10, 2016

daviddias mentioned this issue Aug 10, 2016

feat/ipfs.files.get #337

Merged

1 task

daviddias mentioned this issue Aug 12, 2016

feat(factory): use the factory mode #340

Merged

daviddias changed the title ~~go-ipfs sometimes closes the stream preemptively~~ go-ipfs sometimes closes the stream preemptively Aug 20, 2016

daviddias added the milestone 2 label Aug 20, 2016

daviddias added js-ipfs-backlog and removed milestone 2 labels Oct 30, 2016

daviddias mentioned this issue Nov 7, 2016

Async Crypto Endeavour #390

Merged

daviddias added a commit to ipfs-inactive/interface-js-ipfs-core that referenced this issue Nov 10, 2016

feat: disable nested array test until ipfs-inactive/js-ipfs-http-clie…

6d07f92

…nt#339 is fixed

This was referenced Nov 10, 2016

Update interface-ipfs-core to version 0.18.4 🚀 ipfs/js-ipfs#579

Closed

interface-ipfs-core@0.18.4 breaks build ⚠️ #417

Closed

This was referenced Nov 17, 2016

Update interface-ipfs-core to version 0.19.0 🚀 #430

Closed

Update interface-ipfs-core to version 0.20.0 🚀 #431

Closed

daviddias added ready help wanted difficulty:hard and removed js-ipfs-backlog labels Dec 5, 2016

daviddias added backlog and removed ready labels Jan 29, 2017

daviddias self-assigned this Nov 17, 2017

daviddias added in progress and removed ready labels Nov 17, 2017

daviddias added a commit to ipfs-inactive/interface-js-ipfs-core that referenced this issue Nov 17, 2017

enable all tests to fix ipfs-inactive/js-ipfs-http-client#339

f6aba4a

daviddias closed this as completed in ipfs-inactive/interface-js-ipfs-core@8f34b0e Nov 17, 2017

ghost removed the in progress label Nov 17, 2017

daviddias removed their assignment Nov 17, 2017

daviddias reopened this Nov 17, 2017

daviddias added ready P1 - High and removed P0 - Critical labels Nov 20, 2017

olizilla mentioned this issue Apr 19, 2018

crash when dropping a big file ipfs/ipfs-companion#464

Closed

2 tasks

alanshaw added a commit that referenced this issue Jun 6, 2018

fix: adds skips for #339

73591fc

License: MIT Signed-off-by: Alan Shaw <alan@tableflip.io>

alanshaw added a commit that referenced this issue Jun 25, 2018

fix: adds skips for #339

600ad98

License: MIT Signed-off-by: Alan Shaw <alan@tableflip.io>

alanshaw added a commit that referenced this issue Jun 26, 2018

fix: adds skips for #339

e2168ed

License: MIT Signed-off-by: Alan Shaw <alan@tableflip.io>

alanshaw added a commit that referenced this issue Jun 29, 2018

fix: adds skips for #339

f325e50

License: MIT Signed-off-by: Alan Shaw <alan@tableflip.io>

achingbrain closed this as completed Mar 9, 2020

github-actions bot mentioned this issue Sep 22, 2021

chore: release js-ipfs 1.0.0 ipfs/js-ipfs#3887

Closed

snyk-bot mentioned this issue Dec 18, 2021

[Snyk] Upgrade ky from 0.15.0 to 0.28.7 thexdesk/js-ipfs-http-client#2

Open

github-actions bot mentioned this issue Jan 25, 2022

chore: release js-ipfs 0.1.0 ipfs/js-ipfs#4026

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

go-ipfs sometimes closes the stream preemptively #339

go-ipfs sometimes closes the stream preemptively #339

daviddias commented Aug 10, 2016 •

edited

Loading

whyrusleeping commented Aug 10, 2016

daviddias commented Aug 10, 2016

whyrusleeping commented Aug 10, 2016

whyrusleeping commented Aug 11, 2016

whyrusleeping commented Aug 18, 2016

daviddias commented Aug 25, 2016

whyrusleeping commented Aug 25, 2016

daviddias commented Aug 29, 2016

daviddias commented Sep 7, 2016

daviddias commented Dec 5, 2016

daviddias commented Nov 17, 2017

GTRI-nathan commented Jan 17, 2018

julientregoat commented Mar 19, 2018 •

edited

Loading

achingbrain commented Mar 9, 2020

go-ipfs sometimes closes the stream preemptively #339

go-ipfs sometimes closes the stream preemptively #339

Comments

daviddias commented Aug 10, 2016 • edited Loading

How to reproduce

whyrusleeping commented Aug 10, 2016

daviddias commented Aug 10, 2016

whyrusleeping commented Aug 10, 2016

whyrusleeping commented Aug 11, 2016

whyrusleeping commented Aug 18, 2016

daviddias commented Aug 25, 2016

whyrusleeping commented Aug 25, 2016

daviddias commented Aug 29, 2016

daviddias commented Sep 7, 2016

daviddias commented Dec 5, 2016

daviddias commented Nov 17, 2017

GTRI-nathan commented Jan 17, 2018

julientregoat commented Mar 19, 2018 • edited Loading

achingbrain commented Mar 9, 2020

daviddias commented Aug 10, 2016 •

edited

Loading

julientregoat commented Mar 19, 2018 •

edited

Loading