feat/ipfs.files.get

[daviddias]

Builds on top of #296

Missing:

• Fix adding a directory in the browser

  1) .files callback API .add add a nested dir as array:
     Error: Uncaught AssertionError: expected 'QmbyMGUYJ6WMeNYrVSJitxGMznaCictvwwUpAyiEiPGy4B' to equal 'QmVvjDy7yF7hdnqE8Hrf4MHo5ABDtb5AbX6hWbD3Y42bXP' (webpack:///~/chai/lib/chai/assertion.js?023a:107:0 <- test/api/files.spec.js:28388)



  2) .files .get directory:
     Error: timeout of 80000ms exceeded. Ensure the done() callback is being called in this test.

The second test times out because it is looking for a different hash

[daviddias]

Ok, I thought I had fixed the adding dirs on the browser, but then realised that there is a bug that only happens sometimes (like 1 out of 20), where we don't receive all the hashes for the files added.

I thought this could be an js-ipfs-api problem, but further testing (and some fixes) narrowed down the problem to how go-ipfs is responding, even when requests are 100% the same, the response may differ.

The failing failing test is: https://github.com/ipfs/interface-ipfs-core/blob/feat/ipfs.files.get/src/files.js#L113-L143 (again: just fails in the browser and some of the times)

(left -> success; center -> fails)

Full request + response of a Success
Full request + response of a Failure
wireshark pcap (filter for HTTP and then look for the POST /add

How to reproduce

> git clone git@github.com:ipfs/interface-ipfs-core.git -b feat/ipfs.files.get 
> cd interface-ipfs-core
> npm i && npm run build && npm link
> cd ..
> git clone git@github.com:ipfs/js-ipfs-api.git -b feat/ipfs.files.get
> cd js-ipfs-api 
> npm i && npm link interface-ipfs-core
> npm run test:browser # repeat this one till it fails

[hackergrrl]

While this is definitely a bug, it's not a blocker for getting files.get in. How about we file a separate issue for this and get js-ipfs-api and interface-ipfs-core in?

[whyrusleeping]

@diasdavid very strange... i'll look into it.

[whyrusleeping]

@diasdavid those steps fail for me:

➜  js-ipfs-api git:(feat/ipfs.files.get) npm run test:browser

> ipfs-api@6.0.3 test:browser /home/whyrusleeping/code/js-ipfs-api
> gulp test:browser

[14:26:57] Using gulpfile ~/code/js-ipfs-api/gulpfile.js
[14:26:57] Starting 'test:browser'...
[14:26:57] Starting 'daemons:start'...
  ipfs init done - (bootstrap and mdns off) - a
  ipfs init done - (bootstrap and mdns off) - b
  ipfs init done - (bootstrap and mdns off) - c
[14:26:58] Finished 'daemons:start' after 1.07 s
[14:26:58] Starting 'test:browser:before'...
[14:26:58] Finished 'test:browser:before' after 6.33 μs
[14:26:58] Starting 'karma'...
09 08 2016 14:26:58.608:ERROR [preprocess]: Can not load "webpack", it is not registered!
  Perhaps you are missing some plugin?
09 08 2016 14:26:58.609:ERROR [preprocess]: Can not load "sourcemap", it is not registered!
  Perhaps you are missing some plugin?
09 08 2016 14:26:58.615:ERROR [reporter]: Can not load reporter "mocha-own", it is not registered!
  Perhaps you are missing some plugin?
09 08 2016 14:26:58.648:INFO [karma]: Karma v1.1.2 server started at http://localhost:9876/
09 08 2016 14:26:58.648:INFO [launcher]: Launching browsers Chrome, PhantomJS with concurrency 1
09 08 2016 14:26:58.649:ERROR [karma]: Found 3 load errors

npm ERR! Linux 4.6.3-1-ARCH
npm ERR! argv "/usr/bin/node" "/usr/bin/npm" "run" "test:browser"
npm ERR! node v5.11.0
npm ERR! npm  v3.10.3
npm ERR! code ELIFECYCLE
npm ERR! ipfs-api@6.0.3 test:browser: `gulp test:browser`
npm ERR! Exit status 130
npm ERR! 
npm ERR! Failed at the ipfs-api@6.0.3 test:browser script 'gulp test:browser'.
npm ERR! Make sure you have the latest version of node.js and npm installed.
npm ERR! If you do, this is most likely a problem with the ipfs-api package,
npm ERR! not with npm itself.
npm ERR! Tell the author that this fails on your system:
npm ERR!     gulp test:browser
npm ERR! You can get information on how to open an issue for this project with:
npm ERR!     npm bugs ipfs-api
npm ERR! Or if that isn't available, you can get their info via:
npm ERR!     npm owner ls ipfs-api
npm ERR! There is likely additional logging output above.

npm ERR! Please include the following file with any support request:
npm ERR!     /home/whyrusleeping/code/js-ipfs-api/npm-debug.log

[nginnever]

@whyrusleeping, did you see any errors pop up when running the npm install and link commands?

It sounds like you may have a global installation conflicting with the local karma pluggins as commented on by our very own @dignifiedquire here. You could try uninstalling all of the global karma deps and then rerun the local installs. Or maybe dig knows a better solution.

I was able to reproduce the bug on the first try.

[daviddias]

thank you @nginnever, yeah @whyrusleeping it is probably that. You can uninstall the global deps with npm uninstall <module name> --global

[daviddias]

@noffle I was thinking about something along those lines, @dignifiedquire would you like to give your opinion?

[dignifiedquire]

I agree that this has nothing to do with this PR in particular as the test fails on a different feature.

[daviddias]

Ok, going to move ahead and merge. For reference, the tests that fail are:

  1) .files callback API .add add a nested dir as array:
     AssertionError: expected [ Array(2) ] to have a length of 6 but got 2 (:0)

  2) .files .get directory:
     timeout of 80000ms exceeded. Ensure the done() callback is being called in this test.

  3) .bitswap .wantlist:
     AssertionError: expected { Object (Keys) } to deeply equal { Keys: null } (webpack:///~/assertion-error/index.js?5548*********************:74:0 <- test/api/bitswap.spec.js:2648)

  4) .bitswap promise .wantlist:
      expected { Object (Keys) } to deeply equal { Keys: null }
      + expected - actual
       {
      -  "Keys": [
      -    "QmVvjDy7yF7hdnqE8Hrf4MHo5ABDtb5AbX6hWbD3Y42bXP"
      -  ]
      +  "Keys": [null]
       }

Because it is a cascading failure, since add doesn't complete well, get tried to get a wrong hash (and timesout) and then because we tried to get an hash that doesn't live in the network, bitswap has one more hash in the want list

Just going to release after #339 is fixed