tutorial: File Exchange - Transfer files between desktop and browser using go-ipfs, js-ipfs and orbit-db #716

haadcode · 2017-01-18T13:53:20Z

This is a WIP PR for the second interop example. A File Manager in the browser, which eventually will be able to display files added in go-ipfs. It currently works only between browser js-ipfs nodes.

No need to review it yet, still working on it.

daviddias · 2017-01-20T10:19:53Z

Not getting this example to work:

daviddias · 2017-01-20T10:22:55Z

This example is looking very similar to PaperHub that @dignifiedquire made a while ago, it is interesting, but it the main goal is to have a file manager that can open directories and traverse to them, these directories should be added through go-ipfs.

I know you are still heavily working on this one and that you have the intent of making it the actual File Browser, so I will review it again when it is fulfilling that goal.

daviddias · 2017-01-20T10:24:33Z

Ah I added a file on firefox and it showed up in chrome:

And then, I added an extra one to Chrome, and the 3 files appeared in Firefox too:

daviddias · 2017-01-20T10:26:54Z

I believe that this 'injection' of orbit into the example is creating a distraction to the main goal, which is why developing it incrementally, as suggested, would have been a big win.

Now, the better is just consider the time-left/effort to achieve that main goal and see if we get closer by taking out orbit out of the equation or not.

daviddias · 2017-01-20T12:13:23Z

@haadcode updated the name of this PR to reflect the decision made through IRC chat. One task that needs to also happen is:

12:01 <haad> daviddias: eating lunch, will get back to you in a few. re. go-ipfs aprt of the tutorial for file feed: I was planning on writing a small .js file that spawns a go-ipfs daemon and adds the file to ipfs+orbit-db.
12:02 <haad> daviddias: ok so re that ^, should object.get/object.put and pubsub interop work with go-ipfs<-->js-ipfs?
12:02 <haad> because if they won't, then we can't do that and we're better off doing the file browser
12:05 <@daviddias> haad: I would say pubsub interop: yes it works, but won't officially say it unless there is a test in interop for that - https://github.com/ipfs/js-ipfs/tree/master/test/interop, it only thing I can say is: It used to work, now something might have changed, we can't tell.
12:05 <@daviddias> That interop test should be part of the list of tasks for that interop example (File Feed)

haadcode · 2017-01-23T09:37:54Z

The browserify-zlib-next dependecy problems have been solved. What is still needed is to merge this: ipfs-shipyard/browserify-zlib-next#23. cc @diasdavid @dignifiedquire

haadcode · 2017-01-23T09:38:23Z

I'm fixing and updating everything right now to make sure this can be finished by someone.

dignifiedquire · 2017-01-23T10:48:29Z

browserify-zlib-next@1.0.1 was just released with the fix

haadcode · 2017-01-23T11:21:32Z

Thank you @dignifiedquire 🎉

haadcode · 2017-01-23T11:37:30Z

Can verify that everything works now with browserify-zlib-next 1.0.1. Hooray \o/

daviddias · 2017-01-27T10:16:03Z

TODOs left by @haadcode

TODO

Need to merge fix: Use Buffer.isBuffer to check buffer type ipfs-shipyard/browserify-zlib-next#23 and use the updated module. Otherwise the UI will error on zlib._transform.
Make sure go-ipfs and the UI connect to each other (last time I checked they didn't)
Use new signal server address everywhere (src/App.js, ipfe-daemon.js, ipfe-add.js)
ipfe-add currently uses js-ipf(node.js) daemon. But in order to add files from go-ipfs daemon, it would need to use js-ipfd-ctl. A quick way would be to use ipfs-daemon (I would highly recommend to do so), otherwise need to add the init dance for js-ipfsd-ctl much like src/ipfs.js.

daviddias · 2017-01-27T10:37:18Z

The cli tool to add files is pointing at local signalling server
still uses ipfs-daemon

daviddias · 2017-01-27T10:39:26Z

Found that it works in the browser with no signalling server at all because the browser is mounting always on top of the same repo.

daviddias · 2017-01-27T14:42:31Z

Got the Desktop example to work with DNS

daviddias · 2017-01-27T15:35:43Z

examples/file-feed/complete/src/browser-app/DataStore.js

+      }
+      this.ipfs = node
+      this.emit('ready')
+    })


Async operations in a constructor are an anti-pattern, a good source of errors

Getting issues from this

ah got it, this always happens if we open without being on a feed which is controlled by the top level domain. This is a bug

Indeed, this is not good. Will change it.

daviddias · 2017-01-27T16:22:25Z

@haadcode how did you manage to use WebPack@2? I'm missing to find any note and ipfs doesn't really work if bundled with WebPack@1 and as @dignifiedquire just mentioned, WebPack@2 will only come in the next version of create-react-app.

If there was alternative, shall we eject and update the react deps?

daviddias · 2017-01-28T08:43:59Z

Clicking the "number of peers" throws

daviddias · 2017-01-28T08:44:55Z

If a peer joins after files where added, it stays on 'Loading database' forever

daviddias

Currently, the App isn't in a state where it is easy to use it, as it shows several bugs during its usage. It might be necessary, since it is a complex app, to have tests for it, it will help the debugging while the remaining of the development is finished.

I've structured it in the same way the other tutorial is, added a bit more information to the Readme and talked with @RichardLitt about making the full tutorial. I'll still add a diagram of the app.

@haadcode I would appreciate your help on getting this ready for Monday, I'll be around Sunday when you come back for us to catch up :)

daviddias · 2017-01-28T09:24:10Z

examples/file-feed/.gitignore

+npm-debug.log
+
+# orbit-db
+orbit-db


Why do we need to ignore a folder named orbit-db?

OrbitDB writes its cache file to that directory when run with go-ipfs or js-ipfs from the CLI. This is per user and we shouldn't add it to the repo.

daviddias · 2017-01-28T09:25:07Z

examples/file-feed/README.md

+## Step-by-step instructions
+
+`TODO`
+


@RichardLitt is on track to do this once the example is completely done. @RichardLitt I'll notify you when that happens ❤️

daviddias · 2017-01-28T09:25:59Z

examples/file-feed/complete/config/webpack.config.dev.js

+    net: 'empty',
+    tls: 'empty'
+  }
+}


Can we validate that we actually need all of this configuration?

Yeah, we need all this configuration. See my long comment below as to why.

Short: the webpack config file here is the create-react-app/react-script config file with the only difference that we add the zlib shim to it.

daviddias · 2017-01-28T09:27:49Z

examples/file-feed/complete/package.json

@@ -0,0 +1,25 @@
+{
+  "name": "file-feed",


I've renamed it to file-feed as I'm afraid of stamping something with a "InterPlanetary X" name that is not a project (i.e example/tutorial) that we support and make it have the same level of activity of a product.

Pointing people to the 'file-feed tutorial' has a completely different tone than 'InterPlanetary FileExchange'

@haadcode hope you understand, we can chat about it and reach an agreement.

@diasdavid I'm totally fine changing the name! However, let's catch up on IRC today to discuss the name as I think we can find something more descriptive and less "loaded" word than "feed" (people will make assumption about what a fedd is that I think we can't provide in this example).

I'll be around for the rest of the day

daviddias · 2017-01-28T09:28:48Z

examples/file-feed/complete/package.json

+{
+  "name": "file-feed",
+  "version": "0.0.0",
+  "bin": "src/ff-cli/bin.js",


I've also put the CLI under the bin path, for ease of use.

Good! I was thinking we could yargsify the CLI tool so that we can run
ff-cli daemon hello-world and ff-cli add hello-world file.txt instead of having two separate programs. This would make the code sharing between the two also a lot easier. Unless you're against yargsifying it, I'll go ahead and do that today.

👍 not against at all :)

daviddias · 2017-01-28T09:33:34Z

examples/file-feed/complete/src/browser-app/DataStore.js

+  }
+
+  getFile (hash) {
+    return new Promise((resolve, reject) => {


Back in the January meetings, we talked about using Promises in examples and we agreed at the table that we would use only callbacks for our examples. Is the use of Promises strictly necessary here?

I don't remember us talking about Promises, but sure, we can make these callbacks too.

daviddias · 2017-01-28T09:34:00Z

examples/file-feed/complete/src/browser-app/DataStore.js

+          stream.on('end', () => {
+            this.emit('load', hash, bytes)
+            resolve(buffer)
+          })


Use bl or concat for this

I would prefer not adding a library to abstract the underlaying functionality of the stream. After all, this is an example and people will have different approaches to how they use streams, some will use bl or concat, some won't., so keeping it clear what ipfs returns is clearer in this case imo.

daviddias · 2017-01-28T09:34:26Z

examples/file-feed/complete/src/browser-app/DataStore.js

+
+  addFiles (file) {
+    const addToIpfs = (name, content) => {
+      return this.ipfs.files.add([{


IPFS might not exist in this context due to the async operation in the constructor.

Indeed. Will fix the constructor problem.

daviddias · 2017-01-28T09:34:48Z

examples/file-feed/complete/src/browser-app/DataStore.js

+    }
+
+    const addToOrbitDB = (file, type) => {
+      return this.feed.add({


this.feed doens't exist if we aren't in a path.

You're right, feed is locally scoped. Will address this.

This was fixed in a7b6049. Constructor is not async anymore.

daviddias · 2017-01-28T09:36:25Z

examples/file-feed/complete/src/ff-cli/daemon.js

+
+  setInterval(checkForPeers, 1000)
+  setInterval(checkForFiles, 1000)
+})


The daemon and the cli tool share a ton of code, do we really need too files for this?

See my previous comment re. yargisyfying and combining the two program into one. Yes, we should share the code :)

daviddias · 2017-01-28T10:00:20Z

feature request: A very cool features would be to 'watch' a folder, and for every file to gets added to the folder, it syncs to the browser and for every file that is added to the browser, it syncs back to the folder.

Rebase on master. Make interop work on CLI.

daviddias · 2017-01-30T15:46:32Z

Btw. once we're done and ready to merge, we should squash all these commits to one with a nice commit message as per our guidelines.

Could you squash now? If we get into the habit of pushing clean commit messages, it will be easier to keep it consistent

daviddias · 2017-05-25T15:26:14Z

Hey @pgte it seems that this example froze in time, however, it is pretty valuable.

Could you take a look and make it work just with IPFS primitives? You would end up creating a 'small crdt-log' to make it work on top of pubsub, but if that is a single file vs a full external dependency, I feel that would create way more value and direction for the users.

pgte · 2017-05-25T16:11:26Z

@diasdavid sure, I'll look at this once I'm done with ipfs-level.

daviddias · 2017-12-14T09:52:31Z

Closing this PR as the example was never developed further and since then there were a ton of advancements in js-ipfs and orbit

haadcode added the status/in-progress In progress label Jan 18, 2017

haadcode mentioned this pull request Jan 18, 2017

Sprint: Browser accesses go-ipfs content ipfs/team-mgmt#310

Closed

50 tasks

daviddias assigned haadcode Jan 19, 2017

daviddias changed the title ~~(WIP) Advanced interop example~~ (WIP) Example: File Browser - Traverse through directories added from go-ipfs, in the browser Jan 19, 2017

This was referenced Jan 20, 2017

js/go interop not working #718

Closed

browser/nodejs interop issues #720

Closed

daviddias changed the title ~~(WIP) Example: File Browser - Traverse through directories added from go-ipfs, in the browser~~ (WIP) Tutorial: File File - Track Files being added with go-ipfs using orbit-db Jan 20, 2017

daviddias changed the title ~~(WIP) Tutorial: File File - Track Files being added with go-ipfs using orbit-db~~ (WIP) Tutorial: Files Feed - Track Files being added with go-ipfs using orbit-db Jan 20, 2017

daviddias force-pushed the dcos/interop-example-2 branch from 3313cf3 to 07d97fe Compare January 25, 2017 15:01

daviddias reviewed Jan 27, 2017

View reviewed changes

daviddias force-pushed the dcos/interop-example-2 branch from 074e1f1 to 8b826c5 Compare January 28, 2017 09:23

daviddias suggested changes Jan 28, 2017

View reviewed changes

haadcode and others added 8 commits January 30, 2017 17:11

WIP

e389566

Rebase on master. Make interop work on CLI.

Use latest Buffer as dev dep

e155f4d

Tweak Peers view

0f57166

fix: MDNS on config + avoid using always the same repo

d2317fd

Change name to File Exchange

ec57ebb

Display Swarm peers

100abb9

Improve CLI output

1b6fbf7

Update instructions in README

7d120c6

haadcode force-pushed the dcos/interop-example-2 branch from 97be250 to 7d120c6 Compare January 30, 2017 15:11

Quick fixes for demo

e2eb227

daviddias changed the title ~~(WIP) Example: File Exchange - Transfer files between desktop and browser using go-ipfs, js-ipfs and orbit-db~~ tutorial: File Exchange - Transfer files between desktop and browser using go-ipfs, js-ipfs and orbit-db Jan 31, 2017

daviddias mentioned this pull request Mar 11, 2017

Finish js-ipfs Tutorial: transfer-files ipfs-inactive/docs#28

Closed

1 task

daviddias added status/ready Ready to be worked and removed status/in-progress In progress labels Mar 13, 2017

daviddias added status/deferred Conscious decision to pause or backlog and removed status/ready Ready to be worked labels Jun 20, 2017

daviddias closed this Dec 14, 2017

ghost removed the status/ready Ready to be worked label Dec 14, 2017

daviddias deleted the dcos/interop-example-2 branch December 14, 2017 09:52

dryajov restored the dcos/interop-example-2 branch March 3, 2018 18:04

daviddias deleted the dcos/interop-example-2 branch June 4, 2018 11:49

tutorial: File Exchange - Transfer files between desktop and browser using go-ipfs, js-ipfs and orbit-db #716

tutorial: File Exchange - Transfer files between desktop and browser using go-ipfs, js-ipfs and orbit-db #716

Conversation

haadcode commented Jan 18, 2017

daviddias commented Jan 20, 2017

daviddias commented Jan 20, 2017

daviddias commented Jan 20, 2017

daviddias commented Jan 20, 2017 • edited Loading

daviddias commented Jan 20, 2017

haadcode commented Jan 23, 2017

haadcode commented Jan 23, 2017

dignifiedquire commented Jan 23, 2017

haadcode commented Jan 23, 2017

haadcode commented Jan 23, 2017

daviddias commented Jan 27, 2017

TODO

daviddias commented Jan 27, 2017 • edited Loading

daviddias commented Jan 27, 2017

daviddias commented Jan 27, 2017

Choose a reason for hiding this comment

daviddias Jan 28, 2017 • edited Loading

Choose a reason for hiding this comment

daviddias Jan 28, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

daviddias commented Jan 27, 2017 • edited Loading

daviddias commented Jan 28, 2017

daviddias commented Jan 28, 2017

daviddias left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

daviddias commented Jan 28, 2017

daviddias commented Jan 30, 2017

daviddias commented May 25, 2017 • edited Loading

pgte commented May 25, 2017

daviddias commented Dec 14, 2017

daviddias commented Jan 20, 2017 •

edited

Loading

daviddias commented Jan 27, 2017 •

edited

Loading

daviddias Jan 28, 2017 •

edited

Loading

daviddias Jan 28, 2017 •

edited

Loading

daviddias commented Jan 27, 2017 •

edited

Loading

daviddias commented May 25, 2017 •

edited

Loading