Next ChluIPFS Iteration

[fazo96]

PR Content:

• Circuit Relay (all this stuff is disabled by default for now)

  • make it possible to disable usage of rendezvous endpoints, but enable them by default
  • allow relay on all Chlu nodes, meaning they will all be able to use relays, but disable it by default
  • add option to act as relay (relay-hop) to service node binary, disabled by default
  • set up Node.js nodes to listen for incoming websocket connections, but disable by default
  • add a list of Chlu bootstrap nodes and try to connect to them on start, but don't crash on failure and disable it by default (we don't need it yet)
  • set up the EC2 service node to receive WSS connections and add it to the bootstrap node list
  • use a release of IPFS that supports circuit relay (blocked until js-ipfs is released, using master now) we'll leave this for later since circuit relay is not ready for our use case
  • test circuit relay to make sure network is stable not yet

• IPFS API (all this stuff is disabled by default for now)

  • use a version of orbit-db that supports IPFS-API (blocked ultil orbit-db is released, using master now) leave this for later
  • add configuration options to connect to remote IPFS node, but hide it by default (doesn't work until orbit-db releases)
  • add CLI option to service node to connect to remote IPFS node
  • add configuration option and CLI option to start a go-ipfs node in the background instead of a js one when running in node.js. Maybe even enable this by default. not yet, the js and go networks are still kind of separated
  • update multinode tests to use js-ipfsd-ctl for stability, and have the test service node run on go-ipfs (leave this for last) not yet, but I managed to optimize the tests anyway by reusing ONLY the IPFS instance between tests (orbit-db and everything else still gets wiped)

• Offline Mode

  • service node has an --offline flag that starts a rendezvous server and uses it
  • other Chlu apps on the same machine as an offline service node can detect it and turn on offline mode as well, allowing for full Chlu connectivity locally without an internet connection

• Verify BTC transactions: The First Pass

  • bitcoin module that uses blockcypher and chlu-wallet-support-js to fetch BTC transaction info
  • pass TX hash to publishReviewRecord
    • include it in pubsub message (is this necessary? Is this whole pubsub thing necessary anymore? Need to think about this)
    • include it in orbitDb operation
  • validator for BTC transaction
    • checks OP_RETURN
    • checks amounts with RR and PoPR
    • checks addresses
  • service node: support turning on blockchain access, and fail if it's not done
  • upgrade OrbitDB index to keep track of Tx IDs. Maybe try using level for storing it
    • include required data in operations
    • keep it in the index and get it back when validating
  • tests
    • validator
    • bitcoin module
    • update multinode tests to use bitcoin module (with mocked blockcypher api)
      Events
  • use a rigid, namespaced scheme for names
    Pubsub
  • longer timeouts and less retries when submitting messages and awaiting replies

Issues remaining:

• due to missing persistence of tx<->rr mapping, if a customer makes a review record without a tx it will fail because the service node will recognize it's not an update and will require a tx ID to approve it. However, if the customer then updates this invalid RR, the update and the old version will now be valid because there is no way to check a transaction unless the RR is being created right now fixed
• accidentally poisoning cache: when writing RR from demo, the first time it's validated without a txId. This caches the validation result (successful) so if then it's validated before publishing, with the Tx, it will always pass due to hitting the cache (only on the customer) Fixed by disabling validation cache when storing/publishing
• getChain CORS issue in the browser. Fixed by Misc Fixes chlu-wallet-support-js#17 and fix getChain API in the browser node-client#3
• don't keep the whole DB Index in memory.. but how do we handle it then?

Refs:

• fixes part of Transaction validation #86
• closes verify that a matching transaction exist for a review record #85
• fixes part of Use Circuit Relay instead of rendezvous servers #78
• closes go-ipfs support #3
• closes ipfs-api support #9
• closes Offline mode #57
• closes Use rigid naming scheme for events #65
• closes Tweak RR submission timeouts #82

[fazo96]

@kulpreet I updated this PR so that we can merge it now:

I left there the support for circuit relay, IPFS API, the Chlu Bootstrap Nodes etc that I added but disabled or hidden it all for now (since it does not work)

I also disabled the usage of unreleased IPFS and OrbitDB versions.

This means that we can merge this without worries and can re enable the nice stuff once it's ready, let me know how you feel about that.

Also since this PR also contains #81 either we merge that one first or we close that one and we merge this. I think it's best to only keep this one open