Dispersy mini: Dispersy no longer stores any data in databases #2778
Comments
synctext
changed the title
|
Maybe we could even look at the byte stats that go through the socket (with tunnel prefix?) instead of using notifications from the TunnelCommunity. Just an idea. |
|
If we want to be pragmatic and fast, we should follow a concrete plan. So I hereby present: Quinten's 6 month concrete action planI believe the best way to approach this, is starting by refactoring out the current Dispersy walker into a separate module. Then we would write the new walker interface as we would like it to be in the future. If this does not fit the old interface, the old walker would have to be adjusted to keep everything functional. This is the first milestone which can be merged into the repository. At this point the new walker can be created. It would consist (roughly) of the following 3 components, which can be implemented (more-or-less) concurrently:
Once this is done, another milestone would be able to be merged into Dispersy: dropping the old walker and using the new one. At this point backward compatibility will probably be broken (if the peer discovery system uses Protocol Buffers messages already). After this, we should probably finish off with refactoring the requestcache.py, community.py and dispersy.py. Based on existing work and my intuition, this would be the time allocation:
Feedback or alternatives are welcomed. |
|
Just a mental note here: I think that Dispersy communities should adhere to the separation of concerns principle where each community has it's own functionality and responsibilities. The communication between these communities should be as little as possible i.e. they should be able to run in isolation. |
|
Dispersy-identity message: remove and always include full public key. |
|
@devos50 I agree, but as far as I know communities already run separately. Are you referring to some specific instance where communities communicate with each other where they shouldn't? @synctext I completely agree with removing the dispersy-identity/dispersy-missing-identity message stuff. However, note that this requires upgrading all communities to v2 (allchannel,channel,multichain! are using v1), so the old communities would no longer be accessible. If we are going to do this, we may as well use curve25519/ed25519 members everywhere. Then we can remove the M2Crypto dependency. Also a good time to switch to protocol buffers... |
|
@egbertbouman Yes to V2, yes for curve25519, yes to protocol buffers. Cleanup year.. |
|
The first refactoring is in: https://github.com/qstokkink/py-ipv8. |
|
Small update: just now the first bytes were sent to - and received from - a tracker, marking the first backwards compatible communication. Notable changes between IPv8 and Dispersy so far are (1) the removal of the We still have quite a way to go (many messages to convert to the new system) for full on backwards compatibilty though. |
|
Now that I can communicate with Dispersy, I had some fun and captured a little under 10 seconds of peer discovery:
|
|
Some internal imagery for my own convenience: EDIT: In other news, I am on a roll: |
|
Cool update: the basic (no hidden services) TunnelCommunity is integrated and working in IPv8. EDIT: The TrustChain and hidden services extensions are now also in. Excluding attestation logic, IPv8 is now feature complete. Ergo, on a level playing field with our current technology stack (regarding messaging, anonymization and distributed ledger tech.). Whereas the non-hiddenservices TunnelCommunity and Dispersy functionality could be tested over the wild Internet, the TrustChain and hidden services were implemented blind. Therefore, I will now commence writing tests to make sure that these latest additions still work as advertised. |
|
The first test is up and running for the TunnelCommunity 🎉 . Performance is notably better. Doing the following:
Now takes 0.03~0.04 seconds, instead of 20 seconds. So: 500 times faster |
|
making impressive progress.. 40msec, interesting speedup metric. Obviously I'm anxiously awaiting if IPv8 also has an impact on Bittorrent tunnel MBps.. But cleaning, refactoring, and bugfree code is what Tribler needs first. |
|
@synctext I think MBps will go up a little bit due to the cleaner back-end. It won't be as big of a boost as using multiple sockets/multiprocessing though. Also, progress:
|
|
year 2018 toughts.. move whole Libtorrent API into a storage wrapper. becomes part of the IPv8 stack API: trust,identity, secure communication and swarm-based storage. |
|
More progress: TrustChain now also has tests and appears to be working correctly. Now I'll move on to the boring stuff like (1) hooking up a coverage check to see if I am missing lines in my tests, (2) writing a README.md with all the depedencies, (3) writing docstrings and other assorted documentation and (4) run the tests on other platforms. Overall, IPv8 has all of the beta 1 functionality complete:
IPv8 does not (yet) include:
At some point, all of this functionality should probably be shaped into some nice API. |
|
congrats! now the boring part.. Do we need unverified messages in the future? (no idea what they really are..) crypto==NULL and no authentication could be avoided. |
|
@synctext What I mean with verified is that the messages are correctly signed (message public key and message signature match). In the past Dispersy also added replay attack prevention, but this ate disk space and CPU time and wasn't used in the tunnel and TrustChain communities. So I left it out in IPv8 (for now). The TunnelCommunity and the TrustChainCommunity use their own verification mechanism. So, for the more advanced communities unverified messages are also available. Furthermore, the messages are smaller and faster to construct, making them convenient for NAT hole punching. I do agree that this is a potential security risk for first-time community developers. That is why I made verified messages the default option. |
|
Testing update: now on 82% branch coverage, 132 tests, time in tests 5.721 seconds, time with overhead (test suite/script setup and teardown) 14.496 seconds. Moving on to other boring parts. |
|
We are now sporting a README: https://github.com/qstokkink/py-ipv8 UPDATE: Unit tests and coverage run automatically on Jenkins (unix) when I push to the repository: https://jenkins.tribler.org/job/pers/job/test_ipv8_qstokkink/lastSuccessfulBuild/artifact/coverage/index.html |
|
Ok, I also added missing documentation, which completes the feature set for beta 1. Anyone should now be able to make full use of IPv8's functionality. Time for 4 days of vacation 🌦 😎. |
|
As IPv8 is including web-of-trust and authentication features, please read: https://doi.org/10.6028/NIST.SP.800-63-3 |
|
For my own convenience, this is the executive summary of the NIST identity standards:
|
|
Alright, the Weil pairing seems to be functional now (Python produces the same value as Mathematica): |
|
Attestation update, I can now perform the Stanford encryption. Here is some raw output: |
|
Great doc examples:
btw always central list of servers to bootstrap: https://github.com/libp2p/specs/blob/master/4-architecture.md#443-bootstrap-list |
|
After some major optimizations: managed the first 512-bit-primes-key generation. Still some room left for speedup. Encryption/decryption itself is pretty fast. I also created a super secret repository to keep track of the code. Once I'm happy with the code and its performance, I will remove the repo and move the code to the IPv8 repository. |
|
Please read IPv8 related work from IETF: https://tools.ietf.org/html/draft-padma-ideas-problem-statement-03 |
|
Status update: I optimized the code enough that I can generate a key using 1024 bit primes in about 20 seconds. It's still not fast, but its secure and usable. I'll move on to implementing the actual zero-out scheme on top of the cryptosystem now. |
|
I have generic attribute attestation now, here is the output of a full proof of a 64 bit number: I also want to add support for proving ranges (which should be somewhat easy). After that I'm going to do a big code cleanup and then merge it into IPv8. Also, some output of the certainty function: |
|
Excellent progress.. btw fast prime generation bug |
|
Slight change of plans: I did the code cleanup today and I'll integrate this into IPv8 next. I'll leave the range proofs for after the MVP has been released. |
|
Cardinal requirement for Dispersy re-write: Both metadata and content will move as fast as the underlying network permits (see example requirement list ipfs-inactive/faq#47) |
|
Small milestone: code moved from the secret project repository to the IPv8 repository. UPDATE: The community code is now done. Code-wise I only need to publish the attestation hash + metadata to the TrustChain to complete the MVP. After that I will write some tests and the MVP is done. UPDATE2: The secret project repository is no more. In other news, the first unit tests have been created for the cryptographic primitives. |
|
Larger milestone: unit tests are done for the basic attestation community, now I just need a way to advertise attestation hashes and metadata: I was thinking of using TrustChain for this. |
|
Trustchain for transparency and tamper-proofness. |
|
Ok, IPv8 now has all the basic functionality for creating, sharing and verifying generic attestations. I'll switch to creating documentation and turning IPv8 into a standalone service now. |
|
I realized today that I could make the Boneh scheme more secure by not having it depend upon the factorization problem. Long story short; by simply leaving out |
|
IPv8 can now be started and stopped as a service, as per request from @devos50 . I also updated the README with the new attribute attestation functionality. I guess its time to write some documents on how that works now. |
|
I created an artistic impression of the current attestation stack, in table form: |
|
(Even though the backend is fully functional) the IPv8 app/GUI is still a W.I.P., here is the design: |
|
IPv8 now has a REST API: |
|
Image dump of the working Android app with debug data:
|
|
PKI of IPv8 explain in easy storyline by Urbit: |
|
Urbit magic: limit to 4 billion UDs. |
|
I implemented a visual debugger for Twisted calls and Threads, it's fun to see what happens on the inside of IPv8: |
|
More competition in the claim domain: https://eprint.iacr.org/2018/209.pdf |
|
I think this issue has been resolved, IPv8 is up and running. With #3615 Dispersy will be completely gone. |
Remove the double storage principle of Tribler. Dispersy becomes a state-less and clean overlay.
Each Dispersy community is now responsible for their own storage. We now have significant overlap between the tribler and dispersy database. Move much of this responsibility in several iterations upwards. Downwards we move the low-level connectivity stuff. The walker becomes responsible for storing all public keys, interaction records, IP addresses, etc.
See: Tribler/dispersy#61
We have several issues to reach
dispersy miniand refactor 14000 Dispersy lines. Braindump :Roadmap note: instead of tweaking this layer for years and having fun, we need to be pragmatic. So no 10 month refactoring job and 7 months of nasty bug hunting. Clean-up and move on to real productive work within 6 months!
Good luck!
The text was updated successfully, but these errors were encountered: