2019 Q1 OKRs Planning

[jacobheun]

Q1 is around the corner! OKR presentations are scheduled for the 20th, so we need to get our list together before then.

• 2018 Q4 Retrospective Document.
• Previous OKR Planning for reference
• 2019 Q1 OKR Spreadsheet

cc: @mgoelzer

[vasco-santos]

After thinking about what we should tackle in the next quarter, as well as the discussions regarding the libp2p roadmap, here follows the objectives that I think we should target next.

Q1 Objectives

Connectivity Magic

• Gossipsub

Taking into consideration that the floodsub implementation does not scale in really large networks and that the real-time requirement for applications is essential nowadays, we should guarantee that we provide the gossipsub implementation soon in JS land. This is one of the features that other teams (DDC WG for instance) are really waiting for and I would also really like to implement it.

• DHT search stream

One of the requests from people currently building dapps is to have a streaming API for the DHT and IPNS. go-ipfs currently provides it through the go-libp2p-kad-dht SearchValue. Implementing this would provide tremendous benefits for developers, as they will be able to receive the data obtained in the DHT in stream, instead of looking for several occurences in the DHT before providing the data.

References:

libp2p/go-libp2p-kad-dht#174

• Improve DHT scalability

We have been working along @jhiesey in the last months, in order to improve the current DHT implementation and we have made some progress. I think this topic is essential for the growth of libp2p, however we do not have a concrete plan on how we will tackle this yet. I would like that we have a OKR for improving the DHT scalability and performance, but probably will be one of the things to pass for Q2.

• NAT traversal

We have been blocked on implementing NAT traversal in js-libp2p, while we do not have a complete spec. I think this should have a higher priority this quarter and we should try to have the spec ready asap and have it implemented in JS land.

• QUIC transport

With QUIC advantages regarding performance and security, I think it should also be a focus for us.

References:

#287

• WebBluetooth integration

I think that the use cases where having a bluetooth discovery and transport are really great in the offline first goals and we should start working towards this.

As it is still only available for browsers in chrome for windows, we do not know how far we are from having it available in all the most used browsers across all devices. However, I think it would be greate to to incubate a PoC using WebBluetooth and libp2p, in order to pave the way to have offer it in a near future as a stable and ready to use protocol.

References:

#261

Improve Debuggability and Stability

• Full interoperability with go-libp2p exists

After we started working in interop during 2018 Q4 for the core functionallity, we should target next to have full interop regarding all the building blocks of libp2p and guarantee that the interop tests are now a requirement in the implementation of new features in js-libp2p.

With this, the interop tests must be included in the release process of both js-libp2p and go-libp2p, that is, we must have the interop tests good in order to move forward with a new release.

• Testlab

With the endeavour of both js and go teams with the libp2p testbed, which I believe to have been good for both sides, I think we should continue collaborating. Accordingly, one of the next steps that we have been discussing for the testbed is the creation of a testlab. We should be able to easily deploy large networks for testing, debugging and performance purposes.

• libp2p has a network visualization tool (with plugin for testlab)

This tool did not have any relevant progress last quarter, but it is an important aspect of debuggability for a complex network. Moreover, it may also have potential to be used as a way for people to understand how our subsystems work graphically. It should be also available in the Testlab as a plugin.

• libp2p has a benchmark tool / web app (plugin for testlab)

We have done some groundwork on DHT stress testing this quarter, but we need to go further on it, as well as go over the all libp2p subsystems. With a testlab available, one of the most important information that we want to infer is the performance of the libp2p implementations. The testlab should export metrics that would allow us to have a benchmark tool to visualize the performance of the libp2p subsystems.

One interesting process that we could take advantage from, would be to have the benchmarks for each libp2p release. This way, we should work on keeping / improving performance with each release, guaranteeing that we do not introduce performance problems in newer releases.

Libp2p is seen as the go to p2p networking stack

• Revamped API and improved development experience

Revamping APIs in the context of #266 will allow us to provide a more modern and ergonomic API for developers, as well as to improve code readability and maintanability. Moreover, we will be able to remove some dependencies of that we currently have, decreasing the bundle size of js-libp2p. Finally, we will be able to improve the error handling and documentation during this work.

• Usage examples and proper documentation exist for all the modules of libp2p

We started working on this in 2018 Q4, with the main modules of libp2p, but now we should target the all JS modules of libp2p, as each one of this modules may be used by itself.

Partner use cases

• Add support for TCP hole punching

Context: agoric-labs/PlaygroundVat#8

Overview

This list is a subset of the things that I believe to be important for the next steps of js-libp2p. They are not sorted by priority and I think that we will not probably be able to do all of this in the next quarter. However, they are detailed here so that we can decide what we should tackle and also provide an opportunity for the community to give their input.

The majority of the work described in the Improve Debuggability and Stability is implementation agnostic and is the type of work that would be really benefitial for the libp2p team to have collaboration between the teams, in order to create better tools and have everybody on the same page.

[jacobheun]

Based on the 2019 roadmap and community needs, here's a list of what I think are the most important items to tackle in Q1.

Q1 Objectives

✨Connectivity Magic

Rendezvous protocol

Getting the rendezvous protocol working in go and js would be a huge win. Being able to do discovery of specific node/applications types would greatly improve the networks ability to intelligently find peers. In some of the js DApps, pubsub is being used to find peers that are interested in the same topic. Having the rendezvous protocol would allow peers to find one another much more efficiently. This would also enable nodes to find and register as certain node types, such as circuit relays.

References:

• RFC: rendezvous specs#56

Connection Manager Dial Backoff

As mentioned in https://github.com/libp2p/js-libp2p-connection-manager/issues/15, js-libp2p doesn't handle any dial backoff to peers. This can result in dial attempts being spammed to a given node. Currently it's up to each implementation of libp2p to handle this at the application level (reference: See Agoric "Thundering Herd" issue). This could help prevent a large number of unnecessary or ineffective dials on the network.

References:

• https://github.com/libp2p/js-libp2p-connection-manager/issues/15

🚗 Nat Traversal

Nat Traversal is a request we see come up pretty regularly. Often users are asking for options when using ec2 instances. I think it would be beneficial to finish up the initial implementation of the nat manager, without the dialme protocol, and create a working example of deploying a libp2p node on ec2. An example running behind an Elastic IP would be a huge help for developers.

Secondarily, I think the dialme protocol would be a great improvement for connectivity of nodes. Allowing for nodes with public addresses to tell other nodes connected over relays, like browser nodes, to dial them directly at one of their public addresses could greatly improve performance of the whole network. Circuit Relay nodes could potentially see a large reduction in long term connections.

References:

• DialMe Protocol: RFC: dialMe protocol specs#64
• NAT Traversal #104

⏩ Improved Performance

multistream 2.0

Efficient connection handshaking is a critical part of making libp2p highly performant. As our list of protocols grow, the ping pong of multistream 1.0 will become increasingly inefficient. Moving to multistream 2.0 should help us improve connection latency and reduce round trips needed for handshaking.

References:

• [WIP] [RFC] Multistream-2.0 specs#95

Gossipsub

Scalability is a big goal of IPFS and libp2p for 2019. Gossipsub over Floodsub is a good step in improving our ability to scale. As this is already implemented in go, we should get that done in js.

⛰ Improved Debuggability and Stability

go and js interop testing

As libp2p continues to move towards spec driven development, it's going to be important to have a place for the various implementations to validate their interoperability against the spec. The interop repo has been mostly untouched. With the creation of the go and js daemons, we'll be in a great place to make significant additions to our interop testing.

It would be great to get to a point where we are doing nightly runs of the interop tests with the latest dependencies to ensure that we catch any issues early and often.

References:

• https://github.com/libp2p/interop

Testbed is scaled to 50000 nodes

This is a continuation from Q4 2018. We're getting closer to creating our libp2p clusters and need to be able to scale them to 50k nodes and beyond. This is going to be critical for the IPFS 2019 goal of millions of nodes.

Benchmark testing

As we work on scaling libp2p with the testbed, it's going to be increasingly important to run regular benchmark tests to determine where our weaknesses are and ensure that we are continuing to improve performance.

🥇 Libp2p is the go to network stack

(+1 from @vasco-santos's list)

Revamped API and improved development experience

Usage examples and proper documentation exist for all the modules of libp2p

[jacobheun]

@vasco-santos I will get these itemized and into the readme so we can better discuss them.

[jacobheun]

I reworded a few things and the Partner Use Case: TCP Hole Punching has been incorporated into the Nat Traversal item.

[jacobheun]

I don't think we'll be able to get the transport completed, but I think we should work to support the QUIC implementation in JS. Whether that's via contributing or testing via early adoption. Concurrently developing this while development continues could be a feasible approach, but it will likely continue into Q2.

[vasco-santos]

Considering that there some discussions around implementing QUIC as a native node module and all our different front lines, I agree that QUIC should be one of the OKRs that should fall to Q2.

[jacobheun]

I think we should define what we want to have tested. I don't think having "full" interop testing is going to be feasible in Q1, but I think we can at least tackle the core components and figure out our process for adding interop tests for new things, like the Rendezvous Protocol.

[vasco-santos]

Agree! DO you have a list suggestion?

[jacobheun]

Suggested change

	* go and js are interoperable
	* go and js are interoperable
	* The dht has a full, passing suite of interop tests
	* Circuit Relay has a full, passing suite of interop tests
	* TCP and Websocket transports have full, passing suites of interop tests over SECIO and mplex

Here are my suggestions. We additionally have gossipsub, but it's covered by it's own OKR. I think floodsub is a nice to have, but with our current OKR list I think these are the higher priorities.

[vasco-santos]

This list seems really awesome! 🚀

I agree that we probably should drop QUIC for the reason that I left in the comment. Other than that, we have a really ambitious OKR Planning, but if we work together with the go-libp2p team in the Improve Debuggability and Stability section, I think we can have a boost

[vasco-santos]

For improving the DHT scalability we need @jhiesey feedback. @jhiesey do you have any idea of how long it will take to get the spec for the new DHT?

[jacobheun]

I've removed the QUIC transport since we will push that to Q2. I've also added my thoughts on priorities so we can discuss those.

[jacobheun]

OKRs have been moved to the spreadsheet, with assignments, and the reference to that has been added to the OKR readme.