Interoperability testing for libp2p

[daviddias]

libp2p is becoming a fundamental piece for many more projects than just IPFS and with that, it requires to be handled with more mature open source approaches in the same way that we treat IPFS (e.g docs, examples, demos, tutorials, testing, etc). Some of which was developed alongside it, namely testing, docs and tutorials, but interoperability and compliance testing has been only done by testing IPFS directly, which in part tests libp2p. We need to improve that.

Today, there are 3 libp2p implementations:

• https://github.com/libp2p/go-libp2p
• https://github.com/libp2p/js-libp2p
• https://github.com/paritytech/rust-libp2p (soon to be moved to the libp2p org as well)

In order to build a set of robust set of interoperability and compliance tests, I'm proposing two approaches:

• Create the libp2p service (aka daemon) which would let us interact with libp2p nodes in the same way that we IPFS nodes and in which we build interop tests -- https://github.com/ipfs/js-ipfs/tree/master/test/interop.
• Create PDD tests. Tests designed as stories that libp2p language implementers can follow and create a program that does exactly what the story describes. These stories always have more than one node interacting, so that different implementations can be used against each other.

The libp2p service (aka daemon)

We have had some discussions about building the libp2p service. I believe that in order to motivate the creation of a first iteration, we need a simple utility that uses the libp2p service and is really well scoped. The idea in mind is creating a p2pcat for libp2p.

I know that we want to avoid arriving at the same situation that IPFS and its HTTP-API are, it is hard to test, document, create compliance tests and that forces everyone to implement manually their RPC clients for every single language. There has to be a better way.

After looking into multiple RPC libraries, I also started thinking that I wanted to avoid any RPC swiss army knife and focus on the core of an RPC library, something that enables a user to do remote procedure calls and that's it.

I'll continue the thread on research for the better RPC library for libp2p in a separate thread. The design must fulfil the goal of a) not take us into the same issues we have had with the HTTP-API of IPFS, b) enable us to generate docs, tests, and clients automatically and c) be mountable over any libp2p stream so that libp2p users can also have access to a great RPC library for their applications.

All this to mean that yes, this is work we want to do and will enable us to create interoperable tests, however, in order to do it right it will require some more time and so it is better to follow the second proposal first.

Update: I've added issues on libp2p/libp2p for

• libp2p service (aka daemon) libp2p service (aka Daemon) libp2p#22
• simple-rpc simple-rpc libp2p#23

PDD - Protocol Driven Development

PDD is a technique to test implementations of protocols through stories generated from their specification. These stories describe a test case and if the implementation is correct, a test can use two or more implementations to take roles in the story without causing any failure.

The core principle is that the truth lies in the wire and if one implementation in language Y can understand the input generated by language X and generate the same output, then the implementation is compliant.

Some of these tests can be done asynchronously, this means that the output and input can be captured and frozen for later comparison, however, some of these tests require to be synchronous, in which both implementations must run at the same time because there are some time variants which the protocol doesn't need to specify (i.e order of messages in a stream protocol or any crypto channel).

I'll be creating these stories for libp2p during this week focusing on:

• Qualitative Tests - Also known as compliance tests, ensuring that the 3 implementations have tha same functionality
• Quantitative Tests - Also known as stress tests, ensuring that more than just working as expected for one case, the implementation is stable when it has to handle multiple cases at the same time.

[daviddias]

@lgierth @whyrusleeping would love your thoughts too on this thread :) I started this PDD Stories at #2

[daviddias]

Update

I've worked on getting the first set of PDD Stories out of the door -- #2 -- and implemented 4 of them in JS -- libp2p/js-libp2p#137 --.

With this, I learned that it is definitely useful to have scripts (the stories) to help us write the narrative of what the tests should do, however, implementing them manually is even more tedious than what I expected and takes a considerable amount of time.

I decided to go this route initially mainly because we want to have some libp2p interop tests out of the door ASAP for rust-libp2p, js-libp2p, and go-libp2p. I was concerned that getting the libp2p service/daemon would take a very long time to get done since it will require a ton of API design and potential 🚲🏠. However, I don't see this as being a good short term solution right now.

We desperately need the libp2p service/daemon to be capable of writing these tests once and using them against all the languages.

@lgierth has shared some good notes about its design at libp2p/libp2p#22 (comment)

[victorb]

ton of API design and potential

I don't think "a ton" is necessary to have something valuable out of the door quickly. As far as I understand, the only methods needed for an API to be able to test things (albeit basic), are:

• .createNode / .deleteNode
• .startNode / .stopNode
• .listen / .unlisten
• .dial
• .handle

Then we should be able to create a base layer of tests at least.

[paralin]

I'm beginning to use libp2p as the core of a couple of my projects, and I'm realizing more and more how necessary it is to create a daemon for libp2p, particularly due to the fact that every libp2p application at the moment seems to have to have its own node running with duplicate listeners and outgoing connections for everything, which is obviously a waste of resources.

I'm happy to assist in the effort of building a core LibP2P node/daemon which can act as a common shared pool of connected peers / transports which other apps can connect to and use as a sort of "bridge" into the network. Of course it makes sense to also sometimes bundle libp2p into an application, but when more than one thing uses libp2p on a machine, having a common node makes sense.

To me it makes sense to have the daemon eventually manage (among others):

• Local list of listeners / transports, provide interfaces for updating these lists without restart
• "Host" and "Swarm" interfaces networked over some sort of transport to the other apps on the system

[vasco-santos]

Closing this issue as we now have a set of interop tests in this repo