Architecture to add libp2p tests

[Elexy]

Question that came out is they want to write benchmarks against libp2p. We can clone the repo and infrastructure for libp2p or add the login in the runner to use the libp2p or js-ipfs repo. The same dashboard could be used and filter by project.
@e_lexy @Matteo ^^^^^ when you get a chance thoughts on the above.

matteo [5:12 PM]
@e_lexy you do the architecture for that solution? Sharing things might be good.. up to deploying a double.
What’d be the cleanest way?
Can you assemble a proposal on an gdoc and I’ll review?

[vasco-santos]

Any update on this @Elexy @mcollina ?

[Elexy]

Working on this today.

[Elexy]

@vasco-santos I've added a branch with a design diagram.
https://github.com/ipfs/benchmarks/blob/add_libp2p_arch/runner/Libp2p%20Runner.png
The key differentiator would be the provisioning.
This takes care of setting up the right libs at the right version on the Minion.

[Elexy]

@vasco-santos I've made a design diagram

The main difference is the Provisioning. This is where we make sure the minion has the right libs at the right version before we run the benchmarks.

[vasco-santos]

Thanks for the proposal @Elexy !

So, in this approach we need to use this repo for all the tests and mark each test as being part of a project?

[Elexy]

Hi @vasco-santos
The way we get the library in the test bench is quite specific to this setup.
https://github.com/ipfs/benchmarks/blob/master/tests/getIpfs.sh
The config.js also relies on paths defined by ENV vars (or defaults) for this project.

We could generalize is even more and make it a generic runner, able to run arbitrary scripts and retrieve the output, but I'd say we'd probably be building a full blow CI system.

[vasco-santos]

I understand that generalize this solution more, at least at this moment seems not the best approach.

I would like to have the libp2p benchmark tests in its own repo (libp2p/benchmarks). Moreover, I think it would be ok if we use this repo and infrastructure for both, and it gets the libp2p/benchmark tests from there. What do you think?

[Elexy]

@vasco-santos that makes sense.
@litzenberger should we lead the way by splitting the js-ipfs tests off of this repo?

[litzenberger]

@Elexy I think the js-ipfs tests in the repo are fine, he just wants to pull another repo that will need to be synced by the runner. @vasco-santos did I read that right?

[vasco-santos]

Yes, we should be able to get tests from other repos, such as libp2p/benchmarks repo, and in the future probably ipld/benchmarks as well.

[Elexy]

Currently we are rsyncing the tests over to the minion from the controller from this repo.In the new situation I'd need to run a git clone of a specified repo before we can run the tests run the tests.

[Elexy]

As discussed in the latest sync call I want to present a proposal to allow the runner to run benchmark tests from a different repository.
The proposal is here.
/cc @vasco-santos @alanshaw @daviddias

[vasco-santos]

Thanks for the proposal @Elexy !