Substra Orchestrator

Substra is an open source federated learning (FL) software. This specific repository contains the logic to orchestrate Substra assets.

Mission statement

This component's purpose is to orchestrate task processing in multiple channels of Substra partners:

• it is the single source of truth of Substra organizations;
• it exposes necessary data to Substra instances to process their tasks and register their assets;
• its API is aimed to serve backends, not end-users;
• it enforces that all registered data are valid;
• it ensures data consistency under multiple concurrent requests;

Building the orchestrator

Dev tools versions

Make sure you have these requirements fulfilled before trying to build the orchestrator:

• go: v1.21+
• protoc: v3.18.0
• proto-gen-go: v1.28.0
• golang-migrate: optional, used to create migration files
• skaffold: used to run the orchestrator locally
• mockery: used to generate mocks
• goimports: used for formatting

Build

make

Run tests

make test

Before running e2e tests, you may need to generate and retrieve a client certificate.

./examples/tools/download_client_cert.sh

End-to-end testing requires a running orchestrator. Assuming you have one up and ready on orchestrator.org-1.com port 443, here is how to launch the tests:

go test -tags=e2e ./e2e -short -tls \
    -cafile ../examples/tools/ca.crt \
    -keyfile ../examples/tools/client-org-1.key \
    -certfile ../examples/tools/client-org-1.crt \
    -server_addr orchestrator.org-1.com:443

Refer to go test -tags=e2e ./e2e -args --help for more options.

Developing the orchestrator

An overview of the code structure is available in the docs directory and there is also a documentation of the assets. If you are interested in adding a new asset there is a step by step documentation on this subject.

A good entry point to get an overview of the codebase is to launch godoc -http=:6060 and open module documentation.

If you want to run the orchestrator with Skaffold you will need to add the jetstack and bitnami helm repo:

helm repo add jetstack https://charts.jetstack.io
helm repo add bitnami https://charts.bitnami.com/bitnami

Standalone mode

When running in standalone mode, the orchestrator needs a postgres database to persist its data.

To launch the orchestrator:

skaffold dev

or

skaffold run

Assuming orchestrator.org-1.com is pointing to your local k8s cluster IP (edit your /etc/hosts file for that), the following command should list available services:

grpcurl -insecure orchestrator.org-1.com:443 list

You can also deploy substra-backend with a skaffold dev or skaffold run

Testing

You can call the local orchestrator gRPC endpoint using evans

Before launching Evans you may need to generate and retrieve a client certificate.

./examples/tools/download_client_cert.sh

evans --tls --cacert examples/tools/ca.crt --host orchestrator.org-1.com -p 443 -r repl --cert examples/tools/client-org-1.crt --certkey examples/tools/client-org-1.key

Then you can launch call like this:

package orchestrator
service OrganizationService
header mspid=MyOrg1MSP channel=mychannel
call GetAllOrganizations

or the one-liner

echo '{}' | evans \
    --host orchestrator.org-1.com -p 443  \
    --tls \
    --cacert examples/tools/ca.crt \
    --cert examples/tools/client-org-1.crt \
    --certkey examples/tools/client-org-1.key \
    -r cli \
    --header 'mspid=MyOrg1MSP' --header 'channel=mychannel' \
    call orchestrator.OrganizationService.RegisterOrganization

Note that you need your ingress manager to support SSL passthrough (--enable-ssl-passthrough with nginx-ingress).

For additional development tips, please refer to the documentation.

How to generate the changelog

The changelog is managed with towncrier, a Python tool. To add a new entry in the changelog, add a file in the changes folder. The file name should have the following structure: <unique_id>.<change_type>. The unique_id is a unique identifier, we currently use the PR number. The change_type can be of the following types: added, changed, removed, fixed.

To generate the changelog (for example during a release), you need to have towncrier installed. You can either install it in a virtual env, or use pipx (please refer to pipx documentation for installation instructions).

$ pipx install towncrier

Then use the following command :

$ towncrier build --version=<x.y.z>

You can use the --draft option to see what would be generated without actually writing to the changelog (and without removing the fragments).