Synapse: Add a step for running Complement tests in worker mode #122
Conversation
anoadragon453
changed the title
anoadragon453
requested review from
a team and
kegsay
and removed request for
a team
anoadragon453
force-pushed
the
anoa/synapse_pipeline_workers_complement
branch
from
0e0ddcc
to
3029ae5
Compare
anoadragon453
force-pushed
the
anoa/synapse_pipeline_workers_complement
branch
from
3029ae5
to
36793d8
Compare
synapse/pipeline.yml
Outdated
| # Complement needs to know if it is running under CI | ||
| - "CI=true" |
There was a problem hiding this comment.
I'm planning to change that myself once these PRs are in :)
synapse/pipeline.yml
Outdated
| # Enabled Complement's certificate authority for authentication of | ||
| # federation requests | ||
| - "COMPLEMENT_CA=true" |
There was a problem hiding this comment.
Possibly not, though existing images expecting to fill the /ca directory with their own files (for example, Synapse Monolith's) will need to be updated if this option is on by default.
I've created an issue for discussion on the Complement repo here: matrix-org/complement#65
synapse/pipeline.yml
Outdated
| - command: | ||
| # Build a docker image from the checked out Synapse source | ||
| - "docker build -t matrixdotorg/synapse:latest -f docker/Dockerfile ." | ||
| # Build a second docker image on top of the above image. This one sets up Synapse workers | ||
| # as well as everything else it needs to run a federating setup | ||
| - "docker build -t matrixdotorg/synapse:workers -f docker/Dockerfile-workers ." | ||
| # We use the complement:latest image to provide Complement's dependencies, but want | ||
| # to actually run against the master branch of Complement, so download it here. | ||
| - "wget https://github.com/matrix-org/complement/archive/master.tar.gz" | ||
| - "tar -xzf master.tar.gz" | ||
| # Build an image for running the above worker setup in Complement. This involves | ||
| # disabling rate-limiting, using Complement's CA etc. | ||
| - "docker build -t complement-synapse -f complement-master/dockerfiles/SynapseWorkers.Dockerfile complement-master/dockerfiles" | ||
| # Finally, compile and run the tests. | ||
| - "cd complement-master" | ||
| - "COMPLEMENT_BASE_IMAGE=complement-synapse:latest go test -v -tags synapse_blacklist ./tests" | ||
| label: "\U0001F9EA Complement | Synapse Workers" | ||
| agents: | ||
| # Running every worker takes a beefy system | ||
| queue: "xlarge" | ||
| plugins: | ||
| - docker#v3.7.0: | ||
| # The dockerfile for this image is at https://github.com/matrix-org/complement/blob/master/dockerfiles/ComplementCIBuildkite.Dockerfile. | ||
| image: "matrixdotorg/complement:latest" | ||
| mount-buildkite-agent: false | ||
| environment: | ||
| # Complement needs to know if it is running under CI | ||
| - "CI=true" | ||
| # Enable debug logging | ||
| - "COMPLEMENT_DEBUG=1" | ||
| # Enabled Complement's certificate authority for authentication of | ||
| # federation requests | ||
| - "COMPLEMENT_CA=true" | ||
| # Starting up all the processes in this container can take a little while. | ||
| # The default is 200. Let's wait a bit longer to decrease the chance of a | ||
| # startup timeout | ||
| - "COMPLEMENT_VERSION_CHECK_ITERATIONS=500" | ||
| publish: [ "8448:8448" ] | ||
| # Complement uses Docker so pass through the docker socket. This means Complement shares | ||
| # the hosts Docker. | ||
| volumes: | ||
| - "/var/run/docker.sock:/var/run/docker.sock" |
There was a problem hiding this comment.
It would be nice to make use of yaml entities so that the bits that are shared between the complement runs (the docker-plugin settings, and some of the commands) don't have to be C&Ped.
There was a problem hiding this comment.
I was able to reduce the plugins section quite significantly, but didn't see as clear a path with commands: as there's some differences and commonalities interspersed with one another.
There was a problem hiding this comment.
looks the same apart from the name of the dockerfile? so have two yaml entities, one which does the bits before that, and one which does the bits after?
or even: stick the name of a dockerfile in an env var and set it somewhere else in the step config?
There was a problem hiding this comment.
It seems as though environment variables specified in environment aren't used when launching commands specified in commands. The environment variables that seem to work there are those found in docker-compose-env. But that doesn't really help us much.
Additionally move the COMPLEMENT_IMAGE env var to the environment: section.
anoadragon453
force-pushed
the
anoa/synapse_pipeline_workers_complement
branch
from
d41ddf5
to
c6a4e8a
Compare
Synapse's pipeline shouldn't really have this turned on.
This reverts commit 6998c3e.
…napse_pipeline_workers_complement
Mostly so we don't have to remember to update the -tags in both commands
Also remove redundant :latest tag for base Synapse images.
|
closing this, because synapse has moved its CI to GHA. |
This PR extends the Complement section of Synapse's pipeline to run Complement tests under worker mode. This PR is ready for review, but should not be merged until the relevant Synapse and Complement PRs are.
How does this work?
We start with the official Synapse docker image. This creates a python environment with all of Synapse's dependencies installed. Then we make use of the new
Dockerfile-workersto install supervisord, redis and nginx. Redis is used for inter-process communication, nginx is used for routing network requests to each worker and supervisord will start and maintain each process.Finally, we use the
SynapseWorkersdockerfile in Complement to install postgresql into the container, set up caddy server to easily take advantage of Complement's CA, inject some Synapse options to disable rate-limiting, enable registration etc. and then finally run a script which generates config files supervisord, nginx and Synapse depending on which workers you want to run.