Testing with Complement has gotten so slow - 2 minutes to see test results

[matrixbot]

This issue has been migrated from #13204.

---

I used to be able to make a change in Synapse and see Complement results after 20 seconds. Now it takes 1 minute 15 seconds just to build the Docker images at the start. And over 2 minutes to see the test results for running a single Complement test. This makes local dev such a pain

Discussed at https://matrix.to/#/!alCakyySsFIAVfZLDL:matrix.org/$HuKnVFoLYs_uiDNbENnYs_ALqgQKjbhDrm3wbYYkLwI?via=matrix.org&via=element.io&via=termina.org.uk

Docker build is too slow

When I'm just doing feature work, I don't care about workers, I don't care about Postgres (whatever is new and making it slow).

1 minute 15 seconds of Docker building after changing Synapse:

Complement run ./scripts-dev/complement.sh

$ COMPLEMENT_ALWAYS_PRINT_SERVER_LOGS=1 COMPLEMENT_DIR=../complement ./scripts-dev/complement.sh -run TestJumpToDateEndpoint/parallel/federation/can_get_pagination_token_after_getting_remote_event_from_timestamp_to_event_endpoint
[+] Building 22.0s (25/25) FINISHED
 => [internal] load build definition from Dockerfile                                                                                                                                                                   0.0s
 => => transferring dockerfile: 5.62kB                                                                                                                                                                                 0.0s
 => [internal] load .dockerignore                                                                                                                                                                                      0.0s
 => => transferring context: 35B                                                                                                                                                                                       0.0s
 => resolve image config for docker.io/docker/dockerfile:1                                                                                                                                                             0.3s
 => CACHED docker-image://docker.io/docker/dockerfile:1@sha256:443aab4ca21183e069e7d8b2dc68006594f40bddf1b15bbd83f5137bd93e80e2                                                                                        0.0s
 => [internal] load build definition from Dockerfile                                                                                                                                                                   0.0s
 => [internal] load .dockerignore                                                                                                                                                                                      0.0s
 => [internal] load metadata for docker.io/library/python:3.9-slim                                                                                                                                                     0.3s
 => [internal] load build context                                                                                                                                                                                      0.3s
 => => transferring context: 7.35MB                                                                                                                                                                                    0.3s
 => [requirements 1/6] FROM docker.io/library/python:3.9-slim@sha256:c01a2db78654c1923da84aa41b829f6162011e3a75db255c24ea16fa2ad563a0                                                                                  0.0s
 => CACHED [builder 2/7] RUN    --mount=type=cache,target=/var/cache/apt,sharing=locked    --mount=type=cache,target=/var/lib/apt,sharing=locked  apt-get update -qq && apt-get install -yqq     build-essential       0.0s
 => CACHED [requirements 2/6] RUN    --mount=type=cache,target=/var/cache/apt,sharing=locked    --mount=type=cache,target=/var/lib/apt,sharing=locked  apt-get update -qq && apt-get install -yqq git     && rm -rf /  0.0s
 => CACHED [requirements 3/6] RUN --mount=type=cache,target=/root/.cache/pip   pip install --user "poetry-core==1.1.0a7" "git+https://github.com/python-poetry/poetry.git@fb13b3a676f476177f7937ffa480ee5cff9a90a5"    0.0s
 => CACHED [requirements 4/6] WORKDIR /synapse                                                                                                                                                                         0.0s
 => CACHED [requirements 5/6] COPY pyproject.toml poetry.lock /synapse/                                                                                                                                                0.0s
 => CACHED [requirements 6/6] RUN /root/.local/bin/poetry export --extras all -o /synapse/requirements.txt ${TEST_ONLY_SKIP_DEP_HASH_VERIFICATION:+--without-hashes}                                                   0.0s
 => CACHED [builder 3/7] COPY --from=requirements /synapse/requirements.txt /synapse/                                                                                                                                  0.0s
 => CACHED [builder 4/7] RUN --mount=type=cache,target=/root/.cache/pip   pip install --prefix="/install" --no-deps --no-warn-script-location -r /synapse/requirements.txt                                             0.0s
 => [builder 5/7] COPY synapse /synapse/synapse/                                                                                                                                                                       0.5s
 => [builder 6/7] COPY pyproject.toml README.rst /synapse/                                                                                                                                                             0.0s
 => [builder 7/7] RUN pip install --prefix="/install" --no-deps --no-warn-script-location /synapse                                                                                                                     8.8s
 => CACHED [stage-2 2/5] RUN    --mount=type=cache,target=/var/cache/apt,sharing=locked    --mount=type=cache,target=/var/lib/apt,sharing=locked   apt-get update -qq && apt-get install -yqq     curl     gosu     l  0.0s
 => [stage-2 3/5] COPY --from=builder /install /usr/local                                                                                                                                                              2.6s
 => [stage-2 4/5] COPY ./docker/start.py /start.py                                                                                                                                                                     0.1s
 => [stage-2 5/5] COPY ./docker/conf /conf                                                                                                                                                                             0.1s
 => exporting to image                                                                                                                                                                                                 2.5s
 => => exporting layers                                                                                                                                                                                                2.5s
 => => writing image sha256:cf22ee98c4023602eaa86fe9b09d8043f4f01138f2aec7c745387444d0d45249                                                                                                                           0.0s
 => => naming to docker.io/matrixdotorg/synapse                                                                                                                                                                        0.0s

Use 'docker scan' to run Snyk tests against images to find vulnerabilities and learn how to fix them
[+] Building 15.3s (12/12) FINISHED
 => [internal] load build definition from Dockerfile-workers                                                                                                                                                           0.0s
 => => transferring dockerfile: 1.45kB                                                                                                                                                                                 0.0s
 => [internal] load .dockerignore                                                                                                                                                                                      0.0s
 => => transferring context: 35B                                                                                                                                                                                       0.0s
 => [internal] load metadata for docker.io/matrixdotorg/synapse:latest                                                                                                                                                 0.0s
 => [internal] load build context                                                                                                                                                                                      0.1s
 => => transferring context: 30.16kB                                                                                                                                                                                   0.0s
 => [stage-0 1/7] FROM docker.io/matrixdotorg/synapse:latest                                                                                                                                                           0.6s
 => [stage-0 2/7] RUN    --mount=type=cache,target=/var/cache/apt,sharing=locked    --mount=type=cache,target=/var/lib/apt,sharing=locked   apt-get update -qq &&   DEBIAN_FRONTEND=noninteractive apt-get install -y  8.5s
 => [stage-0 3/7] RUN --mount=type=cache,target=/root/.cache/pip     pip install supervisor~=4.2                                                                                                                       4.9s
 => [stage-0 4/7] RUN rm /etc/nginx/sites-enabled/default                                                                                                                                                              0.4s
 => [stage-0 5/7] COPY ./docker/conf-workers/* /conf/                                                                                                                                                                  0.0s
 => [stage-0 6/7] COPY ./docker/prefix-log /usr/local/bin/                                                                                                                                                             0.1s
 => [stage-0 7/7] COPY ./docker/configure_workers_and_start.py /configure_workers_and_start.py                                                                                                                         0.0s
 => exporting to image                                                                                                                                                                                                 0.4s
 => => exporting layers                                                                                                                                                                                                0.4s
 => => writing image sha256:0b0ae26a584d7ae58927a2daba3b508d61245e1a41d4ac9ba5b92371a96d5445                                                                                                                           0.0s
 => => naming to docker.io/matrixdotorg/synapse-workers                                                                                                                                                                0.0s

Use 'docker scan' to run Snyk tests against images to find vulnerabilities and learn how to fix them
[+] Building 42.5s (13/13) FINISHED
 => [internal] load build definition from Dockerfile                                                                                                                                                                   0.0s
 => => transferring dockerfile: 37B                                                                                                                                                                                    0.0s
 => [internal] load .dockerignore                                                                                                                                                                                      0.0s
 => => transferring context: 2B                                                                                                                                                                                        0.0s
 => [internal] load metadata for docker.io/matrixdotorg/synapse-workers:latest                                                                                                                                         0.0s
 => [internal] load build context                                                                                                                                                                                      0.0s
 => => transferring context: 177B                                                                                                                                                                                      0.0s
 => [1/8] FROM docker.io/matrixdotorg/synapse-workers:latest                                                                                                                                                           0.2s
 => [2/8] RUN apt-get update &&   DEBIAN_FRONTEND=noninteractive apt-get install --no-install-recommends -yqq postgresql-13                                                                                           34.7s
 => [3/8] RUN pg_ctlcluster 13 main start &&  su postgres -c "echo  "ALTER USER postgres PASSWORD 'somesecret';  CREATE DATABASE synapse   ENCODING 'UTF8'   LC_COLLATE='C'   LC_CTYPE='C'   template=template0;" | p  3.8s
 => [4/8] RUN mv /conf/shared.yaml.j2 /conf/shared-orig.yaml.j2                                                                                                                                                        0.4s
 => [5/8] COPY conf/workers-shared-extra.yaml.j2 /conf/shared.yaml.j2                                                                                                                                                  0.1s
 => [6/8] WORKDIR /data                                                                                                                                                                                                0.1s
 => [7/8] COPY conf/postgres.supervisord.conf /etc/supervisor/conf.d/postgres.conf                                                                                                                                     0.1s
 => [8/8] COPY conf/start_for_complement.sh /                                                                                                                                                                          0.1s
 => exporting to image                                                                                                                                                                                                 2.9s
 => => exporting layers                                                                                                                                                                                                2.9s
 => => writing image sha256:df7d7a1d4443960b49dbb310bdbd4af323c33c7900d26bf291c3fad894683453                                                                                                                           0.0s
 => => naming to docker.io/library/complement-synapse                                                                                                                                                                  0.0s

Use 'docker scan' to run Snyk tests against images to find vulnerabilities and learn how to fix them
Images built; running complement
...

Relevant code:
https://github.com/matrix-org/synapse/blob/dcc7873700da4a818e84c44c6190525d39a854cb/scripts-dev/complement.sh#L89-L107

---

With cache, the Synapse build used to take ~10s seconds, matrix-org/synapse#9610, which has now doubled to ~20 seconds. And the rest was trivial.

Complement being slow

@kegsay pointed out these couple issues which would also slow down single test runs (stopping and shutting down containers):

• Gracefully shut down container when we want logs matrix-org/complement#392
• Gracefully stop blueprint containers before committing them matrix-org/complement#363

The deploys at the start seem noticeable now at 20 seconds:

=== RUN   TestJumpToDateEndpoint
2022/07/07 08:44:59 Sharing [SERVER_NAME=hs1 SYNAPSE_COMPLEMENT_DATABASE=sqlite SYNAPSE_COMPLEMENT_USE_WORKERS=] host environment variables with container
2022/07/07 08:45:07 Sharing [SERVER_NAME=hs2 SYNAPSE_COMPLEMENT_DATABASE=sqlite SYNAPSE_COMPLEMENT_USE_WORKERS=] host environment variables with container
2022/07/07 08:45:17 Sharing [SERVER_NAME=hs1 SYNAPSE_COMPLEMENT_DATABASE=sqlite SYNAPSE_COMPLEMENT_USE_WORKERS=] host environment variables with container
2022/07/07 08:45:17 Sharing [SERVER_NAME=hs2 SYNAPSE_COMPLEMENT_DATABASE=sqlite SYNAPSE_COMPLEMENT_USE_WORKERS=] host environment variables with container
    msc3030_test.go:24: Deploy times: 17.9124869s blueprints, 5.628477139s containers

Test output no longer streams as it runs

This is just a related extra.

The tests used to stream out what it was doing as it went instead of only barfing it all up at the end. This used to work until we got multiple packages in Complement (tests and csapi ), matrix-org/complement#215. This doesn't help with making Complement go faster but it does help with the feeling of it doing something vs just waiting around arbitrarily.

The way I workaround it now is to remove the ... from the end of

https://github.com/matrix-org/synapse/blob/dcc7873700da4a818e84c44c6190525d39a854cb/scripts-dev/complement.sh#L164

so it only runs against the single tests package where the feature work I care about is in.