scion.sh, topogen: adopt docker compose plugin and cleanup (#4398)

Make the build and test environment easier to setup and use:
* Obsolete (and slow) docker-compose V1 replaced with V2 compose plugin:
  * V1 all gone, invoke `docker compose` not `docker-compose`
  * Made scion.sh mstatus robust wrt docker-compose output formatting and make output more intelligible.
  * Work-around issue in some  V2 versions: set some safe $HOME for python tests that use docker-compose.
* Jeager and prometheus now started/stopped in one command and neither is started nor stopped by default. Documentation updated accordingly too.
* scion.sh commands are start/stop, not run/stop, all subcommands kebab-case
* Doc is a bit clearer regarding supervisor vs docker.
* Fixed revocation test to pass -d to end2end-integration when required.
* Delete the install_docker script: gets outdated too fast, better refer the user to official docker install instructions.

.buildkite/hooks/pre-command

@@ -43,7 +43,7 @@ echo "~~~ Starting bazel remote cache proxy"
 # Start bazel remote cache proxy for S3
 # Note that S3 keys are injected by buildkite, see
 # https://buildkite.com/docs/pipelines/secrets#storing-secrets-with-the-elastic-ci-stack-for-aws
-docker-compose -f .buildkite/hooks/bazel-remote.yml -p bazel_remote up -d
+docker compose --compatibility -f .buildkite/hooks/bazel-remote.yml -p bazel_remote up -d
 
 echo "~~~ Starting go module proxy"
-docker-compose -f .buildkite/hooks/go-module-proxy.yml -p athens up -d
+docker compose --compatibility -f .buildkite/hooks/go-module-proxy.yml -p athens up -d

BUILD.bazel

@@ -245,6 +245,7 @@ updatesrc_update_all(
 write_source_files(
     name = "write_all_source_files",
     additional_update_targets = [
+        "//doc/_build/_static/command:write_files",
         "//doc/command:write_files",
     ],
 )

Makefile

@@ -14,6 +14,10 @@ clean:
 	bazel clean
 	rm -f bin/*
 
+scrub:
+	bazel clean --expunge
+	rm -f bin/*
+
 bazel:
 	rm -f bin/*
 	bazel build //:scion //:scion-ci

acceptance/common/README.md

@@ -56,7 +56,7 @@ Tests can use:
 
 - `self.artifacts`: the specified directory for test outputs, created during setup.
 - `self.get_executable(<name>)`: returns an executable specified using the `--executable` switch.
-- `self.dc`: a wrapper for `docker-compose`, instantiated during `TestTopogen.setup`.
+- `self.dc`: a wrapper for `docker compose`, instantiated during `TestTopogen.setup`.
 
 The `base.main` function is the main entry point to run the tests and must be
 invoked in `__main__`.

acceptance/common/base.py

@@ -203,7 +203,7 @@ def setup_start(self):
             raise Exception("Failed services.\n" + ps)
 
     def teardown(self):
-        # Avoid running docker-compose teardown if setup_prepare failed
+        # Avoid running docker compose teardown if setup_prepare failed
         if self._setup_prepare_failed:
             return
         out_dir = self.artifacts / "logs"

acceptance/common/docker.py

@@ -51,7 +51,8 @@ def __call__(self, *args, **kwargs) -> str:
         # Note: not using plumbum here due to complications with encodings in the captured output
         try:
             res = subprocess.run(
-                ["docker-compose", "-f", self.compose_file, "-p", self.project, *args],
+                ["docker", "compose", "--compatibility",
+                 "-f", self.compose_file, "-p", self.project, *args],
                 check=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE, encoding="utf-8")
         except subprocess.CalledProcessError as e:
             raise _CalledProcessErrorWithOutput(e) from None

acceptance/common/raw.bzl

@@ -1,13 +1,19 @@
 load("//tools/lint:py.bzl", "py_binary", "py_library", "py_test")
 load("@com_github_scionproto_scion_python_deps//:requirements.bzl", "requirement")
 
+# Bug in bazel: HOME isn't set to TEST_TMPDIR.
+# Bug in docker-compose v2.21 a writable HOME is required (eventhough not used).
+# Poor design in Bazel, there's no sane way to obtain the path to some
+# location that's not a litteral dependency.
+# So, HOME must be provided by the invoker.
 def raw_test(
         name,
         src,
         args = [],
         deps = [],
         data = [],
         tags = [],
+        homedir = "",
         local = False):
     py_library(
         name = "%s_lib" % name,
@@ -63,5 +69,6 @@ def raw_test(
             "PYTHONUNBUFFERED": "1",
             # Ensure that unicode output can be printed to the log/console
             "PYTHONIOENCODING": "utf-8",
+            "HOME": homedir,
         },
     )

acceptance/common/topogen.bzl

@@ -1,6 +1,11 @@
 load("//tools/lint:py.bzl", "py_binary", "py_library", "py_test")
 load("@com_github_scionproto_scion_python_deps//:requirements.bzl", "requirement")
 
+# Bug in bazel: HOME isn't set to TEST_TMPDIR.
+# Bug in docker-compose v2.21 a writable HOME is required (eventhough not used).
+# Poor design in Bazel, there's no sane way to obtain the path to some
+# location that's not a litteral dependency.
+# So, HOME must be provided by the invoker.
 def topogen_test(
         name,
         src,
@@ -10,6 +15,7 @@ def topogen_test(
         args = [],
         deps = [],
         data = [],
+        homedir = "",
         tester = "//docker:tester"):
     """Creates a test based on a topology file.
 
@@ -105,6 +111,7 @@ def topogen_test(
             "PYTHONUNBUFFERED": "1",
             # Ensure that unicode output can be printed to the log/console
             "PYTHONIOENCODING": "utf-8",
+            "HOME": homedir,
         },
     )
 

acceptance/router_multi/BUILD.bazel

@@ -30,6 +30,7 @@ raw_test(
         "--bfd",
     ],
     data = data,
+    homedir = "$(rootpath :router)",
     # This test uses sudo and accesses /var/run/netns.
     local = True,
 )
@@ -39,6 +40,7 @@ raw_test(
     src = "test.py",
     args = args,
     data = data,
+    homedir = "$(rootpath :router)",
     # This test uses sudo and accesses /var/run/netns.
     local = True,
 )

acceptance/sig_short_exp_time/test

@@ -55,24 +55,24 @@ run_test() {(set -e
     docker image load -i acceptance/sig_short_exp_time/sig1.tar
     docker image load -i acceptance/sig_short_exp_time/sig2.tar
 
-    docker-compose -f acceptance/sig_short_exp_time/docker-compose.yml up -d dispatcher1 dispatcher2 sig1 sig2 patha pathb
+    docker compose --compatibility -f acceptance/sig_short_exp_time/docker-compose.yml up -d dispatcher1 dispatcher2 sig1 sig2 patha pathb
 
     # Set up forward route on network stack 1 and 2 through the sig tunnel
     # device. The route is a property of the network stack, and persists after
     # the container that added it has exited.
     #
     # If the route configuration fails, the test is not stopped.
-    docker-compose -f acceptance/sig_short_exp_time/docker-compose.yml run --name route1 --rm tester1 ip route add 242.254.200.2/32 dev sig || true
-    docker-compose -f acceptance/sig_short_exp_time/docker-compose.yml run --name route2 --rm tester2 ip route add 242.254.100.2/32 dev sig || true
+    docker compose --compatibility -f acceptance/sig_short_exp_time/docker-compose.yml run --name route1 --rm tester1 ip route add 242.254.200.2/32 dev sig || true
+    docker compose --compatibility -f acceptance/sig_short_exp_time/docker-compose.yml run --name route2 --rm tester2 ip route add 242.254.100.2/32 dev sig || true
 
     echo "Start background ping, ping every 0.2 seconds"
-    docker-compose -f acceptance/sig_short_exp_time/docker-compose.yml run --name tester1 -d tester1 ping -i 0.2 242.254.200.2
+    docker compose --compatibility -f acceptance/sig_short_exp_time/docker-compose.yml run --name tester1 -d tester1 ping -i 0.2 242.254.200.2
 
     echo "Waiting 10 seconds for path A to expire..."
     sleep 10
     echo "Path A expired, simulating it by shutting down path A proxy"
     # Traffic should have switched beforehand to path b, and no pings should be lost
-    docker-compose -f acceptance/sig_short_exp_time/docker-compose.yml stop patha
+    docker compose --compatibility -f acceptance/sig_short_exp_time/docker-compose.yml stop patha
     sleep 1
     docker kill -s SIGINT tester1
 
@@ -104,9 +104,9 @@ OUTPUT_DIR=$TEST_UNDECLARED_OUTPUTS_DIR
 mkdir -p $OUTPUT_DIR/logs
 
 for CNTR in sig1 sig2 dispatcher1 dispatcher2; do
-    docker-compose -f acceptance/sig_short_exp_time/docker-compose.yml logs "$CNTR" > "$OUTPUT_DIR/logs/$CNTR.log"
+    docker compose --compatibility -f acceptance/sig_short_exp_time/docker-compose.yml logs "$CNTR" > "$OUTPUT_DIR/logs/$CNTR.log"
 done
 
-docker-compose -f acceptance/sig_short_exp_time/docker-compose.yml down -v
+docker compose --compatibility -f acceptance/sig_short_exp_time/docker-compose.yml down -v
 
 exit $RC

acceptance/topo_cs_reload/reload_test.go

@@ -108,16 +108,19 @@ func setupTest(t *testing.T) testState {
 	s.mustExec(t, "docker", "image", "load", "-i", "dispatcher.tar")
 	s.mustExec(t, "docker", "image", "load", "-i", "control.tar")
 	// now start the docker containers
-	s.mustExec(t, "docker-compose", "-f", "docker-compose.yml", "up", "-d")
+	s.mustExec(t, "docker", "compose", "--compatibility", "-f", "docker-compose.yml",
+		"up", "-d")
 	// wait a bit to make sure the containers are ready.
 	time.Sleep(time.Second / 2)
 	t.Log("Test setup done")
-	s.mustExec(t, "docker-compose", "-f", "docker-compose.yml", "ps")
+	s.mustExec(t, "docker", "compose", "--compatibility", "-f", "docker-compose.yml",
+		"ps")
 	return s
 }
 
 func (s testState) teardownTest(t *testing.T) {
-	defer s.mustExec(t, "docker-compose", "-f", "docker-compose.yml", "down", "-v")
+	defer s.mustExec(t, "docker", "compose", "--compatibility",
+		"-f", "docker-compose.yml", "down", "-v")
 
 	outdir, exists := os.LookupEnv("TEST_UNDECLARED_OUTPUTS_DIR")
 	require.True(t, exists, "TEST_UNDECLARED_OUTPUTS_DIR must be defined")
@@ -127,8 +130,8 @@ func (s testState) teardownTest(t *testing.T) {
 		"topo_cs_reload_dispatcher":  "disp.log",
 		"topo_cs_reload_control_srv": "control.log",
 	} {
-		cmd := exec.Command("docker-compose", "-f", "docker-compose.yml", "logs", "--no-color",
-			service)
+		cmd := exec.Command("docker", "compose", "--compatibility",
+			"-f", "docker-compose.yml", "logs", "--no-color", service)
 		logFileName := fmt.Sprintf("%s/logs/%s", outdir, file)
 		logFile, err := os.Create(logFileName)
 		if err != nil {
@@ -146,10 +149,10 @@ func (s testState) teardownTest(t *testing.T) {
 func (s testState) loadTopo(t *testing.T, name string) {
 	t.Helper()
 
-	s.mustExec(t, "docker-compose", "-f", "docker-compose.yml", "exec", "-T",
-		"topo_cs_reload_control_srv", "mv", name, "/topology.json")
-	s.mustExec(t, "docker-compose", "-f", "docker-compose.yml", "kill", "-s", "SIGHUP",
-		"topo_cs_reload_control_srv")
+	s.mustExec(t, "docker", "compose", "--compatibility", "-f", "docker-compose.yml",
+		"exec", "-T", "topo_cs_reload_control_srv", "mv", name, "/topology.json")
+	s.mustExec(t, "docker", "compose", "--compatibility", "-f", "docker-compose.yml",
+		"kill", "-s", "SIGHUP", "topo_cs_reload_control_srv")
 }
 
 func (s testState) mustExec(t *testing.T, name string, arg ...string) {

acceptance/topo_daemon_reload/reload_test.go

@@ -71,16 +71,18 @@ func setupTest(t *testing.T) {
 	mustExec(t, "docker", "image", "load", "-i", "dispatcher.tar")
 	mustExec(t, "docker", "image", "load", "-i", "daemon.tar")
 	// now start the docker containers
-	mustExec(t, "docker-compose", "-f", "docker-compose.yml", "up",
-		"-d", "topo_daemon_reload_dispatcher", "topo_daemon_reload_daemon")
+	mustExec(t, "docker", "compose", "--compatibility", "-f", "docker-compose.yml",
+		"up", "-d", "topo_daemon_reload_dispatcher", "topo_daemon_reload_daemon")
 	// wait a bit to make sure the containers are ready.
 	time.Sleep(time.Second / 2)
 	t.Log("Test setup done")
-	mustExec(t, "docker-compose", "-f", "docker-compose.yml", "ps")
+	mustExec(t, "docker", "compose", "--compatibility", "-f", "docker-compose.yml",
+		"ps")
 }
 
 func teardownTest(t *testing.T) {
-	defer mustExec(t, "docker-compose", "-f", "docker-compose.yml", "down", "-v")
+	defer mustExec(t, "docker", "compose", "--compatibility",
+		"-f", "docker-compose.yml", "down", "-v")
 
 	outdir, exists := os.LookupEnv("TEST_UNDECLARED_OUTPUTS_DIR")
 	require.True(t, exists, "TEST_UNDECLARED_OUTPUTS_DIR must be defined")
@@ -90,7 +92,8 @@ func teardownTest(t *testing.T) {
 		"topo_daemon_reload_dispatcher": "disp.log",
 		"topo_daemon_reload_daemon":     "daemon.log",
 	} {
-		cmd := exec.Command("docker-compose", "-f", "docker-compose.yml", "logs", "--no-color",
+		cmd := exec.Command("docker", "compose", "--compatibility",
+			"-f", "docker-compose.yml", "logs", "--no-color",
 			service)
 		logFileName := fmt.Sprintf("%s/logs/%s", outdir, file)
 		logFile, err := os.Create(logFileName)
@@ -108,10 +111,10 @@ func teardownTest(t *testing.T) {
 func loadTopo(t *testing.T, name string) {
 	t.Helper()
 
-	mustExec(t, "docker-compose", "-f", "docker-compose.yml", "exec", "-T",
-		"topo_daemon_reload_daemon", "mv", name, "/topology.json")
-	mustExec(t, "docker-compose", "-f", "docker-compose.yml", "kill", "-s", "SIGHUP",
-		"topo_daemon_reload_daemon")
+	mustExec(t, "docker", "compose", "--compatibility", "-f", "docker-compose.yml",
+		"exec", "-T", "topo_daemon_reload_daemon", "mv", name, "/topology.json")
+	mustExec(t, "docker", "compose", "--compatibility", "-f", "docker-compose.yml",
+		"kill", "-s", "SIGHUP", "topo_daemon_reload_daemon")
 }
 
 func mustExec(t *testing.T, name string, arg ...string) {

acceptance/trc_update/BUILD.bazel

@@ -7,5 +7,6 @@ topogen_test(
         "--executable=end2end_integration:$(location //tools/end2end_integration)",
     ],
     data = ["//tools/end2end_integration"],
+    homedir = "$(rootpath //tools/end2end_integration)",
     topo = "//topology:tiny4.topo",
 )

doc/dev/run.rst

@@ -14,8 +14,9 @@ Running SCION in this developement setup, is also called running a **local topol
 The scripts support two different process orchestrators as "backends":
 
 - `supervisor <http://supervisord.org/>`_. This is the default and a bit more light-weight. Packets are sent over the loopback interface.
-- `docker-compose <https://docs.docker.com/compose/>`_. Runs individual processes in separate containers connected with docker network bridges. Only this mode supports running a "SCION-IP gateway".
+- `docker compose <https://docs.docker.com/compose/>`_. Runs individual processes in separate containers connected with docker network bridges. Only this mode supports running a "SCION-IP gateway".
 
+  .. hint:: Before attempting to use the docker compose mode, be sure to build the necessary docker images with ``make docker-images``
 
 .. TODO
    - Describe configuration directory (referencing manuals)
@@ -31,7 +32,7 @@ Quickstart
    * Build, using ``make``
 
 #. Generate the control-plane PKI keys and certificates, configuration files, and process
-   orchestrator (supervisor or docker-compose) configuration.
+   orchestrator (supervisor or docker compose) configuration.
 
    .. code-block:: bash
 
@@ -84,6 +85,13 @@ Various helper files are also generated for the benefit of scripts and tooling o
 for example, ``gen/sciond_addresses.json`` is a simple mapping from AS number to the address of the
 corresponding :doc:`scion daemon </manuals/daemon>` instance.
 
+If  :option:`scion.sh topology -d` command is used, configuration files are created to
+enable running the SCION services in docker containers (see :ref:`docker-section`). Otherwise,
+a configuration file is created to enable running the SCION services as plain processes
+(see :ref:`supervisor-section`)
+
+.. _supervisor-section:
+
 supervisor
 ----------
 The ``gen/supervisord.conf`` configuration defines the programs that make up the local topology.
@@ -109,16 +117,18 @@ For example::
    # and now ping this host from inside AS 1-ff00:0:110, with interactive path prompt
    bin/scion ping --sciond $(./scion.sh sciond-addr 110) 1-ff00:0:111,127.0.0.1 --interactive
 
+.. _docker-section:
+
 docker
 ------
-The main docker-compose file is ``gen/scion-dc.yml``.
+The main docker compose file is ``gen/scion-dc.yml``.
 
 Each SCION service or router runs in a separate container, and the network access of the individual
 containers is configured to mimick real-world connectivity.
 
 There are "tester" containers configured in each AS to mimick end hosts in a SCION AS.
 These tester containers can be used to run commands accessing the SCION network.
-As a shorthand for the somewhat unwieldy ``docker-compose`` invocation, the :file-ref:`tools/dc`
+As a shorthand for the somewhat unwieldy ``docker compose`` invocation, the :file-ref:`tools/dc`
 script can be used. For example::
 
    # show paths from 1-ff00:0:112 to 1-ff00:0:110
@@ -140,7 +150,7 @@ scion.sh
 
 .. Note::
    The SCION tools and services need to be built **before** running these commands, using
-   ``make`` or ``make docker-images`` (when using the docker-compose configuration).
+   ``make`` or ``make docker-images`` (when using the docker compose configuration).
 
 The basic usage is ``./scion.sh <subcommand> <options>``. The main subcommands are:
 
@@ -149,7 +159,7 @@ The basic usage is ``./scion.sh <subcommand> <options>``. The main subcommands a
 .. option:: topology
 
    Generate the control-plane PKI keys and certificates, configuration files,
-   and process orchestrator (supervisor or docker-compose) configuration
+   and process orchestrator (supervisor or docker compose) configuration
    for a given network topopology defined in a
    :file-ref:`*.topo configuration file <topology/README.md>`.
 
@@ -161,7 +171,7 @@ The basic usage is ``./scion.sh <subcommand> <options>``. The main subcommands a
 
    .. option:: -d, --docker
 
-      Create a docker-compose configuration (instead of default supervisord).
+      Create a docker compose configuration (instead of default supervisord).
 
    .. option:: --sig
 
@@ -180,6 +190,14 @@ The basic usage is ``./scion.sh <subcommand> <options>``. The main subcommands a
 
    Terminate this run of the local SCION topology.
 
+.. option:: start-monitoring
+
+   Start the monitoring services (jaeger and prometheus).
+
+.. option:: stop-monitoring
+
+   Stop the monitoring services.
+
 .. option:: sciond-addr <ISD-AS>
 
    Return the address for the scion daemon for the matching ISD-AS by consulting
@@ -190,3 +208,15 @@ The basic usage is ``./scion.sh <subcommand> <options>``. The main subcommands a
 .. option:: help
 
    Describe all available subcommands
+
+end2end_integration
+===================
+
+:program:bin/end2end_integration is a basic functional test.
+
+The basic usage is ``./end2end_integration <options>``.
+
+.. option:: -d
+
+   Assume the SCION services are dockerized. Must be consistent with the last
+   invocation of ``scion.sh topology``.