From b73e6e2730b0fc14a537b224d6aa67b555f53fdf Mon Sep 17 00:00:00 2001 From: Robert Steiner Date: Mon, 11 Nov 2024 13:17:38 +0100 Subject: [PATCH 01/11] docs(framework:skip) Update Docker docs for 1.13.0 Signed-off-by: Robert Steiner --- doc/source/docker/enable-tls.rst | 56 +---------- doc/source/docker/persist-superlink-state.rst | 2 +- doc/source/docker/pin-version.rst | 6 +- doc/source/docker/run-as-root-user.rst | 2 +- doc/source/docker/run-as-subprocess.rst | 4 +- .../docker/set-environment-variables.rst | 3 +- .../docker/tutorial-quickstart-docker.rst | 99 +++++++++++-------- 7 files changed, 72 insertions(+), 100 deletions(-) diff --git a/doc/source/docker/enable-tls.rst b/doc/source/docker/enable-tls.rst index 7225f86a5ddb..a3645d22e35d 100644 --- a/doc/source/docker/enable-tls.rst +++ b/doc/source/docker/enable-tls.rst @@ -38,7 +38,8 @@ of the container: flwr/superlink:|stable_flwr_version| \ --ssl-ca-certfile certificates/ca.crt \ --ssl-certfile certificates/server.pem \ - --ssl-keyfile certificates/server.key + --ssl-keyfile certificates/server.key \ + .. dropdown:: Understanding the command @@ -85,7 +86,8 @@ Assuming that the ``ca.crt`` certificate already exists locally, we can use the $ docker run --rm \ --volume ./ca.crt:/app/ca.crt/:ro \ flwr/supernode:|stable_flwr_version| \ - --root-certificates ca.crt + --root-certificates ca.crt \ + .. dropdown:: Understanding the command @@ -100,53 +102,3 @@ Assuming that the ``ca.crt`` certificate already exists locally, we can use the | inside the container. | | The ``ca.crt`` file is used to verify the identity of the SuperLink. - -SuperExec ---------- - -Assuming all files we need are in the local ``certificates`` directory where the -SuperExec will be executed from, we can use the flag ``--volume`` to mount the local -directory into the ``/app/certificates/`` directory of the container: - -.. code-block:: bash - :substitutions: - - $ docker run --rm \ - --volume ./certificates/:/app/certificates/:ro \ - flwr/superexec:|stable_flwr_version| \ - --ssl-ca-certfile certificates/ca.crt \ - --ssl-certfile certificates/server.pem \ - --ssl-keyfile certificates/server.key \ - --executor-config \ - root-certificates=\"certificates/superlink_ca.crt\" - -.. dropdown:: Understanding the command - - * ``docker run``: This tells Docker to run a container from an image. - * ``--rm``: Remove the container once it is stopped or the command exits. - * | ``--volume ./certificates/:/app/certificates/:ro``: Mount the ``certificates`` directory in - | the current working directory of the host machine as a read-only volume at the - | ``/app/certificates`` directory inside the container. - | - | This allows the container to access the TLS certificates that are stored in the certificates - | directory. - * | :substitution-code:`flwr/superexec:|stable_flwr_version|`: The name of the image to be run and the specific - | tag of the image. The tag :substitution-code:`|stable_flwr_version|` represents a specific version of the image. - * | ``--ssl-ca-certfile certificates/ca.crt``: Specify the location of the CA certificate file - | inside the container. - | - | The ``certificates/ca.crt`` file is a certificate that is used to verify the identity of the - | SuperExec. - * | ``--ssl-certfile certificates/server.pem``: Specify the location of the SuperExec's - | TLS certificate file inside the container. - | - | The ``certificates/server.pem`` file is used to identify the SuperExec and to encrypt the - | data that is transmitted over the network. - * | ``--ssl-keyfile certificates/server.key``: Specify the location of the SuperExec's - | TLS private key file inside the container. - | - | The ``certificates/server.key`` file is used to decrypt the data that is transmitted over - | the network. - * | ``--executor-config root-certificates=\"certificates/superlink_ca.crt\"``: Specify the - | location of the CA certificate file inside the container that the SuperExec executor - | should use to verify the SuperLink's identity. diff --git a/doc/source/docker/persist-superlink-state.rst b/doc/source/docker/persist-superlink-state.rst index 214e408c44c3..e61fcd3a63c2 100644 --- a/doc/source/docker/persist-superlink-state.rst +++ b/doc/source/docker/persist-superlink-state.rst @@ -31,7 +31,7 @@ specify the name of the database file. $ docker run --rm \ --volume ./state/:/app/state flwr/superlink:|stable_flwr_version| \ --database state.db \ - ... + As soon as the SuperLink starts, the file ``state.db`` is created in the ``state`` directory on your host system. If the file already exists, the SuperLink tries to diff --git a/doc/source/docker/pin-version.rst b/doc/source/docker/pin-version.rst index 4a69860aa428..36ed4ee5d379 100644 --- a/doc/source/docker/pin-version.rst +++ b/doc/source/docker/pin-version.rst @@ -25,7 +25,7 @@ This will output .. code-block:: bash :substitutions: - flwr/superlink@sha256:|stable__flwr_superlink_docker_digest| + flwr/superlink@sha256:|stable_flwr_superlink_docker_digest| Next, we can pin the digest when running a new SuperLink container: @@ -33,5 +33,5 @@ Next, we can pin the digest when running a new SuperLink container: :substitutions: $ docker run \ - --rm flwr/superlink@sha256:|latest_version_docker_sha| \ - [OPTIONS] + --rm flwr/superlink@sha256:|stable_flwr_superlink_docker_digest| \ + diff --git a/doc/source/docker/run-as-root-user.rst b/doc/source/docker/run-as-root-user.rst index 5f8e5eae43af..24bd39e2f53d 100644 --- a/doc/source/docker/run-as-root-user.rst +++ b/doc/source/docker/run-as-root-user.rst @@ -16,7 +16,7 @@ Run the Docker image with the ``-u`` flag and specify ``root`` as the username: .. code-block:: bash :substitutions: - $ docker run --rm -u root flwr/superlink:|stable_flwr_version| + $ docker run --rm -u root flwr/superlink:|stable_flwr_version| This command will run the Docker container with root user privileges. diff --git a/doc/source/docker/run-as-subprocess.rst b/doc/source/docker/run-as-subprocess.rst index d97319ff52af..074c2e813645 100644 --- a/doc/source/docker/run-as-subprocess.rst +++ b/doc/source/docker/run-as-subprocess.rst @@ -47,7 +47,7 @@ to execute the ClientApp as a subprocess: --detach \ flwr_supernode:0.0.1 \ --insecure \ - --superlink superlink:9092 \ - --node-config "partition-id=1 num-partitions=2" \ + --superlink :9092 \ + --node-config \ --supernode-address localhost:9094 \ --isolation subprocess diff --git a/doc/source/docker/set-environment-variables.rst b/doc/source/docker/set-environment-variables.rst index f5d860812bab..c9c6bc946509 100644 --- a/doc/source/docker/set-environment-variables.rst +++ b/doc/source/docker/set-environment-variables.rst @@ -11,4 +11,5 @@ Example :substitutions: $ docker run -e FLWR_TELEMETRY_ENABLED=0 -e FLWR_TELEMETRY_LOGGING=0 \ - --rm flwr/superlink:|stable_flwr_version| + --rm flwr/superlink:|stable_flwr_version| \ + diff --git a/doc/source/docker/tutorial-quickstart-docker.rst b/doc/source/docker/tutorial-quickstart-docker.rst index 4be82e0685fd..67006f3a4da8 100644 --- a/doc/source/docker/tutorial-quickstart-docker.rst +++ b/doc/source/docker/tutorial-quickstart-docker.rst @@ -54,19 +54,25 @@ Open your terminal and run: :substitutions: $ docker run --rm \ - -p 9091:9091 -p 9092:9092 \ + -p 9091:9091 -p 9092:9092 -p 9093:9093 \ --network flwr-network \ --name superlink \ --detach \ - flwr/superlink:|stable_flwr_version| --insecure + flwr/superlink:|stable_flwr_version| \ + --insecure \ + --executor \ + flwr.superexec.deployment:executor \ + --isolation \ + process .. dropdown:: Understand the command * ``docker run``: This tells Docker to run a container from an image. * ``--rm``: Remove the container once it is stopped or the command exits. - * | ``-p 9091:9091 -p 9092:9092``: Map port ``9091`` and ``9092`` of the container to the same port of - | the host machine, allowing other services to access the Driver API on - | ``http://localhost:9091`` and the Fleet API on ``http://localhost:9092``. + * | ``-p 9091:9091 -p 9092:9092 -p 9093:9093``: Map port ``9091``, ``9092`` and ``9093`` of the + | container to the same port of the host machine, allowing other services to access the + | Driver API on ``http://localhost:9091``, the Fleet API on ``http://localhost:9092`` and + | the ExecAPI on ``http://localhost:9093``. * ``--network flwr-network``: Make the container join the network named ``flwr-network``. * ``--name superlink``: Assign the name ``superlink`` to the container. * ``--detach``: Run the container in the background, freeing up the terminal. @@ -74,6 +80,10 @@ Open your terminal and run: | tag of the image. The tag :substitution-code:`|stable_flwr_version|` represents a :doc:`specific version ` of the image. * | ``--insecure``: This flag tells the container to operate in an insecure mode, allowing | unencrypted communication. + * | ``--executor flwr.superexec.deployment:executor`` Use the + | ``flwr.superexec.deployment:executor`` executor to run the ServerApp. + * | ``--isolation process``: Tells the SuperLink that the ServerApp is created by separate + | independent process. The SuperLink does not attempt to create it. Step 3: Start the SuperNode --------------------------- @@ -107,8 +117,8 @@ Start two SuperNode containers. * ``--network flwr-network``: Make the container join the network named ``flwr-network``. * ``--name supernode-1``: Assign the name ``supernode-1`` to the container. * ``--detach``: Run the container in the background, freeing up the terminal. - * | ``flwr/supernode:|stable_flwr_version|``: This is the name of the image to be run and the specific tag - | of the image. + * | :substitution-code:`flwr/supernode:|stable_flwr_version|`: This is the name of the + | image to be run and the specific tag of the image. * | ``--insecure``: This flag tells the container to operate in an insecure mode, allowing | unencrypted communication. * | ``--superlink superlink:9092``: Connect to the SuperLink's Fleet API at the address @@ -165,7 +175,7 @@ required dependencies. .. dropdown:: Understand the Dockerfile * | :substitution-code:`FROM flwr/clientapp:|stable_flwr_version|`: This line specifies that the Docker image - | to be built from is the ``flwr/clientapp image``, version :substitution-code:`|stable_flwr_version|`. + | to be built from is the ``flwr/clientapp`` image, version :substitution-code:`|stable_flwr_version|`. * | ``WORKDIR /app``: Set the working directory for the container to ``/app``. | Any subsequent commands that reference a directory will be relative to this directory. * | ``COPY pyproject.toml .``: Copy the ``pyproject.toml`` file @@ -232,23 +242,23 @@ required dependencies. flwr_clientapp:0.0.1 \ --supernode supernode-2:9095 -Step 5: Start the SuperExec ---------------------------- +Step 5: Start a ServerApp +------------------------- -The procedure for building and running a SuperExec image is almost identical to the +The procedure for building and running a ServerApp image is almost identical to the ClientApp image. Similar to the ClientApp image, you will need to create a Dockerfile that extends the -SuperExec image and installs the required FAB dependencies. +ServerApp image and installs the required FAB dependencies. -1. Create a SuperExec Dockerfile called ``Dockerfile.superexec`` and paste the following +1. Create a ServerApp Dockerfile called ``Dockerfile.serverapp`` and paste the following code in: .. code-block:: dockerfile - :caption: Dockerfile.superexec + :caption: Dockerfile.serverapp :substitutions: - FROM flwr/superexec:|stable_flwr_version| + FROM flwr/serverapp:|stable_flwr_version| WORKDIR /app @@ -256,12 +266,12 @@ SuperExec image and installs the required FAB dependencies. RUN sed -i 's/.*flwr\[simulation\].*//' pyproject.toml \ && python -m pip install -U --no-cache-dir . - ENTRYPOINT ["flower-superexec", "--executor", "flwr.superexec.deployment:executor"] + ENTRYPOINT ["flwr-serverapp"] .. dropdown:: Understand the Dockerfile - * | :substitution-code:`FROM flwr/superexec:|stable_flwr_version|`: This line specifies that the Docker image - | to be built from is the ``flwr/superexec image``, version :substitution-code:`|stable_flwr_version|`. + * | :substitution-code:`FROM flwr/serverapp:|stable_flwr_version|`: This line specifies that the Docker image + | to be built from is the ``flwr/serverapp`` image, version :substitution-code:`|stable_flwr_version|`. * | ``WORKDIR /app``: Set the working directory for the container to ``/app``. | Any subsequent commands that reference a directory will be relative to this directory. * | ``COPY pyproject.toml .``: Copy the ``pyproject.toml`` file @@ -273,47 +283,41 @@ SuperExec image and installs the required FAB dependencies. | | The ``-U`` flag indicates that any existing packages should be upgraded, and | ``--no-cache-dir`` prevents pip from using the cache to speed up the installation. - * | ``ENTRYPOINT ["flower-superexec"``: Set the command ``flower-superexec`` to be + * | ``ENTRYPOINT ["flwr-serverapp"]``: Set the command ``flwr-serverapp`` to be | the default command run when the container is started. - | - | ``"--executor", "flwr.superexec.deployment:executor"]`` Use the - | ``flwr.superexec.deployment:executor`` executor to run the ServerApps. 2. Afterward, in the directory that holds the Dockerfile, execute this Docker command to - build the SuperExec image: + build the ServerApp image: .. code-block:: bash - $ docker build -f Dockerfile.superexec -t flwr_superexec:0.0.1 . + $ docker build -f Dockerfile.serverapp -t flwr_serverapp:0.0.1 . -3. Start the SuperExec container: +3. Start the ServerApp container: .. code-block:: bash $ docker run --rm \ - -p 9093:9093 \ --network flwr-network \ - --name superexec \ + --name serverapp \ --detach \ - flwr_superexec:0.0.1 \ + flwr_serverapp:0.0.1 \ --insecure \ - --executor-config superlink=\"superlink:9091\" + --superlink superlink:9091 .. dropdown:: Understand the command * ``docker run``: This tells Docker to run a container from an image. * ``--rm``: Remove the container once it is stopped or the command exits. - * | ``-p 9093:9093``: Map port ``9093`` of the container to the same port of - | the host machine, allowing you to access the SuperExec API on ``http://localhost:9093``. * ``--network flwr-network``: Make the container join the network named ``flwr-network``. - * ``--name superexec``: Assign the name ``superexec`` to the container. + * ``--name serverapp``: Assign the name ``serverapp`` to the container. * ``--detach``: Run the container in the background, freeing up the terminal. - * | ``flwr_superexec:0.0.1``: This is the name of the image to be run and the specific tag + * | ``flwr_serverapp:0.0.1``: This is the name of the image to be run and the specific tag | of the image. * | ``--insecure``: This flag tells the container to operate in an insecure mode, allowing | unencrypted communication. - * | ``--executor-config superlink=\"superlink:9091\"``: Configure the SuperExec executor to - | connect to the SuperLink running on port ``9091``. + * | ``--superlink superlink:9091``: Connect to the SuperLink's Driver API at the address + | ``superlink:9091``. Step 6: Run the Quickstart Project ---------------------------------- @@ -347,17 +351,25 @@ Step 7: Update the Application partition_train_test = partition.train_test_split(test_size=0.2, seed=43) # ... -2. Stop the current ClientApp containers: +2. Stop the current ClientApp and ServerApp containers: .. code-block:: bash - $ docker stop $(docker ps -a -q --filter ancestor=flwr_clientapp:0.0.1) + $ docker stop $(docker ps -a -q --filter ancestor=flwr_clientapp:0.0.1) serverapp + +3. Rebuild ClientApp and ServerApp image: + + .. note:: + + If you have modified the dependencies listed in your ``pyproject.toml`` file, it + is essential to rebuild images. -3. Rebuild the FAB and ClientApp image: + If you haven't made any changes, you can skip this step. .. code-block:: bash - $ docker build -f Dockerfile.clientapp -t flwr_clientapp:0.0.1 . + $ docker build -f Dockerfile.clientapp -t flwr_clientapp:0.0.1 . && \ + docker build -f Dockerfile.serverapp -t flwr_serverapp:0.0.1 . 4. Launch two new ClientApp containers based on the newly built image: @@ -373,6 +385,13 @@ Step 7: Update the Application --detach \ flwr_clientapp:0.0.1 \ --supernode supernode-2:9095 + $ docker run --rm \ + --network flwr-network \ + --name serverapp \ + --detach \ + flwr_serverapp:0.0.1 \ + --insecure \ + --superlink superlink:9091 5. Run the updated project: @@ -390,7 +409,7 @@ Remove the containers and the bridge network: $ docker stop $(docker ps -a -q --filter ancestor=flwr_clientapp:0.0.1) \ supernode-1 \ supernode-2 \ - superexec \ + serverapp \ superlink $ docker network rm flwr-network From 4745f174e3e13d0be47f0e29ec5ec5d3c33f43ec Mon Sep 17 00:00:00 2001 From: Robert Steiner Date: Tue, 12 Nov 2024 11:19:42 +0100 Subject: [PATCH 02/11] Update doc/source/docker/tutorial-quickstart-docker.rst Co-authored-by: Chong Shen Ng --- doc/source/docker/tutorial-quickstart-docker.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/source/docker/tutorial-quickstart-docker.rst b/doc/source/docker/tutorial-quickstart-docker.rst index 67006f3a4da8..8e52c0f2f10b 100644 --- a/doc/source/docker/tutorial-quickstart-docker.rst +++ b/doc/source/docker/tutorial-quickstart-docker.rst @@ -316,7 +316,7 @@ ServerApp image and installs the required FAB dependencies. | of the image. * | ``--insecure``: This flag tells the container to operate in an insecure mode, allowing | unencrypted communication. - * | ``--superlink superlink:9091``: Connect to the SuperLink's Driver API at the address + * | ``--superlink superlink:9091``: Connect to the SuperLink's ServerAppIO API at the address | ``superlink:9091``. Step 6: Run the Quickstart Project From 5feb4e5a590c7648518c597eff219503f4624900 Mon Sep 17 00:00:00 2001 From: Robert Steiner Date: Tue, 12 Nov 2024 11:19:50 +0100 Subject: [PATCH 03/11] Update doc/source/docker/tutorial-quickstart-docker.rst Co-authored-by: Chong Shen Ng --- doc/source/docker/tutorial-quickstart-docker.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/source/docker/tutorial-quickstart-docker.rst b/doc/source/docker/tutorial-quickstart-docker.rst index 8e52c0f2f10b..29c2a7564d56 100644 --- a/doc/source/docker/tutorial-quickstart-docker.rst +++ b/doc/source/docker/tutorial-quickstart-docker.rst @@ -71,7 +71,7 @@ Open your terminal and run: * ``--rm``: Remove the container once it is stopped or the command exits. * | ``-p 9091:9091 -p 9092:9092 -p 9093:9093``: Map port ``9091``, ``9092`` and ``9093`` of the | container to the same port of the host machine, allowing other services to access the - | Driver API on ``http://localhost:9091``, the Fleet API on ``http://localhost:9092`` and + | ServerAppIO API on ``http://localhost:9091``, the Fleet API on ``http://localhost:9092`` and | the ExecAPI on ``http://localhost:9093``. * ``--network flwr-network``: Make the container join the network named ``flwr-network``. * ``--name superlink``: Assign the name ``superlink`` to the container. From 41e9e14530171bd81abacd14f733e8253a340db3 Mon Sep 17 00:00:00 2001 From: Robert Steiner Date: Wed, 13 Nov 2024 18:03:50 +0100 Subject: [PATCH 04/11] remove executor flag Signed-off-by: Robert Steiner --- doc/source/docker/tutorial-quickstart-docker.rst | 4 ---- 1 file changed, 4 deletions(-) diff --git a/doc/source/docker/tutorial-quickstart-docker.rst b/doc/source/docker/tutorial-quickstart-docker.rst index 29c2a7564d56..a89e8ba6ce4d 100644 --- a/doc/source/docker/tutorial-quickstart-docker.rst +++ b/doc/source/docker/tutorial-quickstart-docker.rst @@ -60,8 +60,6 @@ Open your terminal and run: --detach \ flwr/superlink:|stable_flwr_version| \ --insecure \ - --executor \ - flwr.superexec.deployment:executor \ --isolation \ process @@ -80,8 +78,6 @@ Open your terminal and run: | tag of the image. The tag :substitution-code:`|stable_flwr_version|` represents a :doc:`specific version ` of the image. * | ``--insecure``: This flag tells the container to operate in an insecure mode, allowing | unencrypted communication. - * | ``--executor flwr.superexec.deployment:executor`` Use the - | ``flwr.superexec.deployment:executor`` executor to run the ServerApp. * | ``--isolation process``: Tells the SuperLink that the ServerApp is created by separate | independent process. The SuperLink does not attempt to create it. From 4e39e3357999fb99ea71cb43060b6aaa99315095 Mon Sep 17 00:00:00 2001 From: Robert Steiner Date: Fri, 15 Nov 2024 19:41:34 +0100 Subject: [PATCH 05/11] update docker docs Signed-off-by: Robert Steiner --- doc/source/docker/run-as-subprocess.rst | 122 ++++++++++++------ .../docker/tutorial-quickstart-docker.rst | 54 ++++---- 2 files changed, 113 insertions(+), 63 deletions(-) diff --git a/doc/source/docker/run-as-subprocess.rst b/doc/source/docker/run-as-subprocess.rst index 074c2e813645..946dfb0d9ec8 100644 --- a/doc/source/docker/run-as-subprocess.rst +++ b/doc/source/docker/run-as-subprocess.rst @@ -1,53 +1,97 @@ -Run ClientApp as a Subprocess -============================= +Run ClientApp or ServerApp as a Subprocess +========================================== -In this mode, the ClientApp is executed as a subprocess within the SuperNode Docker -container, rather than running in a separate container. This approach reduces the number -of running containers, which can be beneficial for environments with limited resources. -However, it also means that the ClientApp is no longer isolated from the SuperNode, -which may introduce additional security concerns. +By default, the ServerApp is executed as a subprocess within the SuperLink Docker +container, and the ClientApp is run as a subprocess within the SuperNode Docker +container, rather than using separate containers as shown in the +:doc:`tutorial-quickstart-docker` guide. -Prerequisites -------------- +This approach reduces the number of running containers, which can be beneficial for +environments with limited resources. However, it also means that the applications are +not isolated from their parent containers, which may introduce additional security +concerns. -1. Before running the ClientApp as a subprocess, ensure that the FAB dependencies have - been installed in the SuperNode images. This can be done by extending the SuperNode - image: +.. tab-set:: - .. code-block:: dockerfile - :caption: Dockerfile.supernode - :linenos: - :substitutions: + .. tab-item:: ClientApp - FROM flwr/supernode:|stable_flwr_version| + **Prerequisites** - WORKDIR /app - COPY pyproject.toml . - RUN sed -i 's/.*flwr\[simulation\].*//' pyproject.toml \ - && python -m pip install -U --no-cache-dir . + 1. Before running the ClientApp as a subprocess, ensure that the FAB dependencies have + been installed in the SuperNode images. This can be done by extending the SuperNode + image: - ENTRYPOINT ["flower-supernode"] + .. code-block:: dockerfile + :caption: supernode.Dockerfile + :linenos: + :substitutions: -2. Next, build the SuperNode Docker image by running the following command in the - directory where Dockerfile is located: + FROM flwr/supernode:|stable_flwr_version| - .. code-block:: shell + WORKDIR /app + COPY pyproject.toml . + RUN sed -i 's/.*flwr\[simulation\].*//' pyproject.toml \ + && python -m pip install -U --no-cache-dir . - $ docker build -f Dockerfile.supernode -t flwr_supernode:0.0.1 . + ENTRYPOINT ["flower-supernode"] -Run the ClientApp as a Subprocess ---------------------------------- + 2. Next, build the SuperNode Docker image by running the following command in the + directory where Dockerfile is located: -Start the SuperNode with the flag ``--isolation subprocess``, which tells the SuperNode -to execute the ClientApp as a subprocess: + .. code-block:: shell -.. code-block:: shell + $ docker build -f supernode.Dockerfile -t flwr_supernode:0.0.1 . - $ docker run --rm \ - --detach \ - flwr_supernode:0.0.1 \ - --insecure \ - --superlink :9092 \ - --node-config \ - --supernode-address localhost:9094 \ - --isolation subprocess + **Run the ClientApp as a Subprocess** + + Start the SuperNode and run the ClientApp as a subprocess: + + .. code-block:: shell + + $ docker run --rm \ + --detach \ + flwr_supernode:0.0.1 \ + --insecure \ + --superlink :9092 \ + --node-config + + .. tab-item:: ServerApp + + **Prerequisites** + + 1. Before running the ServerApp as a subprocess, ensure that the FAB dependencies have + been installed in the SuperLink images. This can be done by extending the SuperLink + image: + + .. code-block:: dockerfile + :caption: superlink.Dockerfile + :linenos: + :substitutions: + + FROM flwr/superlink:|stable_flwr_version| + + WORKDIR /app + COPY pyproject.toml . + RUN sed -i 's/.*flwr\[simulation\].*//' pyproject.toml \ + && python -m pip install -U --no-cache-dir . + + ENTRYPOINT ["flower-superlink"] + + 2. Next, build the superlink Docker image by running the following command in the + directory where Dockerfile is located: + + .. code-block:: shell + + $ docker build -f superlink.Dockerfile -t flwr_superlink:0.0.1 . + + **Run the ServerApp as a Subprocess** + + Start the SuperLink and run the ServerApp as a subprocess: + + .. code-block:: shell + + $ docker run --rm \ + -p 9091:9091 -p 9092:9092 -p 9093:9093 \ + --detach \ + flwr_superlink:0.0.1 \ + --insecure diff --git a/doc/source/docker/tutorial-quickstart-docker.rst b/doc/source/docker/tutorial-quickstart-docker.rst index a89e8ba6ce4d..624497dd2bd4 100644 --- a/doc/source/docker/tutorial-quickstart-docker.rst +++ b/doc/source/docker/tutorial-quickstart-docker.rst @@ -81,8 +81,8 @@ Open your terminal and run: * | ``--isolation process``: Tells the SuperLink that the ServerApp is created by separate | independent process. The SuperLink does not attempt to create it. -Step 3: Start the SuperNode ---------------------------- +Step 3: Start the SuperNodes +---------------------------- Start two SuperNode containers. @@ -100,7 +100,7 @@ Start two SuperNode containers. --insecure \ --superlink superlink:9092 \ --node-config "partition-id=0 num-partitions=2" \ - --supernode-address 0.0.0.0:9094 \ + --clientappio-api-address 0.0.0.0:9094 \ --isolation process .. dropdown:: Understand the command @@ -121,8 +121,8 @@ Start two SuperNode containers. | ``superlink:9092``. * | ``--node-config "partition-id=0 num-partitions=2"``: Set the partition ID to ``0`` and the | number of partitions to ``2`` for the SuperNode configuration. - * | ``--supernode-address 0.0.0.0:9094``: Set the address and port number that the SuperNode - | is listening on. + * | ``--clientappio-api-address 0.0.0.0:9094``: Set the address and port number that the + | SuperNode is listening on to communicate with the ClientApp. * | ``--isolation process``: Tells the SuperNode that the ClientApp is created by separate | independent process. The SuperNode does not attempt to create it. @@ -140,7 +140,7 @@ Start two SuperNode containers. --insecure \ --superlink superlink:9092 \ --node-config "partition-id=1 num-partitions=2" \ - --supernode-address 0.0.0.0:9095 \ + --clientappio-api-address 0.0.0.0:9095 \ --isolation process Step 4: Start the ClientApp @@ -151,11 +151,11 @@ base for building your own ClientApp image. In order to install the FAB dependen you will need to create a Dockerfile that extends the ClientApp image and installs the required dependencies. -1. Create a ClientApp Dockerfile called ``Dockerfile.clientapp`` and paste the following +1. Create a ClientApp Dockerfile called ``clientapp.Dockerfile`` and paste the following code into it: .. code-block:: dockerfile - :caption: Dockerfile.clientapp + :caption: clientapp.Dockerfile :linenos: :substitutions: @@ -199,7 +199,7 @@ required dependencies. .. code-block:: bash - $ docker build -f Dockerfile.clientapp -t flwr_clientapp:0.0.1 . + $ docker build -f clientapp.Dockerfile -t flwr_clientapp:0.0.1 . .. note:: @@ -215,7 +215,8 @@ required dependencies. --network flwr-network \ --detach \ flwr_clientapp:0.0.1 \ - --supernode supernode-1:9094 + --insecure \ + --clientappio-api-address supernode-1:9094 .. dropdown:: Understand the command @@ -223,10 +224,12 @@ required dependencies. * ``--rm``: Remove the container once it is stopped or the command exits. * ``--network flwr-network``: Make the container join the network named ``flwr-network``. * ``--detach``: Run the container in the background, freeing up the terminal. + * | ``--insecure``: This flag tells the container to operate in an insecure mode, allowing + | unencrypted communication. * | ``flwr_clientapp:0.0.1``: This is the name of the image to be run and the specific tag | of the image. - * | ``--supernode supernode-1:9094``: Connect to the SuperNode's Fleet API at the address - | ``supernode-1:9094``. + * | ``--clientappio-api-address supernode-1:9094``: Connect to the SuperNode's ClientAppIO + | API at the address ``supernode-1:9094``. 4. Start the second ClientApp container: @@ -236,7 +239,8 @@ required dependencies. --network flwr-network \ --detach \ flwr_clientapp:0.0.1 \ - --supernode supernode-2:9095 + --insecure \ + --clientappio-api-address supernode-2:9095 Step 5: Start a ServerApp ------------------------- @@ -247,11 +251,11 @@ ClientApp image. Similar to the ClientApp image, you will need to create a Dockerfile that extends the ServerApp image and installs the required FAB dependencies. -1. Create a ServerApp Dockerfile called ``Dockerfile.serverapp`` and paste the following +1. Create a ServerApp Dockerfile called ``serverapp.Dockerfile`` and paste the following code in: .. code-block:: dockerfile - :caption: Dockerfile.serverapp + :caption: serverapp.Dockerfile :substitutions: FROM flwr/serverapp:|stable_flwr_version| @@ -287,7 +291,7 @@ ServerApp image and installs the required FAB dependencies. .. code-block:: bash - $ docker build -f Dockerfile.serverapp -t flwr_serverapp:0.0.1 . + $ docker build -f serverapp.Dockerfile -t flwr_serverapp:0.0.1 . 3. Start the ServerApp container: @@ -299,7 +303,7 @@ ServerApp image and installs the required FAB dependencies. --detach \ flwr_serverapp:0.0.1 \ --insecure \ - --superlink superlink:9091 + --serverappio-api-address superlink:9091 .. dropdown:: Understand the command @@ -312,8 +316,8 @@ ServerApp image and installs the required FAB dependencies. | of the image. * | ``--insecure``: This flag tells the container to operate in an insecure mode, allowing | unencrypted communication. - * | ``--superlink superlink:9091``: Connect to the SuperLink's ServerAppIO API at the address - | ``superlink:9091``. + * | ``--serverappio-api-address superlink:9091``: Connect to the SuperLink's ServerAppIO API + | at the address ``superlink:9091``. Step 6: Run the Quickstart Project ---------------------------------- @@ -364,8 +368,8 @@ Step 7: Update the Application .. code-block:: bash - $ docker build -f Dockerfile.clientapp -t flwr_clientapp:0.0.1 . && \ - docker build -f Dockerfile.serverapp -t flwr_serverapp:0.0.1 . + $ docker build -f clientapp.Dockerfile -t flwr_clientapp:0.0.1 . && \ + docker build -f serverapp.Dockerfile -t flwr_serverapp:0.0.1 . 4. Launch two new ClientApp containers based on the newly built image: @@ -375,19 +379,21 @@ Step 7: Update the Application --network flwr-network \ --detach \ flwr_clientapp:0.0.1 \ - --supernode supernode-1:9094 + --insecure \ + --clientappio-api-address supernode-1:9094 $ docker run --rm \ --network flwr-network \ --detach \ flwr_clientapp:0.0.1 \ - --supernode supernode-2:9095 + --insecure \ + --clientappio-api-address supernode-2:9095 $ docker run --rm \ --network flwr-network \ --name serverapp \ --detach \ flwr_serverapp:0.0.1 \ --insecure \ - --superlink superlink:9091 + ----serverappio-api-address superlink:9091 5. Run the updated project: From 58a8380e2319993487430a0c2e99952153589265 Mon Sep 17 00:00:00 2001 From: Robert Steiner Date: Mon, 18 Nov 2024 18:03:19 +0100 Subject: [PATCH 06/11] update tls and subprocess Signed-off-by: Robert Steiner --- doc/source/docker/enable-tls.rst | 407 ++++++++++++++---- doc/source/docker/run-as-subprocess.rst | 38 +- .../docker/tutorial-quickstart-docker.rst | 3 +- 3 files changed, 347 insertions(+), 101 deletions(-) diff --git a/doc/source/docker/enable-tls.rst b/doc/source/docker/enable-tls.rst index a3645d22e35d..f7d9c3951cf9 100644 --- a/doc/source/docker/enable-tls.rst +++ b/doc/source/docker/enable-tls.rst @@ -2,10 +2,7 @@ Enable TLS for Secure Connections ================================= When operating in a production environment, it is strongly recommended to enable -Transport Layer Security (TLS) for each Flower Component to ensure secure communication. - -To enable TLS, you will need a PEM-encoded root certificate, a PEM-encoded private key -and a PEM-encoded certificate chain. +Transport Layer Security (TLS) for each Flower component to ensure secure communication. .. note:: @@ -14,91 +11,323 @@ and a PEM-encoded certificate chain. `__ page contains a section that will guide you through the process. -Because Flower containers, by default, run with a non-root user ``app``, the mounted -files and directories must have the proper permissions for the user ID ``49999``. - -For example, to change the user ID of all files in the ``certificates/`` directory, you -can run ``sudo chown -R 49999:49999 certificates/*``. - -If you later want to delete the directory, you can change the user ID back to the -current user ID by running ``sudo chown -R $USER:$(id -gn) certificates``. - -SuperLink ---------- - -Assuming all files we need are in the local ``certificates`` directory, we can use the -flag ``--volume`` to mount the local directory into the ``/app/certificates/`` directory -of the container: - -.. code-block:: bash - :substitutions: - - $ docker run --rm \ - --volume ./certificates/:/app/certificates/:ro \ - flwr/superlink:|stable_flwr_version| \ - --ssl-ca-certfile certificates/ca.crt \ - --ssl-certfile certificates/server.pem \ - --ssl-keyfile certificates/server.key \ - - -.. dropdown:: Understanding the command - - * ``docker run``: This tells Docker to run a container from an image. - * ``--rm``: Remove the container once it is stopped or the command exits. - * | ``--volume ./certificates/:/app/certificates/:ro``: Mount the ``certificates`` directory in - | the current working directory of the host machine as a read-only volume at the - | ``/app/certificates`` directory inside the container. - | - | This allows the container to access the TLS certificates that are stored in the certificates - | directory. - * | :substitution-code:`flwr/superlink:|stable_flwr_version|`: The name of the image to be run and the specific - | tag of the image. The tag :substitution-code:`|stable_flwr_version|` represents a specific version of the image. - * | ``--ssl-ca-certfile certificates/ca.crt``: Specify the location of the CA certificate file - | inside the container. - | - | The ``certificates/ca.crt`` file is a certificate that is used to verify the identity of the - | SuperLink. - * | ``--ssl-certfile certificates/server.pem``: Specify the location of the SuperLink's - | TLS certificate file inside the container. - | - | The ``certificates/server.pem`` file is used to identify the SuperLink and to encrypt the - | data that is transmitted over the network. - * | ``--ssl-keyfile certificates/server.key``: Specify the location of the SuperLink's - | TLS private key file inside the container. - | - | The ``certificates/server.key`` file is used to decrypt the data that is transmitted over - | the network. - -SuperNode ---------- - -Assuming that the ``ca.crt`` certificate already exists locally, we can use the flag -``--volume`` to mount the local certificate into the container's ``/app/`` directory. +.. note:: + + Because Flower containers, by default, run with a non-root user ``app``, the mounted + files and directories must have the proper permissions for the user ID ``49999``. + + For example, to change the user ID of all files in the ``certificates/`` directory, + you can run ``sudo chown -R 49999:49999 certificates/*``. + + If you later want to delete the directory, you can change the user ID back to the + current user ID by running ``sudo chown -R $USER:$(id -gn) certificates``. + +.. tab-set:: + + .. tab-item:: Enable TLS in Process Mode ``subprocess`` + + By default, the ServerApp is executed as a subprocess within the SuperLink Docker + container, and the ClientApp is run as a subprocess within the SuperNode Docker + container. You can learn more about the different process modes here: + :doc:`run-as-subprocess`. + + To enable TLS between the SuperLink and SuperNode, as well as between the SuperLink and the ``flwr`` + CLI, you will need two sets of PEM-encoded root certificates, private keys, and certificate chains. + + **SuperLink** + + Assuming all files we need are in the local ``superlink-certificates`` and + ``supernode-certificates`` directories, we can use the flag ``--volume`` to mount the + local directories into the SuperNode container: + + .. code-block:: bash + + $ docker run --rm \ + --volume ./superlink-certificates/:/app/certificates/:ro \ + \ + --ssl-ca-certfile certificates/ca.crt \ + --ssl-certfile certificates/server.pem \ + --ssl-keyfile certificates/server.key \ + + + .. dropdown:: Understanding the command + + * ``docker run``: This tells Docker to run a container from an image. + * ``--rm``: Remove the container once it is stopped or the command exits. + * | ``--volume ./superlink-certificates/:/app/certificates/:ro``: Mount the ``superlink-certificates`` + | directory in the current working directory of the host machine as a read-only volume + | at the ``/app/certificates`` directory inside the container. + | + | This allows the container to access the TLS certificates that are stored in the certificates + | directory. + * ````: The name of your SuperLink image to be run. + * | ``--ssl-ca-certfile certificates/ca.crt``: Specify the location of the CA certificate file + | inside the container. + | + | The ``certificates/ca.crt`` file is a certificate that is used to verify the identity of the + | SuperLink. + * | ``--ssl-certfile certificates/server.pem``: Specify the location of the SuperLink's + | TLS certificate file inside the container. + | + | The ``certificates/server.pem`` file is used to identify the SuperLink and to encrypt the + | data that is transmitted over the network. + * | ``--ssl-keyfile certificates/server.key``: Specify the location of the SuperLink's + | TLS private key file inside the container. + | + | The ``certificates/server.key`` file is used to decrypt the data that is transmitted over + | the network. + + **SuperNode** + + .. note:: + + If you're generating self-signed certificates and the ``ca.crt`` certificate or the + ``supernode-certificates`` directory doesn't exist on the SuperNode, you can copy it over + after the generation step. + + .. note:: + + Each SuperNode can have its own set of keys and certificates, or they can all share + the same set. + + .. code-block:: bash + + $ docker run --rm \ + --volume ./supernode-certificates/:/app/certificates/:ro \ + --volume ./superlink-certificates/ca.crt:/app/ca.crt/:ro \ + \ + --ssl-ca-certfile certificates/ca.crt \ + --ssl-certfile certificates/server.pem \ + --ssl-keyfile certificates/server.key \ + --root-certificates ca.crt \ + + + .. dropdown:: Understanding the command + + * ``docker run``: This tells Docker to run a container from an image. + * ``--rm``: Remove the container once it is stopped or the command exits. + * | ``--volume ./supernode-certificates/:/app/certificates/:ro``: Mount the ``supernode-certificates`` + | directory in the current working directory of the host machine as a read-only volume at the + | ``/app/certificates`` directory inside the container. + | + | This allows the container to access the TLS certificates that are stored in the certificates + | directory. + * | ``--volume ./superlink-certificates/ca.crt:/app/ca.crt/:ro``: Mount the ``ca.crt`` + | file from the ``superlink-certificates`` directory of the host machine as a read-only + | volume at the ``/app/ca.crt`` directory inside the container. + * ````: The name of your SuperNode image to be run. + * | ``--ssl-ca-certfile certificates/ca.crt``: Specify the location of the CA certificate file + | inside the container. + | + | The ``certificates/ca.crt`` file is a certificate that is used to verify the identity of the + | SuperNode. + * | ``--ssl-certfile certificates/server.pem``: Specify the location of the SuperNode's + | TLS certificate file inside the container. + | + | The ``certificates/server.pem`` file is used to identify the SuperNode and to encrypt the + | data that is transmitted over the network. + * | ``--ssl-keyfile certificates/server.key``: Specify the location of the SuperNode's + | TLS private key file inside the container. + | + | The ``certificates/server.key`` file is used to decrypt the data that is transmitted over + | the network. + * | ``--root-certificates ca.crt``: This specifies the location of the CA certificate file + | inside the container. + | + | The ``ca.crt`` file is used to verify the identity of the SuperLink. + + .. tab-item:: Enable TLS in Process Mode ``isolation`` + + In process mode ``isolation``, the Server and ClientApp run in their own processes. + Unlike in process mode ``subprocess``, the SuperLink or SuperNode does not attempt to + create the respective processes; instead, they must be created externally. + + It is possible to run only the SuperLink in process mode ``subprocess`` and the + SuperNode in process mode ``isolation``, or vice versa, or even both with process mode + ``isolation``. + + **SuperLink and ServerApp** + + To enable TLS between all Flower components, you will need two sets of PEM-encoded root + certificates, private keys, and certificate chains. + + Assuming all files we need are in the local ``superlink-certificates`` and + ``supernode-certificates`` directories, we can use the flag ``--volume`` to mount the + local directories into the SuperNode container: + + Start the SuperLink container: + + .. code-block:: bash + :substitutions: + + $ docker run --rm \ + --volume ./superlink-certificates/:/app/certificates/:ro \ + flwr/superlink:|stable_flwr_version| \ + --ssl-ca-certfile certificates/ca.crt \ + --ssl-certfile certificates/server.pem \ + --ssl-keyfile certificates/server.key \ + --isolation process \ + + + .. dropdown:: Understanding the command + + * ``docker run``: This tells Docker to run a container from an image. + * ``--rm``: Remove the container once it is stopped or the command exits. + * | ``--volume ./superlink-certificates/:/app/certificates/:ro``: Mount the + | ``superlink-certificates`` directory in the current working directory of the host + | machine as a read-only volume at the ``/app/certificates`` directory inside the container. + | + | This allows the container to access the TLS certificates that are stored in the certificates + | directory. + * | :substitution-code:`flwr/superlink:|stable_flwr_version|`: The name of the image to be run and the specific + | tag of the image. The tag :substitution-code:`|stable_flwr_version|` represents a specific version of the image. + * | ``--ssl-ca-certfile certificates/ca.crt``: Specify the location of the CA certificate file + | inside the container. + | + | The ``certificates/ca.crt`` file is a certificate that is used to verify the identity of the + | SuperLink. + * | ``--ssl-certfile certificates/server.pem``: Specify the location of the SuperLink's + | TLS certificate file inside the container. + | + | The ``certificates/server.pem`` file is used to identify the SuperLink and to encrypt the + | data that is transmitted over the network. + * | ``--ssl-keyfile certificates/server.key``: Specify the location of the SuperLink's + | TLS private key file inside the container. + | + | The ``certificates/server.key`` file is used to decrypt the data that is transmitted over + | the network. + * | ``--isolation process``: Tells the SuperLink that the ServerApp is created by separate + | independent process. The SuperLink does not attempt to create it. + + Start the ServerApp container: + + .. code-block:: bash + + $ docker run --rm \ + --volume ./superlink-certificates/ca.crt:/app/ca.crt:ro \ + \ + --root-certificates ca.crt \ + + + .. dropdown:: Understand the command + + * ``docker run``: This tells Docker to run a container from an image. + * ``--rm``: Remove the container once it is stopped or the command exits. + * | ``--volume ./superlink-certificates/ca.crt:/app/ca.crt:ro``: Mount the ``ca.crt`` file from + | the ``superlink-certificates`` directory of the host machine as a read-only volume at the + | ``/app/ca.crt`` directory inside the container. + * ````: The name of your ServerApp image to be run. + * | ``--root-certificates ca.crt``: This specifies the location of the CA + | certificate file inside the container. + | + | The ``ca.crt`` file is used to verify the identity of the SuperLink. + + **SuperNode and ClientApp** + + .. note:: + + If you're generating self-signed certificates and the ``ca.crt`` certificate or the + ``supernode-certificates`` directory doesn't exist on the SuperNode, you can copy it over + after the generation step. + + .. note:: + + Each SuperNode can have its own set of keys and certificates, or they can all share + the same set. + + Start the SuperNode container: + + .. code-block:: bash + :substitutions: + + $ docker run --rm \ + --volume ./supernode-certificates/:/app/certificates/:ro \ + --volume ./superlink-certificates/ca.crt:/app/ca.crt/:ro \ + flwr/supernode:|stable_flwr_version| \ + --ssl-ca-certfile=certificates/ca.crt \ + --ssl-certfile=certificates/server.pem \ + --ssl-keyfile=certificates/server.key \ + --root-certificates ca.crt \ + --isolation process \ + + + .. dropdown:: Understanding the command + + * ``docker run``: This tells Docker to run a container from an image. + * ``--rm``: Remove the container once it is stopped or the command exits. + * | ``--volume ./supernode-certificates/:/app/certificates/:ro``: Mount the ``supernode-certificates`` + | directory in the current working directory of the host machine as a read-only volume at the + | ``/app/certificates`` directory inside the container. + | + | This allows the container to access the TLS certificates that are stored in the certificates + | directory. + * | ``--volume ./superlink-certificates/ca.crt:/app/ca.crt/:ro``: Mount the ``ca.crt`` file from the + | ``superlink-certificates`` directory of the host machine as a read-only volume at the ``/app/ca.crt`` + | directory inside the container. + * | :substitution-code:`flwr/supernode:|stable_flwr_version|`: The name of the image to be run and the specific + | tag of the image. The tag :substitution-code:`|stable_flwr_version|` represents a specific version of the image. + * | ``--ssl-ca-certfile certificates/ca.crt``: Specify the location of the CA certificate file + | inside the container. + | + | The ``certificates/ca.crt`` file is a certificate that is used to verify the identity of the + | SuperNode. + * | ``--ssl-certfile certificates/server.pem``: Specify the location of the SuperNode's + | TLS certificate file inside the container. + | + | The ``certificates/server.pem`` file is used to identify the SuperNode and to encrypt the + | data that is transmitted over the network. + * | ``--ssl-keyfile certificates/server.key``: Specify the location of the SuperNode's + | TLS private key file inside the container. + | + | The ``certificates/server.key`` file is used to decrypt the data that is transmitted over + | the network. + * | ``--root-certificates ca.crt``: This specifies the location of the CA certificate file + | inside the container. + | + | The ``ca.crt`` file is used to verify the identity of the SuperLink. + * | ``--isolation process``: Tells the SuperNode that the ClientApp is created by separate + | independent process. The SuperNode does not attempt to create it. + + Start the ClientApp container: + + .. code-block:: bash + + $ docker run --rm \ + --volume ./supernode-certificates/ca.crt:/app/ca.crt:ro \ + \ + --root-certificates ca.crt \ + + + .. dropdown:: Understand the command + + * ``docker run``: This tells Docker to run a container from an image. + * ``--rm``: Remove the container once it is stopped or the command exits. + * | ``--volume ./supernode-certificates/ca.crt:/app/ca.crt:ro``: Mount the ``ca.crt`` file from + | the ``supernode-certificates`` directory of the host machine as a read-only volume at the + | ``/app/ca.crt`` directory inside the container. + * ````: The name of your ClientApp image to be run. + * | ``--root-certificates ca.crt``: This specifies the location of the CA + | certificate file inside the container. + | + | The ``ca.crt`` file is used to verify the identity of the SuperNode. + +Append the following lines to the end of the ``pyproject.toml`` file and save it: + +.. code-block:: toml + :caption: pyproject.toml + + [tool.flwr.federations.local-deployment-tls] + address = "127.0.0.1:9093" + root-certificates = "../superlink-certificates/ca.crt" + +The path of the ``root-certificates`` should be relative to the location of the +``pyproject.toml`` file. .. note:: - If you're generating self-signed certificates and the ``ca.crt`` certificate doesn't - exist on the SuperNode, you can copy it over after the generation step. - -.. code-block:: bash - :substitutions: - - $ docker run --rm \ - --volume ./ca.crt:/app/ca.crt/:ro \ - flwr/supernode:|stable_flwr_version| \ - --root-certificates ca.crt \ - - -.. dropdown:: Understanding the command - - * ``docker run``: This tells Docker to run a container from an image. - * ``--rm``: Remove the container once it is stopped or the command exits. - * | ``--volume ./ca.crt:/app/ca.crt/:ro``: Mount the ``ca.crt`` file from the - | current working directory of the host machine as a read-only volume at the ``/app/ca.crt`` - | directory inside the container. - * | :substitution-code:`flwr/supernode:|stable_flwr_version|`: The name of the image to be run and the specific - | tag of the image. The tag :substitution-code:`|stable_flwr_version|` represents a specific version of the image. - * | ``--root-certificates ca.crt``: This specifies the location of the CA certificate file - | inside the container. - | - | The ``ca.crt`` file is used to verify the identity of the SuperLink. + You can customize the string that follows ``tool.flwr.federations.`` to fit your + needs. However, please note that the string cannot contain a dot (``.``). + + In this example, ``local-deployment-tls`` has been used. Just remember to replace + ``local-deployment-tls`` with your chosen name in both the + ``tool.flwr.federations.`` string and the corresponding ``flwr run .`` command. diff --git a/doc/source/docker/run-as-subprocess.rst b/doc/source/docker/run-as-subprocess.rst index 946dfb0d9ec8..db36a25b80bb 100644 --- a/doc/source/docker/run-as-subprocess.rst +++ b/doc/source/docker/run-as-subprocess.rst @@ -1,15 +1,32 @@ Run ClientApp or ServerApp as a Subprocess ========================================== -By default, the ServerApp is executed as a subprocess within the SuperLink Docker -container, and the ClientApp is run as a subprocess within the SuperNode Docker -container, rather than using separate containers as shown in the -:doc:`tutorial-quickstart-docker` guide. +The SuperLink and SuperNode components support two distinct process modes, allowing for +flexible deployment and control: -This approach reduces the number of running containers, which can be beneficial for -environments with limited resources. However, it also means that the applications are -not isolated from their parent containers, which may introduce additional security -concerns. +1. Subprocess Mode: In this configuration (default), the SuperLink and SuperNode take + responsibility for launching the Server and ClientApp processes internally. This + differs from process mode ``isolation`` by using separate containers, as demonstrated + in the :doc:`tutorial-quickstart-docker` guide. + + This approach reduces the number of running containers, which can be beneficial for + environments with limited resources. However, it also means that the applications are + not isolated from their parent containers, which may introduce additional security + concerns. + +2. Isolation Mode: In this mode, the Server and ClientApp run in completely separate + processes. Unlike the alternative Subprocess mode, the SuperLink or SuperNode does + not attempt to create or manage these processes. Instead, they must be started + externally. + +These modes can be mixed for added flexibility. For instance, you can run the SuperLink +in ``subprocess`` mode while keeping the SuperNode in ``isolation`` mode, or vice versa. +It is also possible to configure both to run entirely in ``isolation`` mode, depending +on your specific requirements. + +To run the SuperLink and SuperNode in process mode ``isolation``, refer to the +:doc:`tutorial-quickstart-docker` guide. To run them in ``subprocess`` mode, follow the +instructions below. .. tab-set:: @@ -52,8 +69,7 @@ concerns. --detach \ flwr_supernode:0.0.1 \ --insecure \ - --superlink :9092 \ - --node-config + --superlink :9092 .. tab-item:: ServerApp @@ -77,7 +93,7 @@ concerns. ENTRYPOINT ["flower-superlink"] - 2. Next, build the superlink Docker image by running the following command in the + 2. Next, build the SuperLink Docker image by running the following command in the directory where Dockerfile is located: .. code-block:: shell diff --git a/doc/source/docker/tutorial-quickstart-docker.rst b/doc/source/docker/tutorial-quickstart-docker.rst index 624497dd2bd4..f5a829a25fdb 100644 --- a/doc/source/docker/tutorial-quickstart-docker.rst +++ b/doc/source/docker/tutorial-quickstart-docker.rst @@ -79,7 +79,8 @@ Open your terminal and run: * | ``--insecure``: This flag tells the container to operate in an insecure mode, allowing | unencrypted communication. * | ``--isolation process``: Tells the SuperLink that the ServerApp is created by separate - | independent process. The SuperLink does not attempt to create it. + | independent process. The SuperLink does not attempt to create it. You can learn more about + | the different process modes here: :doc:`run-as-subprocess`. Step 3: Start the SuperNodes ---------------------------- From 31cc8272207831a1d08f744da96fad58ecd560d7 Mon Sep 17 00:00:00 2001 From: Robert Steiner Date: Mon, 18 Nov 2024 22:51:47 +0100 Subject: [PATCH 07/11] Apply suggestions from code review Co-authored-by: Chong Shen Ng --- doc/source/docker/enable-tls.rst | 14 +++++------ doc/source/docker/run-as-subprocess.rst | 25 ++++++++++--------- .../docker/tutorial-quickstart-docker.rst | 6 +++-- 3 files changed, 24 insertions(+), 21 deletions(-) diff --git a/doc/source/docker/enable-tls.rst b/doc/source/docker/enable-tls.rst index f7d9c3951cf9..619c67b5afbe 100644 --- a/doc/source/docker/enable-tls.rst +++ b/doc/source/docker/enable-tls.rst @@ -24,7 +24,7 @@ Transport Layer Security (TLS) for each Flower component to ensure secure commun .. tab-set:: - .. tab-item:: Enable TLS in Process Mode ``subprocess`` + .. tab-item:: Enable TLS in Isolation Mode ``subprocess`` By default, the ServerApp is executed as a subprocess within the SuperLink Docker container, and the ClientApp is run as a subprocess within the SuperNode Docker @@ -136,15 +136,15 @@ Transport Layer Security (TLS) for each Flower component to ensure secure commun | | The ``ca.crt`` file is used to verify the identity of the SuperLink. - .. tab-item:: Enable TLS in Process Mode ``isolation`` + .. tab-item:: Enable TLS in Isolation Mode ``process`` - In process mode ``isolation``, the Server and ClientApp run in their own processes. - Unlike in process mode ``subprocess``, the SuperLink or SuperNode does not attempt to + In isolation mode ``process``, the ServerApp and ClientApp run in their own processes. + Unlike in isolation mode ``subprocess``, the SuperLink or SuperNode does not attempt to create the respective processes; instead, they must be created externally. - It is possible to run only the SuperLink in process mode ``subprocess`` and the - SuperNode in process mode ``isolation``, or vice versa, or even both with process mode - ``isolation``. + It is possible to run only the SuperLink in isolation mode ``subprocess`` and the + SuperNode in isolation mode ``process``, or vice versa, or even both with isolation mode + ``process``. **SuperLink and ServerApp** diff --git a/doc/source/docker/run-as-subprocess.rst b/doc/source/docker/run-as-subprocess.rst index db36a25b80bb..699416e0aec7 100644 --- a/doc/source/docker/run-as-subprocess.rst +++ b/doc/source/docker/run-as-subprocess.rst @@ -1,30 +1,30 @@ -Run ClientApp or ServerApp as a Subprocess +Run ServerApp or ClientApp as a Subprocess ========================================== -The SuperLink and SuperNode components support two distinct process modes, allowing for +The SuperLink and SuperNode components support two distinct isolation modes, allowing for flexible deployment and control: 1. Subprocess Mode: In this configuration (default), the SuperLink and SuperNode take - responsibility for launching the Server and ClientApp processes internally. This - differs from process mode ``isolation`` by using separate containers, as demonstrated + responsibility for launching the ServerApp and ClientApp processes internally. This + differs from the ``process`` isolation-mode which uses separate containers, as demonstrated in the :doc:`tutorial-quickstart-docker` guide. - This approach reduces the number of running containers, which can be beneficial for + Using the ``subprocess`` approach reduces the number of running containers, which can be beneficial for environments with limited resources. However, it also means that the applications are not isolated from their parent containers, which may introduce additional security concerns. -2. Isolation Mode: In this mode, the Server and ClientApp run in completely separate +2. Process Mode: In this mode, the ServerApp and ClientApps run in completely separate processes. Unlike the alternative Subprocess mode, the SuperLink or SuperNode does not attempt to create or manage these processes. Instead, they must be started externally. -These modes can be mixed for added flexibility. For instance, you can run the SuperLink -in ``subprocess`` mode while keeping the SuperNode in ``isolation`` mode, or vice versa. +Both modes can be mixed for added flexibility. For instance, you can run the SuperLink +in ``subprocess`` mode while keeping the SuperNode in ``process`` mode, or vice versa. It is also possible to configure both to run entirely in ``isolation`` mode, depending on your specific requirements. -To run the SuperLink and SuperNode in process mode ``isolation``, refer to the +To run the SuperLink and SuperNode in isolation mode ``process``, refer to the :doc:`tutorial-quickstart-docker` guide. To run them in ``subprocess`` mode, follow the instructions below. @@ -61,7 +61,8 @@ instructions below. **Run the ClientApp as a Subprocess** - Start the SuperNode and run the ClientApp as a subprocess: + Start the SuperNode and run the ClientApp as a subprocess (note that + the subprocess mode is the default, so you do not have to explicitly set the ``--isolation`` flag): .. code-block:: shell @@ -76,7 +77,7 @@ instructions below. **Prerequisites** 1. Before running the ServerApp as a subprocess, ensure that the FAB dependencies have - been installed in the SuperLink images. This can be done by extending the SuperLink + been installed in the SuperLink images. Similar to the SuperNode, this can be done by extending the SuperLink image: .. code-block:: dockerfile @@ -102,7 +103,7 @@ instructions below. **Run the ServerApp as a Subprocess** - Start the SuperLink and run the ServerApp as a subprocess: + Start the SuperLink and run the ServerApp as a subprocess (Like the SuperNode, the subprocess mode is the default mode when starting the SuperLink. You do not have to explicitly set the `--isolation` argument): .. code-block:: shell diff --git a/doc/source/docker/tutorial-quickstart-docker.rst b/doc/source/docker/tutorial-quickstart-docker.rst index f5a829a25fdb..c492dced5b4a 100644 --- a/doc/source/docker/tutorial-quickstart-docker.rst +++ b/doc/source/docker/tutorial-quickstart-docker.rst @@ -70,7 +70,7 @@ Open your terminal and run: * | ``-p 9091:9091 -p 9092:9092 -p 9093:9093``: Map port ``9091``, ``9092`` and ``9093`` of the | container to the same port of the host machine, allowing other services to access the | ServerAppIO API on ``http://localhost:9091``, the Fleet API on ``http://localhost:9092`` and - | the ExecAPI on ``http://localhost:9093``. + | the Exec API on ``http://localhost:9093``. * ``--network flwr-network``: Make the container join the network named ``flwr-network``. * ``--name superlink``: Assign the name ``superlink`` to the container. * ``--detach``: Run the container in the background, freeing up the terminal. @@ -123,7 +123,9 @@ Start two SuperNode containers. * | ``--node-config "partition-id=0 num-partitions=2"``: Set the partition ID to ``0`` and the | number of partitions to ``2`` for the SuperNode configuration. * | ``--clientappio-api-address 0.0.0.0:9094``: Set the address and port number that the - | SuperNode is listening on to communicate with the ClientApp. + | SuperNode is listening on to communicate with the ClientApp. If + | two SuperNodes are started on the same machine, set two different port numbers for each SuperNode. + | (E.g. In the next step, we set the second SuperNode container to listen on port 9095) * | ``--isolation process``: Tells the SuperNode that the ClientApp is created by separate | independent process. The SuperNode does not attempt to create it. From c2d59c1ab339984104da7f209ea65f84a24d478a Mon Sep 17 00:00:00 2001 From: Robert Steiner Date: Mon, 18 Nov 2024 22:54:25 +0100 Subject: [PATCH 08/11] formatting Signed-off-by: Robert Steiner --- doc/source/docker/enable-tls.rst | 4 ++-- doc/source/docker/run-as-subprocess.rst | 21 +++++++++---------- .../docker/tutorial-quickstart-docker.rst | 2 +- 3 files changed, 13 insertions(+), 14 deletions(-) diff --git a/doc/source/docker/enable-tls.rst b/doc/source/docker/enable-tls.rst index 619c67b5afbe..62afeee5ddff 100644 --- a/doc/source/docker/enable-tls.rst +++ b/doc/source/docker/enable-tls.rst @@ -24,7 +24,7 @@ Transport Layer Security (TLS) for each Flower component to ensure secure commun .. tab-set:: - .. tab-item:: Enable TLS in Isolation Mode ``subprocess`` + .. tab-item:: Isolation Mode ``subprocess`` By default, the ServerApp is executed as a subprocess within the SuperLink Docker container, and the ClientApp is run as a subprocess within the SuperNode Docker @@ -136,7 +136,7 @@ Transport Layer Security (TLS) for each Flower component to ensure secure commun | | The ``ca.crt`` file is used to verify the identity of the SuperLink. - .. tab-item:: Enable TLS in Isolation Mode ``process`` + .. tab-item:: Isolation Mode ``process`` In isolation mode ``process``, the ServerApp and ClientApp run in their own processes. Unlike in isolation mode ``subprocess``, the SuperLink or SuperNode does not attempt to diff --git a/doc/source/docker/run-as-subprocess.rst b/doc/source/docker/run-as-subprocess.rst index 699416e0aec7..2157c5210fa1 100644 --- a/doc/source/docker/run-as-subprocess.rst +++ b/doc/source/docker/run-as-subprocess.rst @@ -1,18 +1,18 @@ Run ServerApp or ClientApp as a Subprocess ========================================== -The SuperLink and SuperNode components support two distinct isolation modes, allowing for -flexible deployment and control: +The SuperLink and SuperNode components support two distinct isolation modes, allowing +for flexible deployment and control: 1. Subprocess Mode: In this configuration (default), the SuperLink and SuperNode take responsibility for launching the ServerApp and ClientApp processes internally. This - differs from the ``process`` isolation-mode which uses separate containers, as demonstrated - in the :doc:`tutorial-quickstart-docker` guide. + differs from the ``process`` isolation-mode which uses separate containers, as + demonstrated in the :doc:`tutorial-quickstart-docker` guide. - Using the ``subprocess`` approach reduces the number of running containers, which can be beneficial for - environments with limited resources. However, it also means that the applications are - not isolated from their parent containers, which may introduce additional security - concerns. + Using the ``subprocess`` approach reduces the number of running containers, which can + be beneficial for environments with limited resources. However, it also means that + the applications are not isolated from their parent containers, which may introduce + additional security concerns. 2. Process Mode: In this mode, the ServerApp and ClientApps run in completely separate processes. Unlike the alternative Subprocess mode, the SuperLink or SuperNode does @@ -21,8 +21,6 @@ flexible deployment and control: Both modes can be mixed for added flexibility. For instance, you can run the SuperLink in ``subprocess`` mode while keeping the SuperNode in ``process`` mode, or vice versa. -It is also possible to configure both to run entirely in ``isolation`` mode, depending -on your specific requirements. To run the SuperLink and SuperNode in isolation mode ``process``, refer to the :doc:`tutorial-quickstart-docker` guide. To run them in ``subprocess`` mode, follow the @@ -103,7 +101,8 @@ instructions below. **Run the ServerApp as a Subprocess** - Start the SuperLink and run the ServerApp as a subprocess (Like the SuperNode, the subprocess mode is the default mode when starting the SuperLink. You do not have to explicitly set the `--isolation` argument): + Start the SuperLink and run the ServerApp as a subprocess (Like the SuperNode, the subprocess mode + is the default mode when starting the SuperLink. You do not have to explicitly set the `--isolation` argument): .. code-block:: shell diff --git a/doc/source/docker/tutorial-quickstart-docker.rst b/doc/source/docker/tutorial-quickstart-docker.rst index c492dced5b4a..22be099aad61 100644 --- a/doc/source/docker/tutorial-quickstart-docker.rst +++ b/doc/source/docker/tutorial-quickstart-docker.rst @@ -123,7 +123,7 @@ Start two SuperNode containers. * | ``--node-config "partition-id=0 num-partitions=2"``: Set the partition ID to ``0`` and the | number of partitions to ``2`` for the SuperNode configuration. * | ``--clientappio-api-address 0.0.0.0:9094``: Set the address and port number that the - | SuperNode is listening on to communicate with the ClientApp. If + | SuperNode is listening on to communicate with the ClientApp. If | two SuperNodes are started on the same machine, set two different port numbers for each SuperNode. | (E.g. In the next step, we set the second SuperNode container to listen on port 9095) * | ``--isolation process``: Tells the SuperNode that the ClientApp is created by separate From c417e6c754009beccc984ce610103f2cd313d572 Mon Sep 17 00:00:00 2001 From: Robert Steiner Date: Tue, 19 Nov 2024 15:39:42 +0100 Subject: [PATCH 09/11] Apply suggestions from code review Co-authored-by: Chong Shen Ng --- doc/source/docker/tutorial-quickstart-docker.rst | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/doc/source/docker/tutorial-quickstart-docker.rst b/doc/source/docker/tutorial-quickstart-docker.rst index 22be099aad61..907d01e83a0a 100644 --- a/doc/source/docker/tutorial-quickstart-docker.rst +++ b/doc/source/docker/tutorial-quickstart-docker.rst @@ -228,7 +228,7 @@ required dependencies. * ``--network flwr-network``: Make the container join the network named ``flwr-network``. * ``--detach``: Run the container in the background, freeing up the terminal. * | ``--insecure``: This flag tells the container to operate in an insecure mode, allowing - | unencrypted communication. + | unencrypted communication. Secure connections will be added in future releases. * | ``flwr_clientapp:0.0.1``: This is the name of the image to be run and the specific tag | of the image. * | ``--clientappio-api-address supernode-1:9094``: Connect to the SuperNode's ClientAppIO @@ -318,7 +318,7 @@ ServerApp image and installs the required FAB dependencies. * | ``flwr_serverapp:0.0.1``: This is the name of the image to be run and the specific tag | of the image. * | ``--insecure``: This flag tells the container to operate in an insecure mode, allowing - | unencrypted communication. + | unencrypted communication. Secure connections will be added in future releases. * | ``--serverappio-api-address superlink:9091``: Connect to the SuperLink's ServerAppIO API | at the address ``superlink:9091``. @@ -354,13 +354,13 @@ Step 7: Update the Application partition_train_test = partition.train_test_split(test_size=0.2, seed=43) # ... -2. Stop the current ClientApp and ServerApp containers: +2. Stop the current ServerApp and ClientApp containers: .. code-block:: bash $ docker stop $(docker ps -a -q --filter ancestor=flwr_clientapp:0.0.1) serverapp -3. Rebuild ClientApp and ServerApp image: +3. Rebuild ServerApp and ClientApp images: .. note:: @@ -371,8 +371,8 @@ Step 7: Update the Application .. code-block:: bash - $ docker build -f clientapp.Dockerfile -t flwr_clientapp:0.0.1 . && \ - docker build -f serverapp.Dockerfile -t flwr_serverapp:0.0.1 . + $ docker build -f clientapp.Dockerfile -t flwr_serverapp:0.0.1 . && \ + docker build -f serverapp.Dockerfile -t flwr_clientapp:0.0.1 . 4. Launch two new ClientApp containers based on the newly built image: From f8d97c53705616094e71d26f8fe1c14c795a3ed0 Mon Sep 17 00:00:00 2001 From: Robert Steiner Date: Tue, 19 Nov 2024 15:58:24 +0100 Subject: [PATCH 10/11] Apply suggestions from code review Co-authored-by: Chong Shen Ng --- doc/source/docker/enable-tls.rst | 87 +++++--------------------------- 1 file changed, 12 insertions(+), 75 deletions(-) diff --git a/doc/source/docker/enable-tls.rst b/doc/source/docker/enable-tls.rst index 62afeee5ddff..6c65ef07d6b6 100644 --- a/doc/source/docker/enable-tls.rst +++ b/doc/source/docker/enable-tls.rst @@ -93,12 +93,8 @@ Transport Layer Security (TLS) for each Flower component to ensure secure commun .. code-block:: bash $ docker run --rm \ - --volume ./supernode-certificates/:/app/certificates/:ro \ --volume ./superlink-certificates/ca.crt:/app/ca.crt/:ro \ \ - --ssl-ca-certfile certificates/ca.crt \ - --ssl-certfile certificates/server.pem \ - --ssl-keyfile certificates/server.key \ --root-certificates ca.crt \ @@ -106,31 +102,10 @@ Transport Layer Security (TLS) for each Flower component to ensure secure commun * ``docker run``: This tells Docker to run a container from an image. * ``--rm``: Remove the container once it is stopped or the command exits. - * | ``--volume ./supernode-certificates/:/app/certificates/:ro``: Mount the ``supernode-certificates`` - | directory in the current working directory of the host machine as a read-only volume at the - | ``/app/certificates`` directory inside the container. - | - | This allows the container to access the TLS certificates that are stored in the certificates - | directory. * | ``--volume ./superlink-certificates/ca.crt:/app/ca.crt/:ro``: Mount the ``ca.crt`` | file from the ``superlink-certificates`` directory of the host machine as a read-only | volume at the ``/app/ca.crt`` directory inside the container. * ````: The name of your SuperNode image to be run. - * | ``--ssl-ca-certfile certificates/ca.crt``: Specify the location of the CA certificate file - | inside the container. - | - | The ``certificates/ca.crt`` file is a certificate that is used to verify the identity of the - | SuperNode. - * | ``--ssl-certfile certificates/server.pem``: Specify the location of the SuperNode's - | TLS certificate file inside the container. - | - | The ``certificates/server.pem`` file is used to identify the SuperNode and to encrypt the - | data that is transmitted over the network. - * | ``--ssl-keyfile certificates/server.key``: Specify the location of the SuperNode's - | TLS private key file inside the container. - | - | The ``certificates/server.key`` file is used to decrypt the data that is transmitted over - | the network. * | ``--root-certificates ca.crt``: This specifies the location of the CA certificate file | inside the container. | @@ -151,11 +126,10 @@ Transport Layer Security (TLS) for each Flower component to ensure secure commun To enable TLS between all Flower components, you will need two sets of PEM-encoded root certificates, private keys, and certificate chains. - Assuming all files we need are in the local ``superlink-certificates`` and - ``supernode-certificates`` directories, we can use the flag ``--volume`` to mount the - local directories into the SuperNode container: + Assuming all files we need are in the local ``superlink-certificates`` + directory, we can use the flag ``--volume`` to mount the + local directory into the SuperLink container: - Start the SuperLink container: .. code-block:: bash :substitutions: @@ -204,35 +178,29 @@ Transport Layer Security (TLS) for each Flower component to ensure secure commun .. code-block:: bash $ docker run --rm \ - --volume ./superlink-certificates/ca.crt:/app/ca.crt:ro \ \ - --root-certificates ca.crt \ + --insecure \ .. dropdown:: Understand the command * ``docker run``: This tells Docker to run a container from an image. * ``--rm``: Remove the container once it is stopped or the command exits. - * | ``--volume ./superlink-certificates/ca.crt:/app/ca.crt:ro``: Mount the ``ca.crt`` file from - | the ``superlink-certificates`` directory of the host machine as a read-only volume at the - | ``/app/ca.crt`` directory inside the container. * ````: The name of your ServerApp image to be run. - * | ``--root-certificates ca.crt``: This specifies the location of the CA - | certificate file inside the container. - | - | The ``ca.crt`` file is used to verify the identity of the SuperLink. + * | ``--insecure``: This flag tells the container to operate in an insecure mode, allowing + | unencrypted communication. Secure connections will be added in future releases. **SuperNode and ClientApp** .. note:: - If you're generating self-signed certificates and the ``ca.crt`` certificate or the - ``supernode-certificates`` directory doesn't exist on the SuperNode, you can copy it over + If you're generating self-signed certificates and the ``ca.crt`` certificate + doesn't exist on the SuperNode, you can copy it over after the generation step. .. note:: - Each SuperNode can have its own set of keys and certificates, or they can all share + Each SuperNode can have its own set of certificates, or they can all share the same set. Start the SuperNode container: @@ -241,12 +209,8 @@ Transport Layer Security (TLS) for each Flower component to ensure secure commun :substitutions: $ docker run --rm \ - --volume ./supernode-certificates/:/app/certificates/:ro \ --volume ./superlink-certificates/ca.crt:/app/ca.crt/:ro \ flwr/supernode:|stable_flwr_version| \ - --ssl-ca-certfile=certificates/ca.crt \ - --ssl-certfile=certificates/server.pem \ - --ssl-keyfile=certificates/server.key \ --root-certificates ca.crt \ --isolation process \ @@ -255,32 +219,11 @@ Transport Layer Security (TLS) for each Flower component to ensure secure commun * ``docker run``: This tells Docker to run a container from an image. * ``--rm``: Remove the container once it is stopped or the command exits. - * | ``--volume ./supernode-certificates/:/app/certificates/:ro``: Mount the ``supernode-certificates`` - | directory in the current working directory of the host machine as a read-only volume at the - | ``/app/certificates`` directory inside the container. - | - | This allows the container to access the TLS certificates that are stored in the certificates - | directory. * | ``--volume ./superlink-certificates/ca.crt:/app/ca.crt/:ro``: Mount the ``ca.crt`` file from the | ``superlink-certificates`` directory of the host machine as a read-only volume at the ``/app/ca.crt`` | directory inside the container. * | :substitution-code:`flwr/supernode:|stable_flwr_version|`: The name of the image to be run and the specific | tag of the image. The tag :substitution-code:`|stable_flwr_version|` represents a specific version of the image. - * | ``--ssl-ca-certfile certificates/ca.crt``: Specify the location of the CA certificate file - | inside the container. - | - | The ``certificates/ca.crt`` file is a certificate that is used to verify the identity of the - | SuperNode. - * | ``--ssl-certfile certificates/server.pem``: Specify the location of the SuperNode's - | TLS certificate file inside the container. - | - | The ``certificates/server.pem`` file is used to identify the SuperNode and to encrypt the - | data that is transmitted over the network. - * | ``--ssl-keyfile certificates/server.key``: Specify the location of the SuperNode's - | TLS private key file inside the container. - | - | The ``certificates/server.key`` file is used to decrypt the data that is transmitted over - | the network. * | ``--root-certificates ca.crt``: This specifies the location of the CA certificate file | inside the container. | @@ -293,23 +236,17 @@ Transport Layer Security (TLS) for each Flower component to ensure secure commun .. code-block:: bash $ docker run --rm \ - --volume ./supernode-certificates/ca.crt:/app/ca.crt:ro \ \ - --root-certificates ca.crt \ + --insecure \ .. dropdown:: Understand the command * ``docker run``: This tells Docker to run a container from an image. * ``--rm``: Remove the container once it is stopped or the command exits. - * | ``--volume ./supernode-certificates/ca.crt:/app/ca.crt:ro``: Mount the ``ca.crt`` file from - | the ``supernode-certificates`` directory of the host machine as a read-only volume at the - | ``/app/ca.crt`` directory inside the container. * ````: The name of your ClientApp image to be run. - * | ``--root-certificates ca.crt``: This specifies the location of the CA - | certificate file inside the container. - | - | The ``ca.crt`` file is used to verify the identity of the SuperNode. + * | ``--insecure``: This flag tells the container to operate in an insecure mode, allowing + | unencrypted communication. Secure connections will be added in future releases. Append the following lines to the end of the ``pyproject.toml`` file and save it: From 5a2ee2940633b3f855ec6c85c859d7f7d58be833 Mon Sep 17 00:00:00 2001 From: Robert Steiner Date: Tue, 19 Nov 2024 16:26:15 +0100 Subject: [PATCH 11/11] updates Signed-off-by: Robert Steiner --- doc/source/conf.py | 4 +- doc/source/docker/enable-tls.rst | 40 ++-- doc/source/docker/run-as-subprocess.rst | 55 +++--- .../docker/tutorial-quickstart-docker.rst | 175 +++++++++--------- 4 files changed, 130 insertions(+), 144 deletions(-) diff --git a/doc/source/conf.py b/doc/source/conf.py index 2eb1d7585a22..56324ba113fb 100644 --- a/doc/source/conf.py +++ b/doc/source/conf.py @@ -90,10 +90,10 @@ author = "The Flower Authors" # The full version of the next release, including alpha/beta/rc tags -release = "1.13.0" +release = "unstable" # The current released version rst_prolog = """ -.. |stable_flwr_version| replace:: 1.12.0 +.. |stable_flwr_version| replace:: unstable .. |stable_flwr_superlink_docker_digest| replace:: 4b317d5b6030710b476f4dbfab2c3a33021ad40a0fcfa54d7edd45e0c51d889c .. |ubuntu_version| replace:: 24.04 .. |setuptools_version| replace:: 70.3.0 diff --git a/doc/source/docker/enable-tls.rst b/doc/source/docker/enable-tls.rst index 6c65ef07d6b6..eaa853298439 100644 --- a/doc/source/docker/enable-tls.rst +++ b/doc/source/docker/enable-tls.rst @@ -32,13 +32,12 @@ Transport Layer Security (TLS) for each Flower component to ensure secure commun :doc:`run-as-subprocess`. To enable TLS between the SuperLink and SuperNode, as well as between the SuperLink and the ``flwr`` - CLI, you will need two sets of PEM-encoded root certificates, private keys, and certificate chains. + CLI, you will need a PEM-encoded root certificate, private key, and certificate chain. **SuperLink** - Assuming all files we need are in the local ``superlink-certificates`` and - ``supernode-certificates`` directories, we can use the flag ``--volume`` to mount the - local directories into the SuperNode container: + Assuming all files we need are in the local ``superlink-certificates`` directory, + we can use the flag ``--volume`` to mount the local directories into the SuperLink container: .. code-block:: bash @@ -81,14 +80,8 @@ Transport Layer Security (TLS) for each Flower component to ensure secure commun .. note:: - If you're generating self-signed certificates and the ``ca.crt`` certificate or the - ``supernode-certificates`` directory doesn't exist on the SuperNode, you can copy it over - after the generation step. - - .. note:: - - Each SuperNode can have its own set of keys and certificates, or they can all share - the same set. + If you're generating self-signed certificates and the ``ca.crt`` certificate doesn't + exist on the SuperNode, you can copy it over after the generation step. .. code-block:: bash @@ -123,12 +116,11 @@ Transport Layer Security (TLS) for each Flower component to ensure secure commun **SuperLink and ServerApp** - To enable TLS between all Flower components, you will need two sets of PEM-encoded root - certificates, private keys, and certificate chains. + To enable TLS between the SuperLink and SuperNode, as well as between the SuperLink and the ``flwr`` + CLI, you will need a PEM-encoded root certificate, private key, and certificate chain. - Assuming all files we need are in the local ``superlink-certificates`` - directory, we can use the flag ``--volume`` to mount the - local directory into the SuperLink container: + Assuming all files we need are in the local ``superlink-certificates`` directory, we can + use the flag ``--volume`` to mount the local directory into the SuperLink container: .. code-block:: bash @@ -188,20 +180,14 @@ Transport Layer Security (TLS) for each Flower component to ensure secure commun * ``--rm``: Remove the container once it is stopped or the command exits. * ````: The name of your ServerApp image to be run. * | ``--insecure``: This flag tells the container to operate in an insecure mode, allowing - | unencrypted communication. Secure connections will be added in future releases. + | unencrypted communication. Secure connections will be added in future releases. **SuperNode and ClientApp** .. note:: - If you're generating self-signed certificates and the ``ca.crt`` certificate - doesn't exist on the SuperNode, you can copy it over - after the generation step. - - .. note:: - - Each SuperNode can have its own set of certificates, or they can all share - the same set. + If you're generating self-signed certificates and the ``ca.crt`` certificate doesn't + exist on the SuperNode, you can copy it over after the generation step. Start the SuperNode container: @@ -246,7 +232,7 @@ Transport Layer Security (TLS) for each Flower component to ensure secure commun * ``--rm``: Remove the container once it is stopped or the command exits. * ````: The name of your ClientApp image to be run. * | ``--insecure``: This flag tells the container to operate in an insecure mode, allowing - | unencrypted communication. Secure connections will be added in future releases. + | unencrypted communication. Secure connections will be added in future releases. Append the following lines to the end of the ``pyproject.toml`` file and save it: diff --git a/doc/source/docker/run-as-subprocess.rst b/doc/source/docker/run-as-subprocess.rst index 2157c5210fa1..c9b8404820b7 100644 --- a/doc/source/docker/run-as-subprocess.rst +++ b/doc/source/docker/run-as-subprocess.rst @@ -28,86 +28,85 @@ instructions below. .. tab-set:: - .. tab-item:: ClientApp + .. tab-item:: ServerApp **Prerequisites** - 1. Before running the ClientApp as a subprocess, ensure that the FAB dependencies have - been installed in the SuperNode images. This can be done by extending the SuperNode - image: + 1. Before running the ServerApp as a subprocess, ensure that the FAB dependencies have + been installed in the SuperLink images. This can be done by extending the SuperLink image: .. code-block:: dockerfile - :caption: supernode.Dockerfile + :caption: superlink.Dockerfile :linenos: :substitutions: - FROM flwr/supernode:|stable_flwr_version| + FROM flwr/superlink:|stable_flwr_version| WORKDIR /app COPY pyproject.toml . RUN sed -i 's/.*flwr\[simulation\].*//' pyproject.toml \ && python -m pip install -U --no-cache-dir . - ENTRYPOINT ["flower-supernode"] + ENTRYPOINT ["flower-superlink"] - 2. Next, build the SuperNode Docker image by running the following command in the + 2. Next, build the SuperLink Docker image by running the following command in the directory where Dockerfile is located: .. code-block:: shell - $ docker build -f supernode.Dockerfile -t flwr_supernode:0.0.1 . + $ docker build -f superlink.Dockerfile -t flwr_superlink:0.0.1 . - **Run the ClientApp as a Subprocess** + **Run the ServerApp as a Subprocess** - Start the SuperNode and run the ClientApp as a subprocess (note that + Start the SuperLink and run the ServerApp as a subprocess (note that the subprocess mode is the default, so you do not have to explicitly set the ``--isolation`` flag): .. code-block:: shell $ docker run --rm \ + -p 9091:9091 -p 9092:9092 -p 9093:9093 \ --detach \ - flwr_supernode:0.0.1 \ - --insecure \ - --superlink :9092 + flwr_superlink:0.0.1 \ + --insecure - .. tab-item:: ServerApp + .. tab-item:: ClientApp **Prerequisites** - 1. Before running the ServerApp as a subprocess, ensure that the FAB dependencies have - been installed in the SuperLink images. Similar to the SuperNode, this can be done by extending the SuperLink + 1. Before running the ClientApp as a subprocess, ensure that the FAB dependencies have + been installed in the SuperNode images. This can be done by extending the SuperNode image: .. code-block:: dockerfile - :caption: superlink.Dockerfile + :caption: supernode.Dockerfile :linenos: :substitutions: - FROM flwr/superlink:|stable_flwr_version| + FROM flwr/supernode:|stable_flwr_version| WORKDIR /app COPY pyproject.toml . RUN sed -i 's/.*flwr\[simulation\].*//' pyproject.toml \ && python -m pip install -U --no-cache-dir . - ENTRYPOINT ["flower-superlink"] + ENTRYPOINT ["flower-supernode"] - 2. Next, build the SuperLink Docker image by running the following command in the + 2. Next, build the SuperNode Docker image by running the following command in the directory where Dockerfile is located: .. code-block:: shell - $ docker build -f superlink.Dockerfile -t flwr_superlink:0.0.1 . + $ docker build -f supernode.Dockerfile -t flwr_supernode:0.0.1 . - **Run the ServerApp as a Subprocess** + **Run the ClientApp as a Subprocess** - Start the SuperLink and run the ServerApp as a subprocess (Like the SuperNode, the subprocess mode - is the default mode when starting the SuperLink. You do not have to explicitly set the `--isolation` argument): + Start the SuperNode and run the ClientApp as a subprocess (note that + the subprocess mode is the default, so you do not have to explicitly set the ``--isolation`` flag): .. code-block:: shell $ docker run --rm \ - -p 9091:9091 -p 9092:9092 -p 9093:9093 \ --detach \ - flwr_superlink:0.0.1 \ - --insecure + flwr_supernode:0.0.1 \ + --insecure \ + --superlink :9092 diff --git a/doc/source/docker/tutorial-quickstart-docker.rst b/doc/source/docker/tutorial-quickstart-docker.rst index 907d01e83a0a..3bc311923f8f 100644 --- a/doc/source/docker/tutorial-quickstart-docker.rst +++ b/doc/source/docker/tutorial-quickstart-docker.rst @@ -146,35 +146,35 @@ Start two SuperNode containers. --clientappio-api-address 0.0.0.0:9095 \ --isolation process -Step 4: Start the ClientApp ---------------------------- +Step 4: Start a ServerApp +------------------------- -The ClientApp Docker image comes with a pre-installed version of Flower and serves as a -base for building your own ClientApp image. In order to install the FAB dependencies, -you will need to create a Dockerfile that extends the ClientApp image and installs the +The ServerApp Docker image comes with a pre-installed version of Flower and serves as a +base for building your own ServerApp image. In order to install the FAB dependencies, +you will need to create a Dockerfile that extends the ServerApp image and installs the required dependencies. -1. Create a ClientApp Dockerfile called ``clientapp.Dockerfile`` and paste the following - code into it: +1. Create a ServerApp Dockerfile called ``serverapp.Dockerfile`` and paste the following + code in: .. code-block:: dockerfile - :caption: clientapp.Dockerfile - :linenos: + :caption: serverapp.Dockerfile :substitutions: - FROM flwr/clientapp:|stable_flwr_version| + FROM flwr/serverapp:|stable_flwr_version| WORKDIR /app + COPY pyproject.toml . RUN sed -i 's/.*flwr\[simulation\].*//' pyproject.toml \ - && python -m pip install -U --no-cache-dir . + && python -m pip install -U --no-cache-dir . - ENTRYPOINT ["flwr-clientapp"] + ENTRYPOINT ["flwr-serverapp"] .. dropdown:: Understand the Dockerfile - * | :substitution-code:`FROM flwr/clientapp:|stable_flwr_version|`: This line specifies that the Docker image - | to be built from is the ``flwr/clientapp`` image, version :substitution-code:`|stable_flwr_version|`. + * | :substitution-code:`FROM flwr/serverapp:|stable_flwr_version|`: This line specifies that the Docker image + | to be built from is the ``flwr/serverapp`` image, version :substitution-code:`|stable_flwr_version|`. * | ``WORKDIR /app``: Set the working directory for the container to ``/app``. | Any subsequent commands that reference a directory will be relative to this directory. * | ``COPY pyproject.toml .``: Copy the ``pyproject.toml`` file @@ -186,7 +186,7 @@ required dependencies. | | The ``-U`` flag indicates that any existing packages should be upgraded, and | ``--no-cache-dir`` prevents pip from using the cache to speed up the installation. - * | ``ENTRYPOINT ["flwr-clientapp"]``: Set the command ``flwr-clientapp`` to be + * | ``ENTRYPOINT ["flwr-serverapp"]``: Set the command ``flwr-serverapp`` to be | the default command run when the container is started. .. important:: @@ -197,84 +197,69 @@ required dependencies. ``flwr`` dependency is removed from the ``pyproject.toml`` after it has been copied into the Docker image (see line 5). -2. Next, build the ClientApp Docker image by running the following command in the - directory where the Dockerfile is located: +2. Afterward, in the directory that holds the Dockerfile, execute this Docker command to + build the ServerApp image: .. code-block:: bash - $ docker build -f clientapp.Dockerfile -t flwr_clientapp:0.0.1 . - - .. note:: - - The image name was set as ``flwr_clientapp`` with the tag ``0.0.1``. Remember - that these values are merely examples, and you can customize them according to - your requirements. + $ docker build -f serverapp.Dockerfile -t flwr_serverapp:0.0.1 . -3. Start the first ClientApp container: +3. Start the ServerApp container: .. code-block:: bash $ docker run --rm \ --network flwr-network \ + --name serverapp \ --detach \ - flwr_clientapp:0.0.1 \ + flwr_serverapp:0.0.1 \ --insecure \ - --clientappio-api-address supernode-1:9094 + --serverappio-api-address superlink:9091 .. dropdown:: Understand the command * ``docker run``: This tells Docker to run a container from an image. * ``--rm``: Remove the container once it is stopped or the command exits. * ``--network flwr-network``: Make the container join the network named ``flwr-network``. + * ``--name serverapp``: Assign the name ``serverapp`` to the container. * ``--detach``: Run the container in the background, freeing up the terminal. + * | ``flwr_serverapp:0.0.1``: This is the name of the image to be run and the specific tag + | of the image. * | ``--insecure``: This flag tells the container to operate in an insecure mode, allowing | unencrypted communication. Secure connections will be added in future releases. - * | ``flwr_clientapp:0.0.1``: This is the name of the image to be run and the specific tag - | of the image. - * | ``--clientappio-api-address supernode-1:9094``: Connect to the SuperNode's ClientAppIO - | API at the address ``supernode-1:9094``. - -4. Start the second ClientApp container: - - .. code-block:: shell - - $ docker run --rm \ - --network flwr-network \ - --detach \ - flwr_clientapp:0.0.1 \ - --insecure \ - --clientappio-api-address supernode-2:9095 + * | ``--serverappio-api-address superlink:9091``: Connect to the SuperLink's ServerAppIO API + | at the address ``superlink:9091``. -Step 5: Start a ServerApp -------------------------- +Step 5: Start the ClientApp +--------------------------- -The procedure for building and running a ServerApp image is almost identical to the -ClientApp image. +The procedure for building and running a ClientApp image is almost identical to the +ServerApp image. -Similar to the ClientApp image, you will need to create a Dockerfile that extends the -ServerApp image and installs the required FAB dependencies. +Similar to the ServerApp image, you will need to create a Dockerfile that extends the +ClientApp image and installs the required FAB dependencies. -1. Create a ServerApp Dockerfile called ``serverapp.Dockerfile`` and paste the following - code in: +1. Create a ClientApp Dockerfile called ``clientapp.Dockerfile`` and paste the following + code into it: .. code-block:: dockerfile - :caption: serverapp.Dockerfile + :caption: clientapp.Dockerfile + :linenos: :substitutions: - FROM flwr/serverapp:|stable_flwr_version| + FROM flwr/clientapp:|stable_flwr_version| WORKDIR /app - COPY pyproject.toml . RUN sed -i 's/.*flwr\[simulation\].*//' pyproject.toml \ - && python -m pip install -U --no-cache-dir . + && python -m pip install -U --no-cache-dir . - ENTRYPOINT ["flwr-serverapp"] + ENTRYPOINT ["flwr-clientapp"] .. dropdown:: Understand the Dockerfile - * | :substitution-code:`FROM flwr/serverapp:|stable_flwr_version|`: This line specifies that the Docker image - | to be built from is the ``flwr/serverapp`` image, version :substitution-code:`|stable_flwr_version|`. + * | :substitution-code:`FROM flwr/clientapp:|stable_flwr_version|`: This line specifies that the Docker image + | to be built from is the ``flwr/clientapp`` image, version :substitution-code:`|stable_flwr_version|`. * | ``WORKDIR /app``: Set the working directory for the container to ``/app``. | Any subsequent commands that reference a directory will be relative to this directory. * | ``COPY pyproject.toml .``: Copy the ``pyproject.toml`` file @@ -286,41 +271,56 @@ ServerApp image and installs the required FAB dependencies. | | The ``-U`` flag indicates that any existing packages should be upgraded, and | ``--no-cache-dir`` prevents pip from using the cache to speed up the installation. - * | ``ENTRYPOINT ["flwr-serverapp"]``: Set the command ``flwr-serverapp`` to be + * | ``ENTRYPOINT ["flwr-clientapp"]``: Set the command ``flwr-clientapp`` to be | the default command run when the container is started. -2. Afterward, in the directory that holds the Dockerfile, execute this Docker command to - build the ServerApp image: +2. Next, build the ClientApp Docker image by running the following command in the + directory where the Dockerfile is located: .. code-block:: bash - $ docker build -f serverapp.Dockerfile -t flwr_serverapp:0.0.1 . + $ docker build -f clientapp.Dockerfile -t flwr_clientapp:0.0.1 . -3. Start the ServerApp container: + .. note:: + + The image name was set as ``flwr_clientapp`` with the tag ``0.0.1``. Remember + that these values are merely examples, and you can customize them according to + your requirements. + +3. Start the first ClientApp container: .. code-block:: bash $ docker run --rm \ --network flwr-network \ - --name serverapp \ --detach \ - flwr_serverapp:0.0.1 \ + flwr_clientapp:0.0.1 \ --insecure \ - --serverappio-api-address superlink:9091 + --clientappio-api-address supernode-1:9094 .. dropdown:: Understand the command * ``docker run``: This tells Docker to run a container from an image. * ``--rm``: Remove the container once it is stopped or the command exits. * ``--network flwr-network``: Make the container join the network named ``flwr-network``. - * ``--name serverapp``: Assign the name ``serverapp`` to the container. * ``--detach``: Run the container in the background, freeing up the terminal. - * | ``flwr_serverapp:0.0.1``: This is the name of the image to be run and the specific tag - | of the image. * | ``--insecure``: This flag tells the container to operate in an insecure mode, allowing | unencrypted communication. Secure connections will be added in future releases. - * | ``--serverappio-api-address superlink:9091``: Connect to the SuperLink's ServerAppIO API - | at the address ``superlink:9091``. + * | ``flwr_clientapp:0.0.1``: This is the name of the image to be run and the specific tag + | of the image. + * | ``--clientappio-api-address supernode-1:9094``: Connect to the SuperNode's ClientAppIO + | API at the address ``supernode-1:9094``. + +4. Start the second ClientApp container: + + .. code-block:: shell + + $ docker run --rm \ + --network flwr-network \ + --detach \ + flwr_clientapp:0.0.1 \ + --insecure \ + --clientappio-api-address supernode-2:9095 Step 6: Run the Quickstart Project ---------------------------------- @@ -356,47 +356,48 @@ Step 7: Update the Application 2. Stop the current ServerApp and ClientApp containers: - .. code-block:: bash - - $ docker stop $(docker ps -a -q --filter ancestor=flwr_clientapp:0.0.1) serverapp - -3. Rebuild ServerApp and ClientApp images: - .. note:: If you have modified the dependencies listed in your ``pyproject.toml`` file, it is essential to rebuild images. - If you haven't made any changes, you can skip this step. + If you haven’t made any changes, you can skip steps 2 through 4. + + .. code-block:: bash + + $ docker stop $(docker ps -a -q --filter ancestor=flwr_clientapp:0.0.1) serverapp + +3. Rebuild ServerApp and ClientApp images: .. code-block:: bash - $ docker build -f clientapp.Dockerfile -t flwr_serverapp:0.0.1 . && \ - docker build -f serverapp.Dockerfile -t flwr_clientapp:0.0.1 . + $ docker build -f clientapp.Dockerfile -t flwr_clientapp:0.0.1 . && \ + docker build -f serverapp.Dockerfile -t flwr_serverapp:0.0.1 . -4. Launch two new ClientApp containers based on the newly built image: +4. Launch one new ServerApp and two new ClientApp containers based on the newly built + image: .. code-block:: bash $ docker run --rm \ --network flwr-network \ + --name serverapp \ --detach \ - flwr_clientapp:0.0.1 \ + flwr_serverapp:0.0.1 \ --insecure \ - --clientappio-api-address supernode-1:9094 + --serverappio-api-address superlink:9091 $ docker run --rm \ --network flwr-network \ --detach \ - flwr_clientapp:0.0.1 \ + flwr_clientapp:0.0.1 \ --insecure \ - --clientappio-api-address supernode-2:9095 + --clientappio-api-address supernode-1:9094 $ docker run --rm \ --network flwr-network \ - --name serverapp \ --detach \ - flwr_serverapp:0.0.1 \ + flwr_clientapp:0.0.1 \ --insecure \ - ----serverappio-api-address superlink:9091 + --clientappio-api-address supernode-2:9095 5. Run the updated project: