Memory Usage Issues with Local Development Mode Using Docker for FEDn

[kamalparsapu]

with tags like this :Severity

• Critical/Blocker (select if the issue makes the application unusable or causes serious data loss)
• High (select if the issue affects a major feature and there is no workaround or the available workaround is very complex)
• Medium (select if the issue affects a minor feature or affects a major feature but has an easy enough workaround to not cause any major inconvenience)
• Low (select if the issue doesn't significantly affect the user experience, like minor visual bugs)

Describe the bug
I am experiencing significant memory usage issues when deploying the Local Development Mode for FEDn using Docker and Docker Compose. Running the API server, combiner, and two clients consumes a considerable amount of memory. Adding a third client spikes the memory usage significantly, requiring an additional 18GB and exhausting my system's memory capacity. In contrast, FEDn Studio does not exhibit the same scalability issues.

Environment:

• Ubuntu 20.04
• Docker 20.10.7

Reproduction Steps
steps to reproduce the behavior:

1. Install FEDn and generate package.tgz and seed.tgz files.
2. Start the pseudo-distributed FEDn network using the following command:

docker compose
-f ../../docker-compose.yaml
-f docker-compose.override.yaml
up
3 .Upload the package and seed model to the FEDn controller using the APIClient in Python:
from fedn import APIClient
client = APIClient(host="localhost", port=8092)
client.set_active_package("package.tgz", helper="numpyhelper")
client.set_active_model("seed.npz")
client.start_session()
Expected behavior
Expected behavior
I expected the memory usage to be consistent and manageable across different numbers of clients, similar to how FEDn Studio handles it without significant increases in memory consumption.

Screenshots

Contact Details
Please contact me at parsapu.raj@mantratec.com if you need any further details regarding this issue.

[Wrede]

Thanks for reporting the issue. Are you using the mnist-pytorch example without any modifications to the Dockerfile?
Because the image size should be similar to this:

[kamalparsapu]

what are the modifications i want to make to the dockerfile? can you please explain me the step by step

…

On Mon, May 20, 2024 at 2:48 PM Fredrik Wrede ***@***.***> wrote: Thanks for reporting the issue. Are you using the mnist-pytorch example without any modifications to the Dockerfile? Because the image size should be similar to this: image.png (view on web) <https://github.com/scaleoutsystems/fedn/assets/11618308/8437204d-67f7-42fd-a9b4-afdb3dbedc35> — Reply to this email directly, view it on GitHub <#610 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AYDBF7JEKWBIG6OFA2C3I7TZDG5VVAVCNFSM6AAAAABH66GKNSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCMRQGAZTGMJZGQ> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[Wrede]

It was simply a question if you have made any modifications, not an indication that you need to modify anything. I'm trying to figure out why you have so large images. Can you please run "ls -shl" at the root of your cloned fedn repository? And paste the output here.

[kamalparsapu]

[image: image.png] i didnt do any modification

…

On Mon, May 20, 2024 at 3:07 PM Fredrik Wrede ***@***.***> wrote: I was simply a question if you have made any modification not an indication that you need to modify anything. I'm trying to figure out why you have so large images. Can you please run "ls -shl" at the root of your cloned fedn repository? And paste the output here. — Reply to this email directly, view it on GitHub <#610 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AYDBF7I74UEZIO3PAHCQW3TZDG75TAVCNFSM6AAAAABH66GKNSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCMRQGA3DMMRRG4> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[Wrede]

I can't see the image unfortunately. Can you tried to attach it again?

[kamalparsapu]

[image: image.png] On Mon, May 20, 2024 at 3:12 PM kamalraj parsapu ***@***.***> wrote:

…

[image: image.png] i didnt do any modification On Mon, May 20, 2024 at 3:07 PM Fredrik Wrede ***@***.***> wrote: > I was simply a question if you have made any modification not an > indication that you need to modify anything. I'm trying to figure out why > you have so large images. Can you please run "ls -shl" at the root of your > cloned fedn repository? And paste the output here. > > — > Reply to this email directly, view it on GitHub > <#610 (comment)>, > or unsubscribe > <https://github.com/notifications/unsubscribe-auth/AYDBF7I74UEZIO3PAHCQW3TZDG75TAVCNFSM6AAAAABH66GKNSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCMRQGA3DMMRRG4> > . > You are receiving this because you are subscribed to this thread.Message > ID: ***@***.***> >

[Wrede]

fixed in #613