Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Instrument metrics reporting to Redis #9

Merged
merged 14 commits into from
Mar 1, 2024
Merged

Instrument metrics reporting to Redis #9

merged 14 commits into from
Mar 1, 2024

Conversation

lotif
Copy link
Collaborator

@lotif lotif commented Feb 27, 2024

PR Type

Feature

Short Description

Clickup Ticket(s): https://app.clickup.com/t/8687c8jj1

Adding a subclass of MetricsReporter that saves the metrics to Redis instead of a file.
Tested by starting a server with it:

metrics_reporter = RedisMetricsReporter(redis_connection=redis.Redis(host="localhost", port=6379))
server = FlServer(client_manager, strategy, metrics_reporter=metrics_reporter)

After training, check if the metrics have been saved to Redis successfully:

$ python
Python 3.9.6 (default, Nov 10 2023, 13:38:27) 
[Clang 15.0.0 (clang-1500.1.0.2.5)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import redis
>>> server_conn = redis.Redis(host="localhost", port=6379)
>>> server_conn.get("da37a3fe-7c68-44a7-98bb-817952212a43")
b'{"type": "server", "fit_start": "2024-02-27 12:11:00.913371",  ...

Tests Added

Unit tests for RedisMetricsReporter.

@lotif lotif changed the base branch from main to add-redis February 27, 2024 20:04
@lotif lotif changed the base branch from add-redis to main February 29, 2024 21:02
@lotif lotif changed the base branch from main to add-redis February 29, 2024 21:03
Base automatically changed from add-redis to main February 29, 2024 22:10
@@ -1,4 +1,4 @@
name: tests
name: integration-tests
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sorry i should have done this! didnt think to split them up

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No problems! Things were still in flux with my open PR on aieng-template changes.

from flwr.common.logger import log


class RedisMetricsReporter(MetricsReporter): # type: ignore
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

curious why we have to type ignore this

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ahhh no output folder

Copy link
Collaborator Author

@lotif lotif Mar 1, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It was this weird error:
https://stackoverflow.com/questions/49888155/class-cannot-subclass-qobject-has-type-any-using-mypy

I tried solving it a couple of different ways without luck, so I ignored it. Seems to be something mypy is not picking up from fl4health's package settings and not really a big issue. Not sure if it's worth trying to find a fix in fl4health. We can also disable this check if it becomes too annoying.

"""
Init an instance of RedisMetricsReporter.

:param redis_connection: (redis.client.Redis) the connection object to a Redis. Should be the output
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I noticed you are using sphinx docstring format and I used the google format in the last PR. I don't have a strong preference and would be happy to adopt sphinx, do you think we should use the same format? Or does it not matter

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I just finished setting up automatic documentation with sphinx (following aieng-template changes) so I think we should stick to the sphinx format.

https://vectorinstitute.github.io/FLorist/reference/api/florist.api.launchers.launch.html

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ahhhhh I seee, makes sense! I will keep that in mind going forward

Copy link
Collaborator

@jewelltaylor jewelltaylor left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Only one comment to clarify if we should stay consistent on docstring format. Other than that, everything looks great and runs on my local machine

@lotif lotif merged commit 62463ed into main Mar 1, 2024
4 checks passed
@lotif lotif deleted the add-metrics-to-redis branch March 1, 2024 15:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants