Issue 525: Tracking Interface with Mlflow

[njbrake]

What's changing

This PR builds on top of the architecture changes introduced in #728.

It introduces the MLflow Tracking Client with a layer of abstraction to support other tracking backends in the future. I implemented a few mlflow methods to be sure that I correctly understand how to use the mlflow library. I have code that integrates it into our service and dependency layer, but I'll put that into a different PR.

This pull request includes significant updates to integrate MLflow for tracking machine learning experiments and workflows. The changes span environment configuration, Docker setup, backend settings, and the implementation of a new tracking client interface.

This pull request introduces significant changes to integrate MLflow for experiment tracking and management. The changes include updates to environment configurations, new MLflow-related classes and methods, and modifications to existing schemas to support MLflow.

MLflow Integration:

• Environment Configuration:

  • Added MLFLOW_TRACKING_URI and related AWS environment variables to .env.template and .devcontainer/docker-compose.override.yaml.
  • Introduced a new mlflow service in the Docker Compose override file.

• Backend Settings:

  • Added TrackingBackendType enum and TRACKING_BACKEND configuration in backend/settings.py.
  • Implemented a computed property TRACKING_BACKEND_URI for determining the tracking URI based on the backend type.

• Tracking Client Implementation:

  • Created MLflowTrackingClient and MLflowClientManager classes in backend/tracking/mlflow.py to handle MLflow operations.
  • Added abstract base classes TrackingClient and TrackingClientManager in backend/tracking/tracking_interface.py.
  • Implemented TrackingClientFactory to create tracking client instances based on settings.
  • Updated __init__.py to include the new tracking client manager.

• Schema Updates:

  • Modified WorkflowCreate, WorkflowResponse, WorkflowSummaryResponse, and WorkflowDetailsResponse schemas to use str instead of UUID for IDs.

• Dependencies:

  • Added mlflow to the project dependencies in pyproject.toml.

If this PR is related to an issue or closes one, please link it here.

Closes #525

How to test it

This PR does not add the tracking client into the dependency or server layer, so there are no changes to behavior. However, to confirm that mlflow works in the docker-compose, run

make local-up

Additional notes for reviewers

I already...

• Tested the changes in a working environment to ensure they work as expected
• Added some tests for any new functionality
• Updated the documentation (both comments in code and product documentation under /docs)
• Checked if a (backend) DB migration step was required and included it if required

[javiermtorres]

A couple of comments, otherwise lgtm

[javiermtorres]

Just a couple of comments.