Evoml 1.12.0 main internal engineering goals

[jose-turintech]

1. Black-widow memory leaks
2. Evoml model cancellation
3. Exasol integration (pipeline-deployer v1.0.0 on PRD).
4. Modularisation of the pipeline and new pipeline formats.
5. Python 3.10 support on all microservices.
6. POC internal pypi repository to improve Ray management of offline behaviour.

[jose-turintech]

Black widow memory leaks

It seems we're facing the same kind of problems arlready addressed on thor-file-api. This has to do with the nature of some features that are manipulating DataFrame objects to provide some results and the nature of python memory management itself.

Turns out once python allocates memory for the python interpreter in use (python process) even when the usage of the objects referencing that memory is finished, memory is still retained on the python process and not given back to the Operative System. This causes a side-effect when black-widow container, due to its nature, keeps N python processes alive all the time as the configured number of uvicorn workers/threads. Each thread will process requests, that eventually will deal with DataFrame management, some of them might operate over big datasets (5GB as the tests performed that made this issue evident), and after the feature is execute such memory will be retained for usage by the python process later on. This behaviour stacks up until black-widow becomes basically a memory exhauster.

Potential solution
We could apply the same execution pattern than the applied one for thor-file-api, where we used a self-spawned Ray cluster in order to take care of this type of memory-intensive workloads.
This patterns solves the issue because remote execution of code in Ray, is basically managed by running the code on a new python interpreter (performing a conda run -n env_name python_code) so once the remote code execution finishes, memory is returned to the OS.

[mjimlittle]

Just a heads up about the self-spawned Ray worker.

In Thor API, we had to reduce the number of Fast API workers from four to one when we introduced this change.
The reason was, that we were limited by the way we could spin up/down Ray by using the lifecycle of Fast API worker.
Those lifecycle hooks are triggered for every worker and that led to effectively spin up 4 different Ray instances.

Reducing the number of workers did not seem to affect the performance of Thor API on the number of requests it could handle. (At least in our usage patterns).