Speed up the searching/validating processes of models on the serving container

[RobertLucian]

Description

When an API with many models is deployed (when using the predictor.models.dir field), the searching and validating processes of the models is very slow - so slow, that for a large number of models (i.e. 1000), it takes > 10 minutes to run it once.

The 2 culprit functions for that are validate_models_dir_paths and find_all_cloud_models, with the former taking the most amount of time.

A good example of this is the translator example #1592.

Motivation

Apart from taking a lot of time to update the models' tree once, it's also the CPU that's wasted in the process. This needs to be faster.

Reason

Most likely a complexity problem.