[WIP] Make adding new Policy Models flexible

[engmubarak48]

Fixes #293

This PR tries to make adding new function approximations (policy models) as flexible as possible.

• Modification of the base.py file to dynamically import and instantiate policy models based on configuration
• Allowing the addition of new model types (e.g., CNN, GNN, Transformer) without altering the core logic.
• Implementation of a clean interface in the ModelBase class

[josephdviviano]

lgtm - a comment, but non blocking.

[alexhernandezgarcia]

Great that you're taking a stab at this, which is one of the important things that remain to be done! Looks good so far!

[engmubarak48]

@alexhernandezgarcia This is a temporary solution to make the CNN policy work on Tetris env. But normally the flattening should happen inside the model but not in the environment (see my other comments)

if you are okay with that, then I can update.

[engmubarak48]

If we move the flattening from the environment to the policy, then we don't need this.

[engmubarak48]

I think any policy could be added now.. i have tested tetris on CNN and everything seems to work well with minimal changes.. I think next thing we could do is have another PR which better documents this and show how one could simply change the Policy etc.

test with python main.py env=tetris proxy=tetris policy=cnn env.flatten=False env.width=4 env.height=4
@alexhernandezgarcia feel free to review..

[AlexandraVolokhova]

Thank you for the great work! I've added a bunch of suggestions. It seems to me that it would be worth to add a config file for a case when one wants to create a uniform or a random policy, but I don't have a strong opinion about it.

[engmubarak48]

Hi @AlexandraVolokhova just got time, let me address your comments...

Regarding:

It seems to me that it would be worth to add a config file for a case when one wants to create a uniform or a random policy, but I don't have a strong opinion about it.

I think I agree with you, maybe it would have been nice to have a separate config for fixed and uniform policies (I am hoping you are referring to the two functions inside the base policy class). To be honest I don't even still know why we need them.. We could also keep them as it is with the default configuration.