🐕 Batch: Pre-submission lintern

[miquelduranfrigola]

Summary

Often, small errors like a short title in the metadata will cause the model incorporation workflows to fail. It would be good to identify these errors earlier using a lintern.

Ideally, we would have a command like:

ersilia lint --repo_path $PATH_TO_WIP_MODEL

Objectives

We can potentially check several things in a model repository, including:

• The metadata.json file
• The Dockerfile
• The model/framework and model/checkpoints folders
• The LICENSE

I would start with the metadata.json file. Most of the heavy lifting has already been done here: https://github.com/ersilia-os/ersilia/blob/master/ersilia/hub/content/card.py

A potential approach would be:

1. Create a module called lintern.py under hub/content/
2. Write a class called Lint with an arbitrary number of methods, including metadata.
3. Write a CLI command that primarily uses the Lint class to check that everything is correct in the --repo_path folder.

Do the test as part of an earlier workflow. These are not mutually exclusive

Documentation

[GemmaTuron]

Is this lint check better suited for ersilia-maintenance? has any work in this direction been done or could this be an entry-level task @miquelduranfrigola and @DhanshreeA ?

[miquelduranfrigola]

Given that we have plans to simplify the CLI commands, I would definitely not include a lint command. However, lint functionalities are included (and could be further expanded) in both the test command as well as the ModelInspector class (used in ersilia-maintenance).

We can certainly think of more "tests" or "inspections" to do and come up with a good entry-level task. This is a good idea.

[DhanshreeA]

Plus to add to @miquelduranfrigola 's point, ersilia pack also has a lint built in for newer model contributions.

[GemmaTuron]

Hi @DhanshreeA
Please if this is a good entry level task, open the appropriate issues in ersilia-maintenance repository or wherever appropriate.

[GemmaTuron]

Hi @DhanshreeA

I am unsure whether this issue is tackled in independent issues or should remain open. Can you update? Thx

[DhanshreeA]

@GemmaTuron I think we already have a lot done on this front, namely, with the following tools/ersilia commands:

1. ersilia inspect: More for maintenance purposes since this works "a posteriori", ie it checks models routinely that already exist in the hub.
2. ersilia test: We want to enforce its usage during model contribution.
3. Linting in ersilia-pack through ersilia_model_lint: This was designed with the intention of being used within the model contribution and submission workflow. However this only works with the new template.

We will have to consolidate functionality across the three of these, however it does not make sense to add another linter into Ersilia. So this issue is redundant at this point.

[GemmaTuron]

Hi @miquelduranfrigola and @DhanshreeA

Should we close this model then?

[miquelduranfrigola]

I'll let @DhanshreeA decide

[DhanshreeA]

Yes, I think it's good to close this issue. We need to only focus on ersilia inspect, test, and the lint command within ersilia-pack (which can be added as a pipeline step on its own)

[miquelduranfrigola]

This makes sense, @DhanshreeA - thanks