Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Tentative refactoring of convert HF #156

Draft
wants to merge 7 commits into
base: main
Choose a base branch
from

Conversation

francoishernandez
Copy link
Member

@francoishernandez francoishernandez commented Dec 6, 2024

Disclaimer

This was only tested on a few models for now. We shall test on a wider panel (like #146) to make sure we did not loose some compatibility.

Rationale

The convert HF entrypoint has grown quite extensively to support various models and architectures along the way, and is becoming hard to follow.
This PR aims at laying out some form of structure to facilitate the understanding, maintenance and extendability of the tool.

Main concepts

Here are some of the main changes:

  • creation of a HuggingfaceFiles dataclass to centralize files and config handling;
  • creation of various functions to split the logic in more manageable chunks, and make the main run() method more readable;
  • main eole configs are built directly as dicts rather than assigning values with a lot of sequential conditions;

TODO

  • factorize/simplify the checkpoint/shard building logic;
  • move more stuff to the HuggingfaceFiles class if relevant;
  • check various namings to improve readability;
  • simplify some function signatures, if possible;
  • run the model-validator recipe to validate compatibility;

Feedback welcome! Any ideas to make this complex but nonetheless crucial script better are worth sharing!

@francoishernandez francoishernandez added enhancement New feature or request refactor Some refactoring, aesthetic or cleanup code changes labels Dec 6, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request refactor Some refactoring, aesthetic or cleanup code changes
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant