Lit-GPT model integration

[aniketmaurya]

Hello HELM team, would you be interested in accepting a PR to integrate Lightning AI's Lit-GPT? If yes, I would love to contribute here.

[yifanmai]

Hi @aniketmaurya, this is something that we'd definitely like to have. HELM is also supporting the NeurIPS efficiency challenge, so it would be nice if users could run HELM on the Lit-GPT checkpoints directly.

Here's some related documentation:

• development setup
• adding new models

Would you like to take a shot at an initial PR? Otherwise I can take a look sometime next week and draft a PR.

Also, if this would be faster with sync communication, feel free to email me and set up a time for a video call. I can be reached at my email address in the git commit log.

One blocker is the Lit-GPT has a lot of configuration parameters; I'm currently working on configuration files for models over the next couple of weeks, which would solve this, but in the meantime, we might have to read configuration from environment variables or something.

[aniketmaurya]

hi @yifanmai, thank you for your reply! I have a initial PR here.

Yeah, I realized that for configuration I need to depend on environment variable. Would be supercool to have a configuration file. I will drop you an email to setup a time for call.

[yifanmai]

I think we should migrate to use the new LitGPT API. We could also consider using supporting LitServe.