Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

External package integration using plugins #126

Merged
merged 3 commits into from
Jun 27, 2024
Merged

External package integration using plugins #126

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
b80a91f
feat: Allow loading model configurations from other packages
lorenzomammana Jun 5, 2024
03947e1
feat: Allow including external tasks from plugins
lorenzomammana Jun 5, 2024
ed2e7f7
Merge branch 'main' into feature/external-package-integration
lorenzomammana Jun 27, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions lmms_eval/__main__.py
View file
Open in desktop
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think it is necessary to have this? Since you can simply use lmms-eval --include-path to include other tasks in your command line

Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
import importlib
import os
import yaml
import sys
Expand Down Expand Up @@ -236,6 +237,12 @@ def cli_evaluate_single(args: Union[argparse.Namespace, None] = None) -> None:
eval_logger.info(f"Including path: {args.include_path}")
include_path(args.include_path)

if os.environ.get("LMMS_EVAL_PLUGINS", None):
for plugin in os.environ["LMMS_EVAL_PLUGINS"].split(","):
package_tasks_location = importlib.util.find_spec(f"{plugin}.tasks").submodule_search_locations[0]
eval_logger.info(f"Including path: {args.include_path}")
include_path(package_tasks_location)

if args.tasks is None:
task_names = ALL_TASKS
elif args.tasks == "list":
Expand Down
17 changes: 17 additions & 0 deletions lmms_eval/models/__init__.py
View file
Open in desktop
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think you should create a function here say include_model just like the include_task and use the file path to include more models instead of the plugins.

I think would be more flexible if user can use something like lmms-eval --include-model <file_path> to include models instead of setting an environment variable

Original file line number Diff line number Diff line change
@@ -1,3 +1,6 @@
import importlib
import os
import hf_transfer
from loguru import logger
import sys

Expand Down Expand Up @@ -38,3 +41,17 @@
except ImportError as e:
# logger.warning(f"Failed to import {model_class} from {model_name}: {e}")
pass

if os.environ.get("LMMS_EVAL_PLUGINS", None):
# Allow specifying other packages to import models from
for plugin in os.environ["LMMS_EVAL_PLUGINS"].split(","):
m = importlib.import_module(f"{plugin}.models")
for model_name, model_class in getattr(m, "AVAILABLE_MODELS").items():
try:
exec(f"from {plugin}.models.{model_name} import {model_class}")
except ImportError:
pass

import hf_transfer

os.environ["HF_HUB_ENABLE_HF_TRANSFER"] = "1"
Loading