Add download_transformers_model

[ckunki]

Function download_transformers_model() is currently implemented in a different repo.

Additionally, we found more items to move to the TE repo

• function download_transformers_model() is moved and renamed to install_huggingface_model()
• function load_transformers_pipline() is moved and renamed to load_huggingface_pipeline()

As soon as added here,

1. The implementation can be removed from the other repo
2. in notebook-connector's file pyproject.toml the extras dependencies can be updated to use a separate category for exasol-text-ai-extension, see Use install_huggingface_model from TE notebook-connector#255

[ckunki]

File upload_model.py already contains function

def upload_model_to_bfs_location(
    model_name: str,
    task_type: str,
    subdir: Path,
    bucketfs_location: PathLike,
    huggingface_token: str | None = None,
) -> Path:

While utils/model_utils.py contains

def install_huggingface_model(
    bucketfs_location: bfs.path.PathLike,
    sub_dir: str,
    task_type: str,
    model_name: str,
    model_factory: ModelFactoryProtocol,
    tokenizer_factory=huggingface.AutoTokenizer,
    huggingface_token: str | None = None,
) -> bfs.path.PathLike:

Differences

model_utils.py	upload_model.py
uses context	uses plain method calls, relying on __del__() to clean up temp directory
Parameter sub_dir, type str	Parameter sub_dir, type Path
Parameter model_factory	derived via BucketFSModelSpecification().get_bucketfs_model_save_path()
Parameter tokenizer_factory (optional)	constant transformers.AutoTokenizer
Return bucketfs_location / upload_path, type bfs.path.PathLike	Returns upload_path, type str

Proposal

• Use approach from model_utils.py:
  • Parameter sub_dir of type str, wrap in Path() inside the function.
  • Optional parameter tokenizer_factory with default transformers.AutoTokenizer.
  • Return bsf.path.PathLike as the only application in TE seems to be a message about the upload to BFS, which by default would print the same result when using the bsf.path.PathLike instead of str sub_dir
• Make parameter model_factory optional, with default from BucketFSModelSpecification
  • BTW: All usages of context download_model_to_bucketfs() in TXAI are using the same model factory as would be derived from BucketFSModelSpecification().
  • Hence the parameter is obsolete and could be removed.