feat: pipeline registry for supporting custom pipelines

[aarnphm]

Feature request

I propose a simple registry abstraction to allow users to dynamically register custom pipelines to transformers.

audio_classification_tmpl = {
        "impl": AudioClassificationPipeline,
        "tf": (),
        "pt": (AutoModelForAudioClassification,) if is_torch_available() else (),
        "default": {"model": {"pt": "superb/wav2vec2-base-superb-ks"}},
        "type": "audio",
    }

PipelineRegistry.register_pipeline("audio-classification", audio_classification_tmpl)

A pseudo example for PipelineRegistry implementation:

class PipelineRegistry:
    SUPPORTED_TASKS: dict[str, dict[str, PiplineBase | dict[str, Any]]

    @classmethod
    def registry_pipeline(cls, task: str, task_metadata: dict[str, Any]:
        cls.SUPPORTED_TASKS[task] = task_metadata

For any custom pipeline user can simply do

from transformers.pipelines import PipelineRegistry

my_custom_task_tmpl = {
        "impl": CustomPipeline,
        "tf": (),
        "pt": (AutoModelForAudioClassification,) if is_torch_available() else (),
        "default": {"model": {"pt": "my_custom_wav2vec"}},
        "type": "custom",
    }

PipelineRegistry.register_pipeline("custom-task", my_custom_task_tmpl)

Motivation

Currently, pipelines abstraction provides users with quick and easy way to run any given tasks. However, it is very difficult to create and adding support for custom pipelines. According to docs, If users want to add a new pipeline, one would have to come in and modify transformers source code.

This is often less than ideal. It would be nice for pipeline to have a "registry" abstraction where transformers can allow users to register their custom pipeline to transformers without the hassle of editing the source code.

Your contribution

#17905