Replace tokenizer with processor

[kylesayrs]

Purpose

• Prepare to support processors and vision datasets
• It's important to rename and retype variable to better reflect its more widened definition

Prerequisites

• Remove unused sparseml.export utilities #950

Postrequisites

• Vision Datasets #943

Changes

• Rename and retype instances of tokenizer to processor
• Add processor pathway argument to which tokenizer is internally reassigned to
• Add typing definitions in src/llmcompressor/typing.py
• Special handling of tokenizer in src/llmcompressor/transformers/finetune/data/base.py, src/llmcompressor/transformers/finetune/data/ultrachat_200k.py, src/llmcompressor/transformers/finetune/session_mixin.py

Testing

• No new functionality is added, CI tests should pass

[github-actions]

👋 Hi! Thank you for contributing to llm-compressor. Please add the ready label when the PR is ready for review.

[rahul-tuli]

LGTM! Thanks for this.

[dsikka]

I think we need to make it very clear:

1. What a processor is vs a tokenizer
2. If either/or can be provided and in what cases

[dsikka]

Seems like we're keeping the tokenizer in the model_args as well? What if both are specified? Or only tokenizer?

[kylesayrs]

See the newly added model args handling logic

[dsikka]

same, what if a tokenizer is provided?

[kylesayrs]

See the newly added model args handling logic

[kylesayrs]

@dsikka The current strategy is to treat all possible tokenizers as a subset of all possible processors, as type-defed here

Processor = Union[
    PreTrainedTokenizer, BaseImageProcessor, FeatureExtractionMixin, ProcessorMixin
]

We should continue to support the tokenizer model arg, but internally reassign it to the processor variable name for code simplicity.

# silently assign tokenizer to processor
if model_args.tokenizer:
    if model_args.processor:
        raise ValueError("Cannot use both a tokenizer and processor")
    model_args.processor = model_args.tokenizer
model_args.tokenizer = None

[dsikka]

@dsikka The current strategy is to treat all possible tokenizers as a subset of all possible processors, as type-defed here

Processor = Union[
    PreTrainedTokenizer, BaseImageProcessor, FeatureExtractionMixin, ProcessorMixin
]

We should continue to support the tokenizer model arg, but internally reassign it to the processor variable name for code simplicity.

# silently assign tokenizer to processor
if model_args.tokenizer:
    if model_args.processor:
        raise ValueError("Cannot use both a tokenizer and processor")
    model_args.processor = model_args.tokenizer
model_args.tokenizer = None

I think this is fine. My two comments about clarity were specific to being clear towards users - either in the model_args or through text_generation.py script

[kylesayrs]

@dsikka

1. There is help text attached to the newly added processor arg which users can read
2. We throw an error if both are passed

I think this should be clear enough messaging without being annoying/verbose

[dsikka]

@dsikka

1. There is help text attached to the newly added processor arg which users can read
2. We throw an error if both are passed

I think this should be clear enough messaging without being annoying/verbose

Oh sorry, missed the help text.
Sounds good