Support subfolder with AutoProcessor

[JingyaHuang]

What does this PR do?

Support subfolder option in AutoProcessor. Currently we are not able to load processors in subfolder, eg.

from transformers import AutoProcessor

AutoProcessor.from_pretrained("sentence-transformers/clip-ViT-B-32", subfolder="0_CLIPModel", trust_remote_code=False)

We will get following error:

*** OSError: sentence-transformers/clip-ViT-B-32 does not appear to have a file named preprocessor_config.json. Checkout 'https://huggingface.co/sentence-transformers/clip-ViT-B-32/main' for available files.

However I am able to load processor in subfolder with directly:

import transformers

transformers.CLIPProcessor.from_pretrained.from_pretrained("sentence-transformers/clip-ViT-B-32", subfolder="0_CLIPModel", trust_remote_code=False)

cc. @fxmarty FYI our maybe_load_preprocessors is not able to store sentence transformers clip processor without this.

Before submitting

• This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
• Did you read the contributor guideline,
  Pull Request section?
• Was this discussed/approved via a Github issue or the forum? Please add a link
  to it if that's the case.
• Did you make sure to update the documentation with your changes? Here are the
  documentation guidelines, and
  here are tips on formatting docstrings.
• Did you write any new necessary tests?

Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[amyeroberts]

Hi @JingyaHuang, thanks for opening this PR!

My understanding is that using subfolder was deliberately not exposed for use in the AutoProcessor API. There's an internal discussion here: https://huggingface.slack.com/archives/C014N4749J9/p1707474720126079

Which, tbh, is a bit surprising as it works with other Auto APIs like AutoModel, so I'd be pro adding it here. @ydshieh do you know why it was added originally?

[JingyaHuang]

Thanks @amyeroberts ! Yeah it's a bit suprise, since in Optimum we store config for compiled models, and it works fine with subfolder for AutoTokenizer and AutoFeatureExtractor, until we got some issues with some CLIP checkpoints under subfolders.

[ArthurZucker]

I don't mind having this if this removes some confusing / is expected from our usual api!

[JingyaHuang]

From optimum perspective, it would be really nice if we could enable subfolder with AutoProcessor, otherwise we might need to override it... @ArthurZucker @amyeroberts

[ydshieh]

Will take a look

[ydshieh]

Hi. First of all,

subfolder was deliberately not exposed for use in the AutoProcessor API

this is not the case, and the internal discussion is about that model type is not exposed publicly, not about subfolder not being used in AutoProcessor.

Looking at the two parts inside

config_dict, _ = ImageProcessingMixin.get_image_processor_dict(pretrained_model_name_or_path, **kwargs)

and

config_dict, _ = FeatureExtractionMixin.get_feature_extractor_dict(...)

in AutoProcessor.from_pretrained, the first line is called without failure (and we get config_dict), but the second one fails.

Looking at the definitions of these 2 methods, you will see subfolder is used in the body of get_image_processor_dict but not in get_feature_extractor_dict (or even in the file src/transformers/feature_extraction_utils.py).

So, to me, it looks like FeatureExtractor is created (much) earlier than ImageProcessor and not being updated properly. @JingyaHuang 's work in this PR is the way to go.

[ydshieh]

However, there is one thing I still think strange. Let me take a deeper look before merging this PR 🙏

[ydshieh]

even with the PR, the loading is done via the following block rather then

        # Last try: we use the PROCESSOR_MAPPING.
        elif type(config) in PROCESSOR_MAPPING:
            return PROCESSOR_MAPPING[type(config)].from_pretrained(pretrained_model_name_or_path, **kwargs)

rather than using the processor_class obtained in

                config_dict, _ = FeatureExtractionMixin.get_feature_extractor_dict(
                    pretrained_model_name_or_path, **kwargs
                )
                processor_class = config_dict.get("processor_class", None)

(as the preprocessor_config.json file in the subfolder has no processor_class attribute)

No big deal. Good for me this PR.

[amyeroberts]

Thanks for adding!