[hf-diffusers] Use the latest transformers for diffusers

[yeounoh]

Description

Use the latest transformers for diffusers, after huggingface/transformers#27799

Tests

Local checkpointing test with the latest transformers

mid_block_only_cross_attention', 'attention_type'} was not found in config. Values will be initialized to default values.
Configuration saved in /tmp/sd-pokemon-model/checkpoint-1/unet_ema/config.json
Model weights saved in /tmp/sd-pokemon-model/checkpoint-1/unet_ema/diffusion_pytorch_model.safetensors
Configuration saved in /tmp/sd-pokemon-model/checkpoint-1/unet/config.json
Model weights saved in /tmp/sd-pokemon-model/checkpoint-1/unet/diffusion_pytorch_model.safetensors
12/11/2023 19:41:19 - INFO - accelerate.checkpointing - Optimizer state saved in /tmp/sd-pokemon-model/checkpoint-1/optimizer.bin
12/11/2023 19:41:19 - INFO - accelerate.checkpointing - Scheduler state saved in /tmp/sd-pokemon-model/checkpoint-1/scheduler.bin
12/11/2023 19:41:19 - INFO - accelerate.checkpointing - Sampler state for dataloader 0 saved in /tmp/sd-pokemon-model/checkpoint-1/sampler.bin
12/11/2023 19:41:19 - INFO - accelerate.checkpointing - Random states saved in /tmp/sd-pokemon-model/checkpoint-1/random_states_0.pkl
12/11/2023 19:41:19 - INFO - __main__ - Saved state to /tmp/sd-pokemon-model/checkpoint-1
Steps: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [01:24<00:00, 67.00s/it, lr=1e-5, step_loss=0.00456]{'requires_safety_checker', 'image_encoder'} was not found in config. Values will be initialized to default values.
                                                                                                                                                                                       `text_config_dict` is provided which will be used to initialize `CLIPTextConfig`. The value `text_config["id2label"]` will be overriden.                           | 0/7 [00:00<?, ?it/s]
`text_config_dict` is provided which will be used to initialize `CLIPTextConfig`. The value `text_config["bos_token_id"]` will be overriden.
`text_config_dict` is provided which will be used to initialize `CLIPTextConfig`. The value `text_config["eos_token_id"]` will be overriden.
Loaded safety_checker as StableDiffusionSafetyChecker from `safety_checker` subfolder of CompVis/stable-diffusion-v1-4.
                                                                                                                                                                                       Loaded feature_extractor as CLIPImageProcessor from `feature_extractor` subfolder of CompVis/stable-diffusion-v1-4.                                        | 1/7 [00:00<00:01,  3.48it/s]
{'timestep_spacing', 'prediction_type'} was not found in config. Values will be initialized to default values.
Loaded scheduler as PNDMScheduler from `scheduler` subfolder of CompVis/stable-diffusion-v1-4.
Loaded tokenizer as CLIPTokenizer from `tokenizer` subfolder of CompVis/stable-diffusion-v1-4.
Loading pipeline components...: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 7/7 [00:00<00:00, 19.33it/s]
Configuration saved in /tmp/sd-pokemon-model/vae/config.json
Model weights saved in /tmp/sd-pokemon-model/vae/diffusion_pytorch_model.safetensors
Configuration saved in /tmp/sd-pokemon-model/unet/config.json
Model weights saved in /tmp/sd-pokemon-model/unet/diffusion_pytorch_model.safetensors
Configuration saved in /tmp/sd-pokemon-model/scheduler/scheduler_config.json
Configuration saved in /tmp/sd-pokemon-model/model_index.json
Steps: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [01:41<00:00, 101.51s/it, lr=1e-5, step_loss=0.00456]