Image Segmentation Guide missing dependencies/imports and overwritten variables

[evanwong1020]

System Info

• transformers version: 4.38.1
• Platform: Windows-10-10.0.19045-SP0
• Python version: 3.11.4
• Huggingface_hub version: 0.20.3
• Safetensors version: 0.4.2
• Accelerate version: 0.27.2
• Accelerate config: not found
• PyTorch version (GPU?): 2.2.1+cu121 (True)
• Tensorflow version (GPU?): 2.15.0 (False)
• Flax version (CPU?/GPU?/TPU?): not installed (NA)
• Jax version: not installed
• JaxLib version: not installed
• Using GPU in script?: No
• Using distributed or parallel set-up in script?: No

Who can help?

I have not tested the TensorFlow version, only the PyTorch version.

So far, I have identified:

• Missing pip install accelerate
• Missing imports:
  • torch
  • torch.nn as nn
  • numpy as np
• train_ds[0] will not display the image, you need to access the ['image'] within the dictionary structure.
  • ds[0]["image"] will run into an error. ds was overridden and will require you to call ds['test'][0]["image"] if you're looking for the test set, and ds['train'][0]["image"] if you wanted the training set.
• train_transforms and val_transforms convert the ds images into pixel values. When trying to access image = ds[0]["image"] later on, it throws a KeyError since the images have been converted already.
• ade_palette() is not defined. It's in the TensorFlow models library in a file called get_dataset_colormap, but I was unable to import it.
• The trainer class requires an extra two parameters tokenizer and data_collator to generate the preprocesing_config.json file.

trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=train_ds,
    eval_dataset=test_ds,
    tokenizer=image_processor,
    data_collator=default_data_collator,
    compute_metrics=compute_metrics,
)

• This should be fixed with the PR linked on this issue I posted earlier

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

1. Follow along the Image Segmentation guide at https://huggingface.co/docs/transformers/tasks/semantic_segmentation
2. Upon reaching the cell that reads train_ds[0], the output will be

{'image': <PIL.JpegImagePlugin.JpegImageFile image mode=RGB size=300x240>,
 'annotation': <PIL.PngImagePlugin.PngImageFile image mode=L size=300x240>,
 'scene_category': 1001}

3. Upon reaching the cell that begins with training_args = TrainingArguments(, you will receive an import error:

`ImportError: Using the `Trainer` with `PyTorch` requires `accelerate>=0.21.0`: Please run `pip install transformers[torch]` or `pip install accelerate -U`

4. Under the inference section that reads image = ds[0]["image"], you will receive a KeyError due to ds being overwritten earlier, and the train_transforms and val_transforms functions converting the images to pixel values.
5. The very last cell attempts to call the ade_palette() function, which is not defined.

Expected behavior

Upon reaching the cell that reads train_ds[0], the output should be a viewable image. Fixable by changing to train_ds[0]['image']

Upon reaching the cell that begins with training_args = TrainingArguments(, the model should start its training process without issue. Should be resolved by the issue and PR mentioned earlier.

Calling image = ds[0]["image"] should display an image. I think it's easier to just reload the dataset, since converting the pixel values back into an image would take a lot of work.

The ade_palette() function should be imported properly or defined at some point. I'm not sure if this is an issue for everyone else, but I can't seem to import it via TensorFlow.

[ArthurZucker]

cc @stevhliu and @merveenoyan