fail to run python MELD/teacher.py

[G-walk]

I ran python MELD/teacher.py and the following message occured:

python MELD/teacher.py
Could not find image processor class in the image processor config or the model config. Loading based on pattern matching with the model's feature extractor configuration.
###Save Path###  ./MELD/save_model
Some weights of the model checkpoint at roberta-large were not used when initializing RobertaModel: ['lm_head.layer_norm.weight', 'lm_head.decoder.weight', 'lm_head.layer_norm.bias', 'lm_head.dense.bias', 'lm_head.bias', 'lm_head.dense.weight']
- This IS expected if you are initializing RobertaModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing RobertaModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
  0%|                                                                                                                                        | 0/10 [02:12<?, ?it/s]
Traceback (most recent call last):
  File "/opt/data/private/TelME-main/MELD/teacher.py", line 167, in <module>
    main(args)
  File "/opt/data/private/TelME-main/MELD/teacher.py", line 160, in main
    model_train(training_epochs, model, train_loader, dev_loader, test_loader, optimizer, scheduler, max_grad_norm, save_path)
  File "/opt/data/private/TelME-main/MELD/teacher.py", line 56, in model_train
    for i_batch, data in enumerate(train_dataloader):
  File "/opt/data/private/miniforge3/envs/telme/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 628, in __next__
    data = self._next_data()
  File "/opt/data/private/miniforge3/envs/telme/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 1313, in _next_data
    return self._process_data(data)
  File "/opt/data/private/miniforge3/envs/telme/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 1359, in _process_data
    data.reraise()
  File "/opt/data/private/miniforge3/envs/telme/lib/python3.9/site-packages/torch/_utils.py", line 543, in reraise
    raise exception
audioread.exceptions.NoBackendError: Caught NoBackendError in DataLoader worker process 5.
Original Traceback (most recent call last):
  File "/opt/data/private/miniforge3/envs/telme/lib/python3.9/site-packages/librosa/core/audio.py", line 176, in load
    y, sr_native = __soundfile_load(path, offset, duration, dtype)
  File "/opt/data/private/miniforge3/envs/telme/lib/python3.9/site-packages/librosa/core/audio.py", line 209, in __soundfile_load
    context = sf.SoundFile(path)
  File "/opt/data/private/miniforge3/envs/telme/lib/python3.9/site-packages/soundfile.py", line 658, in __init__
    self._file = self._open(file, mode_int, closefd)
  File "/opt/data/private/miniforge3/envs/telme/lib/python3.9/site-packages/soundfile.py", line 1216, in _open
    raise LibsndfileError(err, prefix="Error opening {0!r}: ".format(self.name))
soundfile.LibsndfileError: Error opening './dataset/MELD.Raw/train_splits/dia125_utt3.mp4': Format not recognised.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/data/private/miniforge3/envs/telme/lib/python3.9/site-packages/torch/utils/data/_utils/worker.py", line 302, in _worker_loop
    data = fetcher.fetch(index)
  File "/opt/data/private/miniforge3/envs/telme/lib/python3.9/site-packages/torch/utils/data/_utils/fetch.py", line 61, in fetch
    return self.collate_fn(data)
  File "/opt/data/private/TelME-main/MELD/utils.py", line 114, in make_batchs
    audio, rate = librosa.load(video_path)
  File "/opt/data/private/miniforge3/envs/telme/lib/python3.9/site-packages/librosa/core/audio.py", line 184, in load
    y, sr_native = __audioread_load(path, offset, duration, dtype)
  File "/opt/data/private/miniforge3/envs/telme/lib/python3.9/site-packages/decorator.py", line 232, in fun
    return caller(func, *(extras + args), **kw)
  File "/opt/data/private/miniforge3/envs/telme/lib/python3.9/site-packages/librosa/util/decorators.py", line 60, in __wrapper
    return func(*args, **kwargs)
  File "/opt/data/private/miniforge3/envs/telme/lib/python3.9/site-packages/librosa/core/audio.py", line 241, in __audioread_load
    reader = audioread.audio_open(path)
  File "/opt/data/private/miniforge3/envs/telme/lib/python3.9/site-packages/audioread/__init__.py", line 132, in audio_open
    raise NoBackendError()
audioread.exceptions.NoBackendError

I have ffmpeg installed and it is OK if I run python MELD/inference.py with the checkpoint you released in this project. I don't know why this happened.

[Dengdan-1999]

oh my god! I have encounted the same problem with you! did you solve this problem? would you like to give me a hand? thank you very much!

[Dengdan-1999]

I am not very familiar with librosa . I have search the problem on the net, it seems like that the librosa cannot be used to load file with the format of mp4, it is used to load audio file? I am not sure whether it is true or not. so, why this problem happen, does anyone can help me? thank u very much~

[yuntaeyang]

#2 (comment)

[yuntaeyang]

It could be a problem with the librosa version. There was a researcher who had the same issue, but the only assumption I could make was the difference in the library version.

[Dengdan-1999]

It could be a problem with the librosa version. There was a researcher who had the same issue, but the only assumption I could make was the difference in the library version.

dear author, thank you so much. but after change the version of librosa(like change to librosa==0.7.2, 0.8.1, 0.9.2, 0.10.0, 0.10.1, 0.10.2, 0.10.2.post1 ) I still have the same problem! can you post you version of your packages in the requirements.txt . I feel so disturbed of this error! thank you soooooo much! and my versions of packages are as follows:
av==10.0.0
librosa==0.10.0
numpy==1.23.5
opencv-python==4.7.0.72
pandas==1.5.3
scikit-learn==1.2.2
scipy==1.10.1
torch==1.13.1
torchaudio==0.13.1
torchvision==0.14.1
transformers==4.27.2

they are all int the env of python=3.9.0

thank you again!

[G-walk]

oh my god! I have encounted the same problem with you! did you solve this problem? would you like to give me a hand? thank you very much!

Sorry, I haven't solved the problem yet. But I tried to train the model with the IEMOCAP dataset and related codes, this time it worked well. After all the IEMOCAP dataset's format is different from MELD's.

[G-walk]

It could be a problem with the librosa version. There was a researcher who had the same issue, but the only assumption I could make was the difference in the library version.

Thank you for your reply.
I tried the code in #2 (comment), it worked well:

But the same issue as #6 (comment) occurred if I ran python MELD/teacher.py. Maybe the librosa version isn't the problem?

[yuntaeyang]

Hi, I think we need to make sure that dia125_utt3.mp4 in the Meld dataset has no problem.

I share the issue of Meld dataset regarding this issue.
declare-lab/MELD#39

[G-walk]

Hi, I think we need to make sure that dia125_utt3.mp4 in the Meld dataset has no problem.

I share the issue of Meld dataset regarding this issue. declare-lab/MELD#39

I replaced dia125_utt3.mp4 with the video in declare-lab/MELD#39 (comment), it seemed the problem is solved. Thank you very much!

But the video used as a replacement is clipped with subtitles and some irrelevant UIs. I will see if I can make a cleaner one later.

But it is enough to use it to test the code. Thanks again!

[G-walk]

Hi, I think we need to make sure that dia125_utt3.mp4 in the Meld dataset has no problem.
I share the issue of Meld dataset regarding this issue. declare-lab/MELD#39

I replaced dia125_utt3.mp4 with the video in declare-lab/MELD#39 (comment), it seemed the problem is solved. Thank you very much!

But the video used as a replacement is clipped with subtitles and some irrelevant UIs. I will see if I can make a cleaner one later.

But it is enough to use it to test the code. Thanks again!

I made a cleaner version of dia125_utt3.mp4 and put it in this issue:
declare-lab/MELD#39 (comment)

However, I also noticed there were other missing files in the dev set and test set, this issue also mentioned the problem: declare-lab/MELD#22 (comment)

So I think it is better to make a missing file check in utils.py:

if not os.path.exists(video_path):  
    print(f"Missing video file: {video_path}")
    continue