Adding hiera #30356

Namangarg110 · 2024-04-19T19:08:12Z

What does this PR do?

Adds Hiera model from Meta as suggested in #28993

GitHub Repo: https://github.com/facebookresearch/hiera/
arXiv: https://arxiv.org/abs/2306.00989

Model License recently changed to Apache

This PR is to merge the work done by @EduardoPach and me in #29339 and #29945 respectively.

c.c. @amyeroberts

…nto main

EduardoPach · 2024-06-25T11:43:27Z

Hey @amyeroberts I've updated the model cards for all hiera checkpoints here could you help out to transfer them to facebook org?

c.c. @amyeroberts

amyeroberts · 2024-06-26T12:45:51Z

@EduardoPach Sure! Just to confirm, should all of these checkpoints be moved?

EduardoPach · 2024-06-26T22:35:23Z

@EduardoPach Sure! Just to confirm, should all of these checkpoints be moved?

Yeap, we have one for each (HieraModel, HieraForPreTraining, HieraForImageClassification) and 6 different sizes

amyeroberts · 2024-06-27T17:06:44Z

Hi @EduardoPach so, it seems that @NielsRogge was handling this and was in contact with people from meta regarding the weights transfers. Niels is coming back from holiday I think next week, so let's wait for him to confirm!

EduardoPach · 2024-07-01T19:43:17Z

@amyeroberts, @NielsRogge doesn't have permission to move the checkpoints from my account to the facebook orq and I was talking to @dbolya and he is only a part of facebookresearch org and therefore can't add Niels or me as contributors to the org. Could someone with the appropriate permissions just transfer the checkpoints from my account to the Facebook org? (or perhaps facebookresearch idk)

Namangarg110 · 2024-07-04T16:28:51Z

@amyeroberts Any updates regarding the same ?

NielsRogge · 2024-07-04T17:47:12Z

Hi both, finally got to transfer the checkpoints :) they're now all here: https://huggingface.co/models?other=hiera&sort=created

Namangarg110 · 2024-07-04T17:52:14Z

Thankyou @NielsRogge. @amyeroberts, is there anything else required from our side ?

amyeroberts

@EduardoPach @Namangarg110 Two final updates I can see needed before merge:

Updating checkpoints for tests
Removing imports in model_name/__init__.py for objects which don't exist

src/transformers/models/hiera/__init__.py

amyeroberts · 2024-07-05T17:54:48Z

tests/models/hiera/test_modeling_hiera.py

+
+ self.assertTrue(torch.allclose(outputs.logits[0, :5], expected_slice, atol=1e-4))
+
+ def test_inference_interpolate_pos_encoding(self):


Checkpoints here need to be updated too

Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com>

EduardoPach · 2024-07-07T13:30:46Z

@amyeroberts fixed the checkpoints and cleaned the __init__.py the failing test seems unrelated to Hiera

amyeroberts · 2024-07-09T09:26:33Z

@EduardoPach Great! The last thing is to have a run of the slow tests before merge. Could you push an empty commit with: git commit --allow-empty -m "[run_slow] hiera"? This will trigger a run of the slow integration tests for the model. I'll need to approve the workflow after the commit. Once it's all green we're good to merge!

Namangarg110 · 2024-07-09T16:32:31Z

Thank you for your response, @amyeroberts. I have pushed the empty commit, and all tests are green.

amyeroberts · 2024-07-09T20:51:55Z

@Namangarg110 It seems there's some slow tests which fail and need to be addressed before merge:

Single-GPU tests

FAILED tests/models/hiera/test_modeling_hiera.py::HieraModelTest::test_attention_outputs - RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu! (when checking argument for argument index in method wrapper_CUDA_gather)
FAILED tests/models/hiera/test_modeling_hiera.py::HieraModelTest::test_batching_equivalence - RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu! (when checking argument for argument index in method wrapper_CUDA_gather)
FAILED tests/models/hiera/test_modeling_hiera.py::HieraModelTest::test_determinism - RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu! (when checking argument for argument index in method wrapper_CUDA_gather)
FAILED tests/models/hiera/test_modeling_hiera.py::HieraModelTest::test_hidden_states_output - RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu! (when checking argument for argument index in method wrapper_CUDA_gather)
FAILED tests/models/hiera/test_modeling_hiera.py::HieraModelTest::test_model_from_pretrained - OSError: We couldn't connect to 'https://huggingface.co/' to load this file, couldn't find it in the cached files and it looks like facebook/hiera-tiny-224-hf is not the path to a directory containing a file named config.json.
Checkout your internet connection or see how to run the library in offline mode at 'https://huggingface.co/docs/transformers/installation#offline-mode'.
FAILED tests/models/hiera/test_modeling_hiera.py::HieraModelTest::test_model_outputs_equivalence - RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu! (when checking argument for argument index in method wrapper_CUDA_gather)
FAILED tests/models/hiera/test_modeling_hiera.py::HieraModelTest::test_save_load - RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu! (when checking argument for argument index in method wrapper_CUDA_gather)
FAILED tests/models/hiera/test_modeling_hiera.py::HieraModelTest::test_torch_fx - RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu! (when checking argument for argument index in method wrapper_CUDA_gather)
FAILED tests/models/hiera/test_modeling_hiera.py::HieraModelTest::test_torch_fx_output_loss - RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu! (when checking argument for argument index in method wrapper_CUDA_gather)
FAILED tests/models/hiera/test_modeling_hiera.py::HieraModelTest::test_training - RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu! (when checking argument for argument index in method wrapper_CUDA_gather)
FAILED tests/models/hiera/test_modeling_hiera.py::HieraModelTest::test_training_gradient_checkpointing - RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu! (when checking argument for argument index in method wrapper_CUDA_gather)
FAILED tests/models/hiera/test_modeling_hiera.py::HieraModelTest::test_training_gradient_checkpointing_use_reentrant - RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu! (when checking argument for argument index in method wrapper_CUDA_gather)
FAILED tests/models/hiera/test_modeling_hiera.py::HieraModelTest::test_training_gradient_checkpointing_use_reentrant_false - RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu! (when checking argument for argument index in method wrapper_CUDA_gather)
FAILED tests/models/hiera/test_modeling_hiera.py::HieraModelIntegrationTest::test_inference_for_pretraining - AssertionError: False is not true
FAILED tests/models/hiera/test_modeling_hiera.py::HieraModelIntegrationTest::test_inference_fp16 - OSError: facebook/hiera-tiny-224-hf does not appear to have a file named pytorch_model.bin, model.safetensors, tf_model.h5, model.ckpt or flax_model.msgpack.

Multi-GPU tests

FAILED tests/models/hiera/test_modeling_hiera.py::HieraModelTest::test_attention_outputs - RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu! (when checking argument for argument index in method wrapper_CUDA_gather)
FAILED tests/models/hiera/test_modeling_hiera.py::HieraModelTest::test_batching_equivalence - RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu! (when checking argument for argument index in method wrapper_CUDA_gather)
FAILED tests/models/hiera/test_modeling_hiera.py::HieraModelTest::test_determinism - RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu! (when checking argument for argument index in method wrapper_CUDA_gather)
FAILED tests/models/hiera/test_modeling_hiera.py::HieraModelTest::test_hidden_states_output - RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu! (when checking argument for argument index in method wrapper_CUDA_gather)
FAILED tests/models/hiera/test_modeling_hiera.py::HieraModelTest::test_model_outputs_equivalence - RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu! (when checking argument for argument index in method wrapper_CUDA_gather)
FAILED tests/models/hiera/test_modeling_hiera.py::HieraModelTest::test_save_load - RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu! (when checking argument for argument index in method wrapper_CUDA_gather)
FAILED tests/models/hiera/test_modeling_hiera.py::HieraModelTest::test_torch_fx - RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu! (when checking argument for argument index in method wrapper_CUDA_gather)
FAILED tests/models/hiera/test_modeling_hiera.py::HieraModelTest::test_torch_fx_output_loss - RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu! (when checking argument for argument index in method wrapper_CUDA_gather)
FAILED tests/models/hiera/test_modeling_hiera.py::HieraModelTest::test_training - RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu! (when checking argument for argument index in method wrapper_CUDA_gather)
FAILED tests/models/hiera/test_modeling_hiera.py::HieraModelTest::test_training_gradient_checkpointing - RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu! (when checking argument for argument index in method wrapper_CUDA_gather)
FAILED tests/models/hiera/test_modeling_hiera.py::HieraModelTest::test_training_gradient_checkpointing_use_reentrant - RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu! (when checking argument for argument index in method wrapper_CUDA_gather)
FAILED tests/models/hiera/test_modeling_hiera.py::HieraModelTest::test_training_gradient_checkpointing_use_reentrant_false - RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu! (when checking argument for argument index in method wrapper_CUDA_gather)
FAILED tests/models/hiera/test_modeling_hiera.py::HieraModelIntegrationTest::test_inference_for_pretraining - AssertionError: False is not true
FAILED tests/models/hiera/test_modeling_hiera.py::HieraModelIntegrationTest::test_inference_fp16 - ValueError: HieraModel does not support `device_map='auto'`. To implement support, the model class needs to implement the `_no_split_modules` attribute.
FAILED tests/models/hiera/test_modeling_hiera.py::HieraModelIntegrationTest::test_inference_image_classification_head - RuntimeError: Float did not match Half
FAILED tests/models/hiera/test_modeling_hiera.py::HieraModelIntegrationTest::test_inference_interpolate_pos_encoding - RuntimeError: Input type (float) and bias type (c10::Half) should be the same
FAILED tests/models/hiera/test_modeling_hiera.py::HieraBackboneTest::test_backbone_outputs - RuntimeError: Input type (float) and bias type (c10::Half) should be the same
FAILED tests/models/hiera/test_modeling_hiera.py::HieraBackboneTest::test_backbone_stage_selection - RuntimeError: Input type (float) and bias type (c10::Half) should be the same
FAILED tests/models/hiera/test_modeling_hiera.py::HieraBackboneTest::test_create_from_modified_config - RuntimeError: Input type (float) and bias type (c10::Half) should be the same

EduardoPach · 2024-07-11T19:06:26Z

c.c. @amyeroberts

amyeroberts · 2024-07-11T21:13:51Z

Thanks for addressing all the slow tests - we're good to merge!

* initialized Structure * Updated variable names * Added Config class, basic HF setup, convert_to_hf * Fixed Convert function, added hiera to HF files, Initilized test files * better naming for x in forward pass * Moved utils to hiera * Change hiera -> hiera_model * Fixed integration into tranformers * Fix: Convert Checkpoint * added documentation for hiera * added documentation for hiera * added Docstings to models, Transformers based changes * make style and quality * make style and quality * Integration & Block tests running * Fixed bugs * initialized Structure * Updated variable names * Added Config class, basic HF setup, convert_to_hf * Fixed Convert function, added hiera to HF files, Initilized test files * better naming for x in forward pass * Moved utils to hiera * Change hiera -> hiera_model * Fixed integration into tranformers * Fix: Convert Checkpoint * added documentation for hiera * added documentation for hiera * added Docstings to models, Transformers based changes * make style and quality * make style and quality * Integration & Block tests running * Fixed bugs * Removed tim dependency * added HieraBlock * fixed: Model name * added tests for HieraModel, HieraBlock * fixed imports * fixed quality & copies * Fixes * Update docs/source/en/model_doc/hiera.md Fix name Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com> * Update docs/source/en/model_doc/hiera.md Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com> * Update docs/source/en/model_doc/hiera.md Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com> * Update src/transformers/models/hiera/configuration_hiera.py Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com> * Update src/transformers/models/hiera/configuration_hiera.py Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com> * Update src/transformers/models/hiera/modeling_hiera.py Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com> * Update src/transformers/models/hiera/modeling_hiera.py Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com> * Fixed formatting * Code quality & Import differences * quality and repo-consistency fix * fixed no torch error * Docstring fix * Docstring fix * doc string fix * fixed example usage * Resolved issues in modeling_hiera * Removed Hiera MAE * Added test and resolved bug * fixed doc string * First commit * Finished conversion script and model forward working * Resolved all issues * nits * Improving tests * Nits * More nits * Improving HieraForMaskedImageModeling * More improvements and nits * Fixed docstrings of outputs * More fixes * More imrpovments * Updated conversion script * Fixed docstrings * Improved tests * Fixed attentou outputs test * All tests green * Removed unnecessary file * contribution attribution * Resolved a few issues * Resolved Comments * Updated model repo id and fixed bugs * Removed loss print * Make tests green * Updated docstrings * Fix style * Fixed num_heads in config * Removed unnecessary video checkpoint related code in the conversion script * Fix style * Changed atol in conversion script * HieraConfig * Fix copies * Fixed typo * Resolved few issues * make * converted conv_nd -> nn.Module * Removed video complexities * Removed video complexities * fix style * Addressing comments * Update src/transformers/models/hiera/modeling_hiera.py Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com> * Update src/transformers/models/hiera/modeling_hiera.py Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com> * Update src/transformers/models/hiera/modeling_hiera.py Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com> * Fix style * Fixed tests * Fixed typo * Fixed interpolate test * Made torch fx compatible * Made sure imageprocesor is correct * Addressed comments * Noise directly as torch * Remove unnecesary attr * Added return_dit * Update src/transformers/models/hiera/__init__.py Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com> * Updated checkpoints * [run_slow] hiera * Fixed device mismatch * [run_slow] hiera * Fixed GPU tests * [run_slow] hiera --------- Co-authored-by: Ubuntu <ubuntu@ip-172-31-29-50.us-east-2.compute.internal> Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com> Co-authored-by: Eduardo Pacheco <eduardo.pach@hotmail.com> Co-authored-by: Eduardo Pacheco <69953243+EduardoPach@users.noreply.github.com> Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com>

Ubuntu and others added 30 commits February 16, 2024 00:06

initialized Structure

0cd30ee

Merge branch 'huggingface:main' into main

ba8a870

Updated variable names

ec4111f

Merge branch 'main' of https://github.com/namangarg110/transformers i…

8d27d82

…nto main

Merge branch 'huggingface:main' into main

77d2591

Added Config class, basic HF setup, convert_to_hf

126de18

Merge branch 'huggingface:main' into main

479bfdc

Fixed Convert function, added hiera to HF files, Initilized test files

75a3440

Merge branch 'huggingface:main' into main

2e2e638

better naming for x in forward pass

5569dad

Moved utils to hiera

11017c6

Change hiera -> hiera_model

ad959d4

Fixed integration into tranformers

fac7b23

Fix: Convert Checkpoint

866ffc7

Merge branch 'huggingface:main' into main

deba4fc

added documentation for hiera

b3828e1

added documentation for hiera

82672b2

added Docstings to models, Transformers based changes

00478b6

make style and quality

4144fe8

make style and quality

d23a70d

Integration & Block tests running

c677783

Merge branch 'huggingface:main' into main

9cee5e2

Merge branch 'huggingface:main' into main

419ccf8

Fixed bugs

130b55b

initialized Structure

733c59e

Updated variable names

86a43ed

Added Config class, basic HF setup, convert_to_hf

1433a7c

Fixed Convert function, added hiera to HF files, Initilized test files

46d495c

better naming for x in forward pass

a25a3a7

Moved utils to hiera

51d11f5

amyeroberts reviewed Jul 5, 2024

View reviewed changes

EduardoPach and others added 3 commits July 7, 2024 09:09

Update src/transformers/models/hiera/__init__.py

ef2ea02

Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com>

Updated checkpoints

190c662

Merge remote-tracking branch 'upstream/main' into adding-hiera-cont

ec0ad99

amyeroberts added the run-slow label Jul 9, 2024

[run_slow] hiera

d76183e

Namangarg110 and others added 5 commits July 10, 2024 03:42

Fixed device mismatch

c67bacf

[run_slow] hiera

ec2c480

Merge remote-tracking branch 'upstream/main' into adding-hiera-cont

7b840d8

Fixed GPU tests

834bc8e

[run_slow] hiera

0a06b4d

amyeroberts merged commit c1e139c into huggingface:main Jul 11, 2024
26 checks passed

qubvel mentioned this pull request Oct 2, 2024

Add Hiera model #28993

Closed

2 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Adding hiera #30356

Adding hiera #30356

Namangarg110 commented Apr 19, 2024 •

edited

Loading

EduardoPach commented Jun 25, 2024

amyeroberts commented Jun 26, 2024

EduardoPach commented Jun 26, 2024

amyeroberts commented Jun 27, 2024

EduardoPach commented Jul 1, 2024

Namangarg110 commented Jul 4, 2024

NielsRogge commented Jul 4, 2024

Namangarg110 commented Jul 4, 2024

amyeroberts left a comment

amyeroberts Jul 5, 2024

EduardoPach Jul 7, 2024

EduardoPach commented Jul 7, 2024

amyeroberts commented Jul 9, 2024

Namangarg110 commented Jul 9, 2024

amyeroberts commented Jul 9, 2024

EduardoPach commented Jul 11, 2024

amyeroberts commented Jul 11, 2024


		self.assertTrue(torch.allclose(outputs.logits[0, :5], expected_slice, atol=1e-4))

		def test_inference_interpolate_pos_encoding(self):

Adding hiera #30356

Adding hiera #30356

Conversation

Namangarg110 commented Apr 19, 2024 • edited Loading

What does this PR do?

EduardoPach commented Jun 25, 2024

amyeroberts commented Jun 26, 2024

EduardoPach commented Jun 26, 2024

amyeroberts commented Jun 27, 2024

EduardoPach commented Jul 1, 2024

Namangarg110 commented Jul 4, 2024

NielsRogge commented Jul 4, 2024

Namangarg110 commented Jul 4, 2024

amyeroberts left a comment

Choose a reason for hiding this comment

amyeroberts Jul 5, 2024

Choose a reason for hiding this comment

EduardoPach Jul 7, 2024

Choose a reason for hiding this comment

EduardoPach commented Jul 7, 2024

amyeroberts commented Jul 9, 2024

Namangarg110 commented Jul 9, 2024

amyeroberts commented Jul 9, 2024

EduardoPach commented Jul 11, 2024

amyeroberts commented Jul 11, 2024

Namangarg110 commented Apr 19, 2024 •

edited

Loading