Test newly uploaded Flan-T5 weights #2074
Conversation
|
This should be cherry-picked into the release as it's just more test coverage. |
joecummings
requested review from
Nayef211 and
rshraga
and removed request for
Nayef211
| @@ -122,7 +122,7 @@ The library currently consist of following pre-trained models: | |||
| * `DistilRoBERTa <https://github.com/huggingface/transformers/blob/main/examples/research_projects/distillation/README.md>`_ | |||
| * XLM-RoBERTa: `Base and Large Architure <https://github.com/pytorch/fairseq/tree/main/examples/xlmr#pre-trained-models>`_ | |||
| * T5: `Small, Base, Large, 3B, and 11B Architecture <https://github.com/google-research/text-to-text-transfer-transformer>`_ | |||
| * Flan-T5: `Small, Base, Large, XL, and XXL Architecture <https://github.com/google-research/t5x>`_ | |||
There was a problem hiding this comment.
Don't actually support Flan-T5 small due to non divisible # of attention heads.
| @@ -55,7 +55,6 @@ jobs: | |||
| python3 -m pip --quiet install sentencepiece | |||
| python3 -m pip --quiet install tqdm | |||
| python3 -m pip --quiet install expecttest | |||
| python3 -m pip --quiet install transformers | |||
There was a problem hiding this comment.
No longer need transformers install for integration tests.
| decoder_attention_mask=self.decoder_padding_mask, | ||
| output_hidden_states=True, | ||
| output_attentions=True, | ||
| model = T5Bundle.build_model_from_huggingface_ckpt(model_path, encoder_only=is_encoder_only) |
There was a problem hiding this comment.
The above already tests the outputs of HF models, so we just need to confirm that we can load the weights from a HF file and that the model can run.
| @@ -238,12 +239,12 @@ def build_model_from_huggingface_ckpt( | |||
|
|
|||
| for i in range(config.num_decoder_layers): | |||
| if config.is_gated_act: | |||
| t5_model_state_dict[f"encoder.layers.{i}.linear1_0.weight"] = hf_weights[ | |||
There was a problem hiding this comment.
This was a bug.
There was a problem hiding this comment.
I need to confirm that the weights I generated before are correct.
Could you create a release tracker issue similar to #1766? |
| our_output["decoder_hidden_states"][i], hf_output.decoder_hidden_states[i] | ||
| ), f"Mismatched hidden states for decoder layer {i}" | ||
|
|
||
| def test_t5_bundler_load_hf_ckpt_pretrained_encoder_only(self) -> None: |
There was a problem hiding this comment.
Why do we get rid of all of these tests?
There was a problem hiding this comment.
We do the actual testing of the weights in the above tests. Here we just want to make sure the code can be loaded and ran.
* Add tests for loading Flan-T5 weights from HF checkpoints * Add expected outputs and update tests for Flan * Add newline at end of file * pin transformers version for testing * Simplify test for HF loading * Fix linting * Fix integration tests w/ proper download path
* Add tests for loading Flan-T5 weights from HF checkpoints * Add expected outputs and update tests for Flan * Add newline at end of file * pin transformers version for testing * Simplify test for HF loading * Fix linting * Fix integration tests w/ proper download path
This PR adds tests for the Flan-T5 weights and confirms that
build_hf_checkpoint_from_pathworks w/ Flan.