End-to-end regression test between metaseq and huggingface

[shijie-wu]

We are interested in a full regression test between metaseq and huggingface. As @patrickvonplaten confirmed, the current regression test doesn't cover the merging step. We observed different behaviors between models that require merging (1.3B) and the one that doesn’t (350M) (huggingface/transformers#17545). It’s unclear what causes the difference at the moment due to the lack of an end to end regression test.

To run the full regression test, we would need to load the model into metaseq directly. However, this is not well supported currently (e.g. #78 and #31). It would be great if people at Meta could help perform a full regression test between metaseq and huggingface 😄

@suchenzang @stephenroller

[patrickvonplaten]

@shijie-wu could you specify a bit what is meant by

"We observed different behaviors between models that require merging and the ones that don't"

What exactly is different?

Thanks!

[shijie-wu]

@shijie-wu could you specify a bit what is meant by

"We observed different behaviors between models that require merging and the ones that don't"

What exactly is different?

Thanks!

Hi @patrickvonplaten

I'm referring to huggingface/transformers#17545

[patrickvonplaten]

@shijie-wu, could you by any chance reproduce the result using the metaseq code base - or the opposite show that this doesn't happen with the metaseq codebase? Think this could really help in finding out if there is a bug :-)

[shijie-wu]

@shijie-wu, could you by any chance reproduce the result using the metaseq code base - or the opposite show that this doesn't happen with the metaseq codebase? Think this could really help in finding out if there is a bug :-)

@patrickvonplaten I tried to load OPT-1.3B into metaseq but unfortunately it doesn't work (error message). I would love to conduct the end-to-end regression test once I figure out what went wrong. Similar issues (#78 and #31) have been raised before regarding loading checkpoint into metaseq.

[patrickvonplaten]

Ah yeah I see - could you try to run this snippet frist: #88 (comment)

It should put all the sharded checkpoints into one :-)

[punitkoura]

@patrickvonplaten @shijie-wu I added an end to end integration test between metaseq and HF in this PR #214

The test confirms that the logits and predicted words for -

1. Model parallel checkpoint
2. Metaseq singleton checkpoint
   and 3) HF checkpoint

are the same. Let me know if this test contains what you were looking for. Thanks!

[patrickvonplaten]

Wow awesome thanks a lot @punitkoura !

[suchenzang]

Completed in #214 - thanks @punitkoura !!