[loss OVERFLOW] Several Issues

[stas00]

Describe the bug

Splitting of from #1593

This Issue documents multiple issues wrt OVERFLOW event

1. ZeRO2 OVERFLOW goes from initial_scale_power set in config to the one where it gets resolved or can't be resolved in one step (compared to zero3 where it reduces the power one step at a time and reports each decrement) As you can see in port OVERFLOW log to ZeRO-2 #1593 (comment) I get the report of power 1 right away when initial_scale_power=16

OVERFLOW! Rank 0 Skipping step. Attempted loss scale: 1, reducing to 1

2. ZeRO3 OVERFLOW has a one-off issue: Attempted loss scale: 65536, reducing to 65536 - i.e it's not scaling the first time around.

3. Both ZeRO2 and ZeRO3 should probably assert if:
   a. loss is nan - related: should dynamic scaling and overflow check happen only at the beginning? #931
   b. Attempted loss scale: 1, reducing to 1 happens - clearly this is broken.

   since it's impossible to recover from either. But Deepspeed optimizer skips the step and gets stuck in the above scaling

[stas00]

Here is another reported issue where a user reports this endless loop that can't resolve overflow and where I think Deepspeed ought to assert:
huggingface/transformers#14834

[stas00]

Here is another use-case: huggingface/transformers#15570 (comment)

Can we please make deepspeed assert as soon as it hits: Attempted loss scale: 1, reducing to 1

That way the user might be able to figure out what the problem is since he can report the error, but otherwise the training continues, while not updating any weights and at the end the user gets a pile of random bits.

---

Moreover, in this case a higher "initial_scale_power" was the fix:
huggingface/transformers#15570 (comment)

why then the optimizer was trying to lower the scaling factor and not raise it? It appears that initially it always goes down, but I think sometimes it has to go up instead.

@tjruwase

[Quentin-Anthony]

I believe this has been resolved in #1958

[stas00]

yes, definitely item 3 was resolved - not sure about the rest

[Quentin-Anthony]

yes, definitely item 3 was resolved - not sure about the rest

Gotcha. I think item 1 has also been resolved because the loss overflow logic has since been unified between Zero-1/2.

As for item 2 the reason such messages are printing is because of fp16 hysteresis . I agree it's confusing, so I just made a PR to clarify those messages at #3008.

Assuming my PR gets merged, loss overflow logs would look like:

...
OVERFLOW! Rank 0 Skipping step. Attempted loss scale: 65536, but hysteresis is 3. Reducing hysteresis to 2
OVERFLOW! Rank 0 Skipping step. Attempted loss scale: 65536, but hysteresis is 2. Reducing hysteresis to 1
OVERFLOW! Rank 0 Skipping step. Attempted loss scale: 65536, reducing to 32768
OVERFLOW! Rank 0 Skipping step. Attempted loss scale: 32768, reducing to 16384
...