Improve test speeds by up to 30% in multi-gpu settings

[muellerzr]

What does this PR do?

@stas00 and Nikita (sorry don't know your gh handle!) and I were investigating why accelerate launch is so slow during tests.

Turns out pytorch isn't respecting the monitor_interval default of 0.1 for some reason and it's being set to 5 seconds instead. So, magically do that and the test_multigpu.py is sped up by 30% (33.40s -> 23.29s) and test_core was sped up by 25% (102.23s (0:01:42) -> 76.94s (0:01:16)

Fixes # (issue)

Before submitting

• This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
• Did you read the contributor guideline,
  Pull Request section?
• Was this discussed/approved via a Github issue or the forum? Please add a link
  to it if that's the case.
• Did you make sure to update the documentation with your changes? Here are the
  documentation guidelines, and
  here are tips on formatting docstrings.
• Did you write any new necessary tests?

Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@SunMarc @stas00

[muellerzr]

Will be doing similar benchmarks and fixes over in the transformers side, while we wait for the torch team to get back with a clear answer

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[stas00]

Amazing work, Zach!

[SunMarc]

Wow great investigation ! LGTM !

[muellerzr]

cc @nvassilyev, as I now know your github :)

[BenjaminBossan]

This PR seems to be related, as it rolls back the value to 0.1: pytorch/pytorch#124692 (note that the PR was merged, even though the status is "Closed").

[muellerzr]

Great investigation @BenjaminBossan, looking through the diffs we can indeed see our 5!

Merging as all is well