Adpat timeout value of torch.dist.init_process_group depending on elapsed time

[eunwoosh]

Summary

When huge dataset is used, more than 30 seconds is used to prepare dataset, which raises an error when training with multi GPU because current torch.dist.init_process_group timeout is set as 30.
So, I implemented a feature to adapt timeout value if process start time is given.

How to test

Checklist

• I have added unit tests to cover my changes.​
• I have added integration tests to cover my changes.​
• I have added e2e tests for validation.
• I have added the description of my changes into CHANGELOG in my target branch (e.g., CHANGELOG in develop).​
• I have updated the documentation in my target branch accordingly (e.g., documentation in develop).
• I have linked related issues.

License

• I submit my code changes under the same Apache License that covers the project.
  Feel free to contact the maintainers if that's a concern.
• I have updated the license header for each file (see an example below).

# Copyright (C) 2023 Intel Corporation
# SPDX-License-Identifier: Apache-2.0

[goodsong81]

Sorry for the nitpick but could you fix the changelog while resoling the conflict anyway? :)

[sungmanc]

I have a question, are preparing the dataset and dist group initialization conducted in parallel? why prepare dataset affects the dist group initialization?

[eunwoosh]

I have a question, are preparing the dataset and dist group initialization conducted in parallel? why prepare dataset affects the dist group initialization?

Spawning multi process is executed after preparing dataset. So, When main process initializes first and waits for child process initialization, child process should prepare dataset first before initialization. So if dataset preparation is more than timeout seconds, timeout error is raised.

[sungmanc]

I have a question, are preparing the dataset and dist group initialization conducted in parallel? why prepare dataset affects the dist group initialization?

Spawning multi process is executed after preparing dataset. So, When main process initializes first and waits for child process initialization, child process should prepare dataset first before initialization. So if dataset preparation is more than timeout seconds, timeout error is raised.

Understood, thanks for the kind explanation

[eunwoosh]

Sorry I add a commit to resolve conflict of CHANGELOG.md. Could you review PR again, @sungmanc @goodsong81

[sungmanc]

No problem :)