-
-
Notifications
You must be signed in to change notification settings - Fork 16.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
How to use DDP? #8283
Comments
@AyushExel it looks like we should disable LOGGER for all RANKs except for RANK in (0, -1) to avoid multiple printouts like the AMP check above: Line 534 in 1156a32
I think I can do this here: Lines 82 to 99 in 1156a32
EDIT: Oh, haha I've already done this on L88, this line is just printing out because it's set to warning for non RANK (0, -1) messages. Maybe I should just disable it entirely or set it to logging.ERROR for non RANK(0, -1)? |
Improve DDP console response in #8283
@lizhiying-2019 your error doesn't look familiar. For full DDP examples please see our Multi-GPU tutorial below which recommends using our Docker image for all DDP trainings. If you have a reproducible issue inside our Docker container please let us know with exact commands to reproduce. If we can not reproduce there is no action for us to take. YOLOv5 Tutorials
How to create a Minimal, Reproducible ExampleWhen asking a question, people will be better able to provide help if you provide code that they can easily understand and use to reproduce the problem. This is referred to by community members as creating a minimum reproducible example. Your code that reproduces the problem should be:
For Ultralytics to provide assistance your code should also be:
If you believe your problem meets all the above criteria, please close this issue and raise a new one using the 🐛 Bug Report template with a minimum reproducible example to help us better understand and diagnose your problem. Thank you! 😃 |
Improve DDP console response in #8283
Improve DDP console response in ultralytics#8283
Search before asking
Question
I do not know what should I do. When I run train.py without DDP, it can run, but slowly. So I want to use DDP to speed up. Always display this bug. Could you tell me how to solve this problem? Please.
Additional
No response
The text was updated successfully, but these errors were encountered: