-
-
Notifications
You must be signed in to change notification settings - Fork 11
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Multi-GPU does not upload to Hub Web #606
Comments
👋 Hello @willuhmjs, thank you for raising an issue about Ultralytics HUB 🚀! Please visit our HUB Docs to learn more:
If this is a 🐛 Bug Report, please provide screenshots and steps to reproduce your problem to help us get started working on a fix. If this is a ❓ Question, please provide as much information as possible, including dataset, model, environment details etc. so that we might provide the most helpful response. We try to respond to all issues as promptly as possible. Thank you for your patience! |
@willuhmjs hello! 👋 Thank you for bringing this issue to our attention. It seems you're encountering a problem with data not uploading to the Ultralytics HUB when training with multiple GPUs. This is indeed unusual, and we appreciate your detailed report. A few things to consider that might help resolve this issue:
For further troubleshooting and guidance, please refer to our documentation at https://docs.ultralytics.com/hub. If the issue persists, feel free to provide additional details such as logs or error messages, and we'll be happy to assist further. Thanks for your contribution to the Ultralytics community! 🚀 |
I'm having the same problem, using 8 A100 for calculations |
one gpu just work fine |
@darkkun0824 hello again! 👋 It's insightful to know that the issue is specifically occurring in a multi-GPU setup, especially with high-end GPUs like the A100s, but works fine with a single GPU. This narrows down the scope of the problem significantly. Given this additional context, here are a few more tailored suggestions:
If these steps do not resolve the issue, we recommend gathering detailed logs from your training sessions and reaching out for further support through our documentation portal at https://docs.ultralytics.com/hub. Detailed logs can provide crucial insights for deeper troubleshooting. Thank you for your patience and for contributing to the Ultralytics community! We're here to help. 🌟 |
👋 Hello there! We wanted to give you a friendly reminder that this issue has not had any recent activity and may be closed soon, but don't worry - you can always reopen it if needed. If you still have any questions or concerns, please feel free to let us know how we can help. For additional resources and information, please see the links below:
Feel free to inform us of any other issues you discover or feature requests that come to mind in the future. Pull Requests (PRs) are also always welcomed! Thank you for your contributions to YOLO 🚀 and Vision AI ⭐ |
Search before asking
HUB Component
Training
Bug
When training with multiple GPUs, the data is not uploaded to HUB. It stays at 0% until it logs as disconnected.
Environment
Linux on HPC, latest Python and Ultralytics
Minimal Reproducible Example
Additional
No response
The text was updated successfully, but these errors were encountered: