Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Immediate checkpoint serialization #9437

Merged
merged 4 commits into from
Mar 31, 2024
Merged

Immediate checkpoint serialization #9437

merged 4 commits into from
Mar 31, 2024

Conversation

glenn-jocher
Copy link
Member

@glenn-jocher glenn-jocher commented Mar 31, 2024

πŸ› οΈ PR Summary

Made with ❀️ by Ultralytics Actions

🌟 Summary

Enhancements in model checkpoint serialization for efficiency and speed πŸš€.

πŸ“Š Key Changes

  • Introduced io.BytesIO for in-memory checkpoint (ckpt) serialization before saving.
  • Removed repeated metrics and results processing; now processed once and included in the serialization.
  • Simplified checkpoint saving logic, enhancing code readability and maintainability.

🎯 Purpose & Impact

  • Efficiency: By serializing the model checkpoint once and storing it in memory, the PR significantly reduces the time and disk I/O required during saving.
  • Faster Training: This change minimizes disruptions during the training process, allowing for smoother and slightly faster training sessions.
  • Cleaner Save Logic: The updated saving method simplifies future modifications and understanding of the code, beneficial for both maintainers and contributors.

Expect smoother training sessions with minimal save-related disruptions, a boon for users running large-scale model training. πŸ‹οΈβ€β™‚οΈπŸ’Ύβœ¨

Copy link

codecov bot commented Mar 31, 2024

Codecov Report

Attention: Patch coverage is 85.71429% with 1 lines in your changes are missing coverage. Please review.

Project coverage is 76.94%. Comparing base (eed01a2) to head (c2a88b4).

Files Patch % Lines
ultralytics/engine/trainer.py 85.71% 1 Missing ⚠️
Additional details and impacted files
@@           Coverage Diff           @@
##             main    #9437   +/-   ##
=======================================
  Coverage   76.94%   76.94%           
=======================================
  Files         117      117           
  Lines       14853    14854    +1     
=======================================
+ Hits        11429    11430    +1     
  Misses       3424     3424           
Flag Coverage Ξ”
Benchmarks 36.73% <0.00%> (-0.01%) ⬇️
GPU 38.87% <85.71%> (+0.01%) ⬆️
Tests 72.07% <85.71%> (+<0.01%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

β˜” View full report in Codecov by Sentry.
πŸ“’ Have feedback on the report? Share it here.

@glenn-jocher glenn-jocher merged commit 479afce into main Mar 31, 2024
13 checks passed
@glenn-jocher glenn-jocher deleted the immediate-serialize branch March 31, 2024 03:53
hmurari pushed a commit to hmurari/ultralytics that referenced this pull request Apr 17, 2024
gkinman pushed a commit to Octasic/ultralytics that referenced this pull request May 30, 2024
puzhiyuan pushed a commit to puzhiyuan/ultralytics that referenced this pull request Oct 18, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant