Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

integrations: hf accelerate #731

Closed
dberenbaum opened this issue Nov 3, 2023 · 4 comments
Closed

integrations: hf accelerate #731

dberenbaum opened this issue Nov 3, 2023 · 4 comments
Assignees
@dberenbaum
Labels
A: frameworks Area: ML Framework integration p1-important Include in the next sprint

Comments

@dberenbaum
Copy link
Collaborator

DVCLive already supports transformers, but we should add support for accelerate.
@dberenbaum dberenbaum added p1-important Include in the next sprint A: frameworks Area: ML Framework integration labels Nov 3, 2023
@dberenbaum dberenbaum added this to DVC Nov 3, 2023
@github-project-automation github-project-automation bot moved this to Backlog in DVC Nov 3, 2023
@dberenbaum dberenbaum moved this from Backlog to Todo in DVC Nov 3, 2023
@dberenbaum dberenbaum self-assigned this Nov 3, 2023
@dberenbaum dberenbaum mentioned this issue Nov 3, 2023
Closed
2 tasks
@dberenbaum
Copy link
Collaborator Author

dberenbaum commented Nov 4, 2023 •
edited
Loading

Started on #732, but in distributed jobs, AFAICT there is a bug in accelerate with custom/external trackers (the same code works when put inside accelerate but not when outside the library). Rather than try to go deeper into it, I think it's better to finally make the move to migrate these callbacks to their respective ML frameworks.

@dberenbaum
Copy link
Collaborator Author

Related to #221

@dberenbaum dberenbaum moved this from Todo to In Progress in DVC Nov 14, 2023
@dberenbaum dberenbaum moved this from In Progress to Done in DVC Nov 21, 2023
@dberenbaum dberenbaum moved this from Done to In Progress in DVC Nov 21, 2023
@dberenbaum
Copy link
Collaborator Author

Completed in huggingface/accelerate#2139

@github-project-automation github-project-automation bot moved this from In Progress to Done in DVC Nov 22, 2023
@shcheklein
Copy link
Member

Amazing, @dberenbaum ! good stuff.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@dberenbaum dberenbaum

Labels
A: frameworks Area: ML Framework integration p1-important Include in the next sprint
Projects
No open projects
DVC
Archived in project
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

adds hf accelerate tracker iterative/dvclive
2 participants
@dberenbaum @shcheklein