Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[FEATURE] Allow Steps to produce artifacts #834

Closed
gabrielmbmb opened this issue Jul 26, 2024 · 0 comments · Fixed by #871
Closed

[FEATURE] Allow Steps to produce artifacts #834

gabrielmbmb opened this issue Jul 26, 2024 · 0 comments · Fixed by #871
Assignees
Labels
enhancement New feature or request
Milestone

Comments

@gabrielmbmb
Copy link
Member

Is your feature request related to a problem? Please describe.
Some Steps might produce some artifacts (faiss index for example) that could be reused after the pipeline has finished its execution. distilabel currently doesn't allow to save any artifact, or does not provide a clear way, for an Step to save artifacts.

Describe the solution you'd like
Steps can produce and save artifacts that can be later upload along the produced dataset to the Hub. To do so, Step offers a method get_artifact_directory that can be used within the step to a get directory (from the cache of the pipeline) that can be used by the step to store produced artifacts.

@gabrielmbmb gabrielmbmb added the enhancement New feature or request label Jul 26, 2024
@gabrielmbmb gabrielmbmb added this to the 1.4.0 milestone Jul 26, 2024
@gabrielmbmb gabrielmbmb self-assigned this Jul 26, 2024
@gabrielmbmb gabrielmbmb linked a pull request Aug 8, 2024 that will close this issue
@plaguss plaguss closed this as completed Aug 14, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
Status: Done
Development

Successfully merging a pull request may close this issue.

2 participants