You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Is your feature request related to a problem? Please describe.
Some Steps might produce some artifacts (faiss index for example) that could be reused after the pipeline has finished its execution. distilabel currently doesn't allow to save any artifact, or does not provide a clear way, for an Step to save artifacts.
Describe the solution you'd like Steps can produce and save artifacts that can be later upload along the produced dataset to the Hub. To do so, Step offers a method get_artifact_directory that can be used within the step to a get directory (from the cache of the pipeline) that can be used by the step to store produced artifacts.
The text was updated successfully, but these errors were encountered:
Is your feature request related to a problem? Please describe.
Some
Step
s might produce some artifacts (faiss index for example) that could be reused after the pipeline has finished its execution.distilabel
currently doesn't allow to save any artifact, or does not provide a clear way, for anStep
to save artifacts.Describe the solution you'd like
Step
s can produce and save artifacts that can be later upload along the produced dataset to the Hub. To do so,Step
offers a methodget_artifact_directory
that can be used within the step to a get directory (from the cache of the pipeline) that can be used by the step to store produced artifacts.The text was updated successfully, but these errors were encountered: