Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Data sync: allow more advanced workdir and output specs #590

Closed
Tracked by #362
dberenbaum opened this issue May 20, 2022 · 3 comments · Fixed by #699
Closed
Tracked by #362

Data sync: allow more advanced workdir and output specs #590

dberenbaum opened this issue May 20, 2022 · 3 comments · Fixed by #699
Assignees
Labels
enhancement New feature or request external-request You asked, we did p1-important High priority resource-task iterative_task TF resource storage ui/ux User interface/experience

Comments

@dberenbaum
Copy link

It would be great for workdir/output to support:

  • Lists of multiple paths
  • Globs or regexes
@0x2b3bfa0
Copy link
Member

0x2b3bfa0 commented May 20, 2022

Storage syntax can be easily extended to support rclone filters to include and exclude several paths or globs from data transfers:

@0x2b3bfa0 0x2b3bfa0 added enhancement New feature or request storage ui/ux User interface/experience resource-task iterative_task TF resource labels May 20, 2022
@dacbd
Copy link
Contributor

dacbd commented Jun 3, 2022

+1 for a exclude option,

storage {
  workdir = "."
  exclude = [
    ".dvc/cache",
    "venv"
  ]
  output = "./reports"
}

@casperdcl casperdcl added the p1-important High priority label Jun 8, 2022
@casperdcl casperdcl added the external-request You asked, we did label Oct 17, 2022
@casperdcl
Copy link
Contributor

Quick note: exclude should be a list[str] relative to workdir

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request external-request You asked, we did p1-important High priority resource-task iterative_task TF resource storage ui/ux User interface/experience
Projects
None yet
Development

Successfully merging a pull request may close this issue.

5 participants