add support for wildcard/patterns

[dmpetrov]

Sometimes only a subset of files is needed when the user runs import or pull data from a data directory. It is convenient to define a file pattern for an import.

From https://discuss.dvc.org/t/working-with-a-small-subset-of-remote-data/541
Related: #4705, #4815

Patterns to implement:

• simple wildcard dvc pull cats-dogs/data/train/dogs/*.img
• whole wildcard dvc pull cats-dogs/data/train/{dogs,cats}/???.img
• globstar/ricursive dvc pull cats-dogs/data/train/**/*.img
• iterator dvc pull cats-dogs/data/train/dogs/%C.img?counter=1:100
• date dvc pull users/%Y/%m/%d/users.csv?startdata=2020-09-01,enddate=now,ignoremissing

The first three patterns should use a regular Unix file syntax. While the last two require a special language to define the pattern - we need to find a good examples.

[dmpetrov]

Based on my experience I'd assign the priorities like this:

1. simple wildcard *
2. globstar/ricursive **
3. data path/file-%Y-%m-%d.txt
4. iterator/count %C
5. whole wildcard - ?, ., {}

But we need to agree on the common pattern format (how to reflect the pattern in dvc-files) before implementing even the first step.

[efiop]

Regarding the first step

simple wildcard dvc pull cats-dogs/data/train/dogs/*.img

support for dir entries will simply require treating existing filter_info in

dvc/dvc/output/base.py

Line 403 in 6a9ab9c

def collect_used_dir_cache(

appropriately. Right now we only check if filter equals or contains other files.

Regular glob patterns are clearer than the proposed date/counter selectors, those need some research on existing solutions. So this is a multilayer ticket that has a lot of special cases.

[karajan1001]

Related #4419.

[ju0gri]

I will be taking a stab at implementing the first step for this issue.

[efiop]

Related #4912

[dmpetrov]

Sound slike at least this check box could be marked, per #4864?

@jorgeorpinel #4864 is only about dvc add. pull/push/import are missing for checking the first checkbox.

[ju0gri]

I can continue adding this functionality for all commands, if that's alright.

[efiop]

@ju0gri Thanks for looking into it! 🙏

[jorgeorpinel]

Question:

We've introduced the --glob option to a few commands to implement some of these patterns above (the ones covered by glob i.e. 1,2, and 5 from #4816 (comment))

Is the option temporary, expecting to make this default the behavior at some point? Otherwise I think we may need a better term as discussed in #4976 (comment), and even more now that I see patterns 3 (iterator) and 4 (date) which I think aren't covered by "glob".

Thanks

[jorgeorpinel]

Hi, can we include the discussion about wildcards in stage output and dependency definitions (in dvc.yaml and maybe also run/stage add -od)? It's not listed in the check boxes of this issue's description, but it's mentioned in #1462 (comment) (2.A and B). Or I can make a separate issue for outs/deps.

A couple users have brought up the need for this in https://discuss.dvc.org/t/managing-pipelines-operating-per-dataset-element/613

[tibor-mach]

Seconding @jorgeorpinel on this, there is some new demand for wildcards on dvc stage outputs