Make file filters more intuitive

[hannes-ucsc]

1. rename --data-files to --data-filter
2. rename --metadata-files to --metadata-filter
3. Add --no-data as an alias of --data-filter ""
4. Add --no-metadata as an alias of --metadata-filter ""
5. Add all of the above to hca dss download-manifest as well

Specifying both --no-data and --no-metadata simultaneously could be disallowed but we will be adding the download of the actual bundle object (bundle.json) as a third artifact. With --no-data --no-metadata a user would only get the bundle object.

[hewgreen]

This comment may be helpful but ignore it if it isn't.

The --data-files flag means don't give me data files and the --metadata-files flag means don't give me the metadata files. It seems more intuitive for it to work the other way around. Also rather than passing "" it may be better to allow an empty arg to be taken to mean all.

For example, I wanted to only download metadata files so I expected to pass the flag --metadata-files with no argument to do this.

[kozbo]

@hewgreen can you give us an example command line?

[hewgreen]

@malloryfreeberg @willrockout @rolando-ebi
We mentioned that the hca dss download arguments could be more intuitive. Do you guys have any suggestions before I wade in?

This command will only give you the links file:
hca dss download --bundle-uuid <BUNDLE-UUID> --replica aws --data-files “” --metadata-files "links.json”

dropping --data-files “” will give you all the data files

Is this intuitive?

[willrockout]

@hewgreen I'm in agreement with you I would expect --data-files or --metadata-files to just give me those files. So something like the below would give me all the metadata files for a bundle:

hca dss download --bundle-uuid <BUNDLE-UUID> --replica aws --metadata-files

Also, I think it would be beneficial to have an example for the possible arguments to feed to --data-files and --metadata-files as at least for me it wasn't super intuitive.

[hewgreen]

Just regarding --data-files and --metadata-files flags

no flags = give me everything in the bundle
just --data-files = give me all data files
just --metadata-files = give me all metadata files
--metadata-files "links.json" = just give me the links file
--metadata-files "donor*.json" = give me all donor metadata files
so --data-files and --metadata-files would imply = give me no files

Does that look right? @malloryfreeberg @willrockout @rolando-ebi

[willrockout]

so --data-files and --metadata-files would imply = give me no files

@hewgreen I don't think the double flag should mean give me no files. I don't really see a use case where you'd use download and not want any files.

[willrockout]

The rest looks good!

[hewgreen]

I'm not saying you every want that. I'm just saying thats where the logic follows. It would be the same as the current --data-files “" --metadata-files "".

[hannes-ucsc]

Another proposal:

1. Rename --data-files to --data-filter and --metadata-files to --metadata-filter
2. Add two mutually exclusive options
   --data-only (only download data files, ignore metadata files)
   --metadata-only (only download metadata files, ignore data files)

--data-only --metadata-only would be disallowed and lead to an error.

Internally --data-only is implemented as --metdata-filter "" and --metadata-only is implemented as --data-filter="".

[hannes-ucsc]

I propose another slight tweak to the above:

1. rename --data-files to --data-filter
2. rename --metadata-files to --metadata-filter
3. Add --no-data as an alias of --data-filter ""
4. Add --no-metadata as an alias of --metadata-filter ""
5. Add all of the above to hca dss download-manifest as well

Specifying both --no-data and --no-metadata simultaneously could be disallowed but we will be adding the download of the actual bundle object (bundle.json) as a third artifact. With --no-data --no-metadata a user would only get the bundle object.

[diekhans]

IMHO, --data-only and --metadata-only is a bit clearer. Hannes Schmidt <notifications@github.com> writes:

…

I propose another slight tweak to the above: 1. rename --data-files to --data-filter 2. rename --metadata-files to --metadata-filter 3. Add --no-data as an alias of --data-filter "" 4. Add --no-metadata as an alias of --metadata-filter "" 5. Add all of the above to hca dss download-manifest as well Specifying both --no-data and --no-metadata simultaneously could be disallowed but we will be adding the download of the actual bundle object (bundle.json) as a third artifact. With --no-data --no-metadata a user would only get the bundle object. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or mute the thread.*

[jessebrennan]

@hannes-ucsc I assume that this ticket is now to change metadata_files and data_files to match the spec you outline:

I propose another slight tweak to the above:

1. rename --data-files to --data-filter
2. rename --metadata-files to --metadata-filter
3. Add --no-data as an alias of --data-filter ""
4. Add --no-metadata as an alias of --metadata-filter ""
5. Add all of the above to hca dss download-manifest as well

Specifying both --no-data and --no-metadata simultaneously could be disallowed but we will be adding the download of the actual bundle object (bundle.json) as a third artifact. With --no-data --no-metadata a user would only get the bundle object.

Is that correct?

[hannes-ucsc]

Correct. My problem with --metadata-only instead of --no-data is that being alias of --data-filter "" the semantics would break down if there were a third category of files, in addition to data and metadata. If the third type of file were called, say, auxiliary, with --metadata-only being equivalent to --data-filter "" you would get metadata and auxiliary files. That would contradict the name.

As I write this I notice that there will be a 3rd category of files, as I mention in the section you quote: bundle.json is neither metadata nor data.

[diekhans]

Convinced, --no-data makes lots of sense.