Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

refactor commands that accept multiple input files to use improved process_input helper #1496

Merged
merged 7 commits into from
Dec 26, 2023

Conversation

jqnatividad
Copy link
Collaborator

@jqnatividad jqnatividad commented Dec 26, 2023

As per the function doc of process_input

/// Process the input files and return a vector of paths to the input files
///
/// If the input is empty, try to copy stdin to a file named stdin in the passed temp directory
/// If the input is empty and stdin is empty, return an error
/// If it's not empty, check the input files if they exist, and return an error if they don't
///
/// If the input is a directory, add all the files in the directory to the input
/// If the input is a file with the extension ".infile-list" read the file, and add each line as a
/// file to the input If the input is a file, add the file to the input
/// If the input are snappy compressed files, uncompress them before adding them to the input

resolves #1293 for cat. headers command also now uses process_input and gained infile-list support along with other process_input capabilities.

Commands that previously used process_input now also support .infile-list files - sqlp and to commands.

"infile-list" files is qsv's flavor of the "infile-list" support of csvtk as per #1293

In our implementation, providing a file with the ".infile-list" extension to commands that support it (currently, `sqlp` and `to`) will read the file as a list of input files to use for the command.

Will add ".infile-list" support  to `cat` and `headers` command as well
- to accept &[PathBuf] instead of &[String]
- refactor process_input helper to make custom_empty_stdin_errmsg parm have a default value, to reduce unnecessary literals using process_input
adapt to use new process_input with default custom_stdin_errmsg
src/cmd/to.rs Dismissed Show dismissed Hide dismissed
src/cmd/to.rs Dismissed Show dismissed Hide dismissed
@jqnatividad jqnatividad merged commit c162bd4 into master Dec 26, 2023
14 checks passed
@derekmahar
Copy link

The directory argument is a nice addition. Now commands can operate on all CSV files in a directory that contains a very large number of files instead of providing a text file that contains the names of these files. A directory is itself a file that contains a list of file names!

@jqnatividad jqnatividad deleted the 1293-infile-list-support branch January 11, 2024 18:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Add option to cat that accepts the name of a file that contains a list of input CSV data file names.
2 participants