Introduction

filesdb is a simple utility to of files generated under different experimental conditions.

Motivation

Keeping track of files generated in computational experiments is very important, and gets complicated quickly. For simple problems, using filenames is sufficient (e.g., a-1_b-2.txt). However, sometimes code changes, or more parameters become important, and now filenames may look like a-1_b-2_c-5_version-2.txt, breaking your naming scheme and complicating any code you might have that parses file names.

filesdb solves these problems by tracking all your files in an sqlite database. Additional parameters can be added at any time. The database can quickly be searched by parameter values using a command line or python interface.

Examples

Say you are have some code that generates an output file and takes two parameters, a and b. To get a new filename and add it to the database:

Bash	Python
filename=$(filesdb add a=${a} b=${b})	filename = filesdb.add({'a': a, 'b': b})

Then ${filename} can be passed as the output parameter to your code. (filesdb does not actually create the file.) This process can be repeated for different values of a and b. You can also specify the filename yourself:

Bash	Python
filesdb add --filename=myfile.txt a=${a} b=${b}	filesdb.add({'a': a, 'b': b}, filename='myfile.txt')

Note that keys cannot end in "!".

To list all the files with a=${a}

Bash	Python
filesdb search a=${a}	entries = filesdb.search({'a': a})

Later, if you decided you also want to track files by a new parameter, (e.g, c), you can simply add a new parameter:

Bash	Python
filename=$(filesdb add a=${a} b=${b} c=${c})	filename = filesdb.add({'a': a, 'b': b, 'c': c})

Finally, let's say you discover a bug and your code, and all files generated with a=0 are invalid and you decide to delete them.

Bash	Python
filesdb delete a=0	deleted_entries = filesdb.delete({'a': 0})

This WILL delete the file from the file system as well as remove it from the database. The bash version will also print the deleted entries. To make sure that you're only deleting the files you want, delete has a dry run option, which prints (or returns, in the python case) the entries to be deleted but does not actually delete them.

Bash	Python
filesdb delete --dry_run a=0	entries = filesdb.delete({'a': 0}, dryrun=True)

Advanced Features

not equal comparisons

Both search and delete support "not equal" comparisons.

Bash	Python
filesdb search a!=${a}	entries = filesdb.search({'a!': a})

Note that this syntax prevents keys from ending in "!".

copy

The copy command copies a file and its data base entry to a new directory. For example:

Bash	Python
N/A	filesdb.copy(filename, outdir, db='files.db', wd='.', outdb='files.db')

Will copy the ${filename} in the current directory to ${outdir}, and copy its row entry to files.db in the current director to files.db in ${outdir}.

Note that this method does not currently have a command line interface

merge

Merge one database into another.

Bash	Python
filesdb --wd='.' --db=files.db merge files2.db	filesdb.merge('files2.db', 'files.db', wd='.')

will add all entries from files2.db to files.db (if they aren't already present)

Environments

For values that are mostly the same for every experiment (e.g., the versions of each software package used), filesdb supports a separate environment dictionary. (Internally, this is stored as a separate table, with an identifying environment hash value in the main table.)

filesdb.add({'a': a, 'b': b}, filename='myfile.txt',
            environment={'pkgname1': '0.1.1', pkgname2: '2.1.0'})

To return environment data with search results (via a JOIN):

filesdb.search({}, with_environments=True)

To search based on environment values:

filesdb.search({}, environment={'pkgname1': '0.1.1'})