Write script to download and parse datasets

[tmmurali]

Instead of a manual, piecemeal approach to downloading datasets, implement a script that will download every dataset, and parse it into a format that the rest of the code can use. Some of the files we download may not change over the course of a few months (e.g., tissue-specific networks) while others may change rapidly (e.g., COVID-19-specific gene sets).

Initially, we have to regularly update the script every time we see a new source of interesting and useful information. In the longer run, the sources will stabilise and the script will mature.

A config (YAML) file can record and document the different sources, URL, data types, and directories where we will store the files.

[jlaw9]

I implemented a master script and accompanying config file to download and parse network files (See commit 70284f4). Currently only tested using the TissueNet v2 collections. Planning to add gene sets soon.

[tmmurali]

Excellent work! Noticed that the config file has your directory name. We need to make it generic somehow.

[jlaw9]

Oh my bad, that was just for testing. It will be the generic datasets folder. I'll fix that.

[tmmurali]

1. Should we add fields in the config-file that name the key columns that we should parse, e.g., gene id, drug id, etc?
2. Virus-host networks may contain multiple species, certainly on the virus side and maybe on the host side as well. How do the config file and the parser handle such cases?

[jlaw9]

In response to your questions:

1. Yes, but those fields will depend on the file type. For gene sets, I added the file_type field which currently only supports gmt, but I can imagine a table format where we would need to specify the columns.
2. Currently the config file has a species field, but it is unused by the parser. When we have data for multiple species, I plan to have the parser use the mapping file for the specified species.

[jlaw9]

Added the ability to download and parse gene sets in this commit: dc0b518

[tmmurali]

Can we close this issue now? For every new dataset we want to add, we can open another issue.

[jlaw9]

Sounds good.