Standardize filenames and add production DOIs #82
Conversation
All of the other archivers use a file naming pattern like: dataset-part1-part2.ext where: - dataset is the name of the dataset (e.g. ferc1, eia923, mshamines) - part1 is the value of the first partition dimension contained in the file (e.g. ca, ny, wy, for states, 2019 if it's a year) - part2 is the value of the second partition dimension contained in the file, if applicable. - ext is the file name extension indicating file type. I've changed the naming for phmsagas to follow the `dataset-*` part of the naming convention, but have left the filename in place rather than using only the dataset and start year, since if we just use the start year it looks like years are missing in the archive, as each file contains several years of data. The dataset and start_year partitions are still used in datapackage.json to refer to the individual resources for programmatic purposes. This came up when working on #79
Use the same pattern of filenames that the other archivers use: dataset-partition.ext For the `mshamines` dataset this looks like: - `mshamines-accidents.zip` - `mshamines-accidents_definitions.txt` - `mshamines-mines_prod_quarterly.zip` - `mshamines-mines_prod_quarterly_definitions.txt` (All years of data are contained in a single file, but each file contains a different data table.)
zaneselvans
added
eia176
|
|
||
| download_path = self.download_directory / f"phmsagas_{file}.zip" | ||
| download_path = self.download_directory / f"{self.name}-{filename}.zip" |
There was a problem hiding this comment.
All of the other archivers use a file naming pattern like:
dataset-part1-part2.ext
where:
- dataset is the name of the dataset (e.g. ferc1, eia923, mshamines)
- part1 is the value of the first partition dimension contained in
the file (e.g. ca, ny, wy, for states, 2019 if it's a year) - part2 is the value of the second partition dimension contained in
the file, if applicable. - ext is the file name extension indicating file type.
I've changed the naming for phmsagas to follow the dataset-* part of the naming convention, but have left the filename in place rather than using only the dataset and start year, since if we just use the start year it looks like years are missing in the archive, as each file contains several years of data.
The dataset and start_year partitions are still used in datapackage.json to refer to the individual resources for programmatic purposes.
There was a problem hiding this comment.
I agree with the decision not to use start date for the filenames here. But in this case the expected output here is a file like: phmsagas-annual_underground_natural_gas_storage_2017_present.zip ?
Would it be worth reversing the substitution from "-" to underscore in line 69 to result in filenames like phmsagas-annual-underground-natural-gas-storage-2017-present.zip? Having both "-" and underscore is a bit visually confusing here and at least this would bring us towards having a consistent snake case filename, even if the structure is slightly different than those of other archivers.
There was a problem hiding this comment.
I wasn't totally thrilled about this combo either. It also occurs in the MSHA archives (though there we do use the actual partition values, rather than the filename). Really we don't want anybody parsing these filenames directly, and they should be using the metadata in datapackage.json to address individual partitions. I guess even what we have right now isn't literally the original filenames since we're changing the one with dashes to underscores. We haven't had to deal with this data-table based partitioning before. Previously it was either one file per timeslice (with individual tables inside it) or in the case of EPA CEMS, files that were all part of the same table, but partitioned the data based on values (year and state) within that one table.
I just wanted to somehow differentiate the phmsagas part of the name from the other parts, since it's a different kind of thing, and it was possible to follow the same convention as all the FERC and EIA data. And I do like that the value of the data partition is actually in the filename. Maybe I should have just gone with
phmsa-dataset-startyear.zip
Another thing that we're seeing come up is overlap in partitions where what differentiates them is the file format, e.g. years where both XBRL and DBF data are available in FERC, which seems like another type of partitioning we might want to work with and make obvious in the filenames and partition metadata.
|
|
||
| download_path = self.download_directory / f"phmsagas_{file}.zip" | ||
| download_path = self.download_directory / f"{self.name}-{filename}.zip" |
There was a problem hiding this comment.
I agree with the decision not to use start date for the filenames here. But in this case the expected output here is a file like: phmsagas-annual_underground_natural_gas_storage_2017_present.zip ?
Would it be worth reversing the substitution from "-" to underscore in line 69 to result in filenames like phmsagas-annual-underground-natural-gas-storage-2017-present.zip? Having both "-" and underscore is a bit visually confusing here and at least this would bring us towards having a consistent snake case filename, even if the structure is slightly different than those of other archivers.
Tweak the archive filenames slightly to match naming patterns from other archivers.
Add new production DOIs for:
eia176eiawatermshaminesphmsagasI used these archivers to initialize archives on the production Zenodo site, and also updated a bunch of other archivers (See #79). However, I didn't update the FERC archives because of the issue with unique IDs in the RSS feeds described in #73.
Closes #79