Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Make the DBF to XBRL reshaping process more robust
Progress on #2012 #2014 * Fixed a bug in how the DBF row numbers that need to be mapped are identified. Now it looks for any time the row_literal associated with a row number has changed from one year to the next, rather than selecting the first instance of each distinct combination of row_literal and row_number. * Also discovered that there's an obscure row_status field that differentiates between annual (A) and quarterly (Q) row literals, and is part of the f1_row_lit_tbl primary key, but it only shows up in association with the f1_schedules_list table. I integrated it but... maybe that table should just be excluded from the row mapping template? * Added some (janky) helper functions to pudl.transform.ferc1 to manage the generation of the row maps. This location is temporary. They should probably become methods of a Ferc1 abstract transformer class for reshaped tables, or maybe end up in a different module. Not sure how they'll end up getting used yet though. * Updated the dbf_to_xbrl.csv file to include all of the possible rows that could need mapping (4270 in total). * Removed the XBRL specific metadata fields from the dbf_to_xbrl.csv file, since they should (hopefully) be available programmatically from the metadata @zschira is extracting from the XBRL taxonomies, and can be joined to this table based on the xbrl_column_stem. * Updated the plant_in_service transform to use the new row map. Need to test on all of the years.
- Loading branch information