Skip to content

Commit

Permalink
Make the DBF to XBRL reshaping process more robust
Browse files Browse the repository at this point in the history
Progress on #2012 #2014

* Fixed a bug in how the DBF row numbers that need to be mapped are
  identified. Now it looks for any time the row_literal associated with
  a row number has changed from one year to the next, rather than
  selecting the first instance of each distinct combination of
  row_literal and row_number.
* Also discovered that there's an obscure row_status field that
  differentiates between annual (A) and quarterly (Q) row literals, and
  is part of the f1_row_lit_tbl primary key, but it only shows up in
  association with the f1_schedules_list table. I integrated it but...
  maybe that table should just be excluded from the row mapping
  template?
* Added some (janky) helper functions to pudl.transform.ferc1 to manage
  the generation of the row maps. This location is temporary. They
  should probably become methods of a Ferc1 abstract transformer class
  for reshaped tables, or maybe end up in a different module. Not sure
  how they'll end up getting used yet though.
* Updated the dbf_to_xbrl.csv file to include all of the possible rows
  that could need mapping (4270 in total).
* Removed the XBRL specific metadata fields from the dbf_to_xbrl.csv
  file, since they should (hopefully) be available programmatically from
  the metadata @zschira is extracting from the XBRL taxonomies, and can
  be joined to this table based on the xbrl_column_stem.
* Updated the plant_in_service transform to use the new row map. Need to
  test on all of the years.
  • Loading branch information
zaneselvans committed Nov 2, 2022
1 parent e2b4752 commit e20a73f
Show file tree
Hide file tree
Showing 2 changed files with 4,386 additions and 201 deletions.
Loading

0 comments on commit e20a73f

Please sign in to comment.