First draft of plant_in_service_ferc1 DBF + XBRL transform
#2025
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
zaneselvans
added
ferc1
Codecov ReportBase: 84.0% // Head: 84.2% // Increases project coverage by
Additional details and impacted files@@ Coverage Diff @@
## xbrl_integration #2025 +/- ##
==================================================
+ Coverage 84.0% 84.2% +0.2%
==================================================
Files 72 72
Lines 7935 7972 +37
==================================================
+ Hits 6668 6717 +49
+ Misses 1267 1255 -12
Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here. ☔ View full report at Codecov. |
This was
linked to
issues
Oct 26, 2022
A prototype reshaping table transformer, which processes the electric plant in service table. * Added a new mapping that links historical DBF row numbers to XBRL column labels. * Swapped old `plant_in_service` field and resource definitions for the new ones. This was a big change since the DB table is now long rather than wide form. * Still need to generalize and parametrize the reshaping transform, but that's going to require working with some additional tables to get more familiar with what kinds of things need to be done.
Progress on #2012 #2014 * Fixed a bug in how the DBF row numbers that need to be mapped are identified. Now it looks for any time the row_literal associated with a row number has changed from one year to the next, rather than selecting the first instance of each distinct combination of row_literal and row_number. * Also discovered that there's an obscure row_status field that differentiates between annual (A) and quarterly (Q) row literals, and is part of the f1_row_lit_tbl primary key, but it only shows up in association with the f1_schedules_list table. I integrated it but... maybe that table should just be excluded from the row mapping template? * Added some (janky) helper functions to pudl.transform.ferc1 to manage the generation of the row maps. This location is temporary. They should probably become methods of a Ferc1 abstract transformer class for reshaped tables, or maybe end up in a different module. Not sure how they'll end up getting used yet though. * Updated the dbf_to_xbrl.csv file to include all of the possible rows that could need mapping (4270 in total). * Removed the XBRL specific metadata fields from the dbf_to_xbrl.csv file, since they should (hopefully) be available programmatically from the metadata @zschira is extracting from the XBRL taxonomies, and can be joined to this table based on the xbrl_column_stem. * Updated the plant_in_service transform to use the new row map. Need to test on all of the years.
zaneselvans
commented
Nov 3, 2022
There was a problem hiding this comment.
Comments for @cmgosnell and @TrentonBush in anticipation of their reviews!
This was
unlinked from
issues
Nov 4, 2022
This was
unlinked from
issues
Nov 4, 2022
cmgosnell
requested changes
Nov 8, 2022
14 tasks
cmgosnell
reviewed
Nov 10, 2022
zaneselvans
changed the title
plant_in_service_ferc1 DBF + XBRLplant_in_service_ferc1 DBF + XBRL transform
cmgosnell
approved these changes
Nov 11, 2022
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
plant_in_service_ferc1table into the new TableTransformer system, and integrates both DBF + XBRL data.