Skip to content

Commit

Permalink
Fix percentage inversion, update tests (#1848)
Browse files Browse the repository at this point in the history 
Looking through the transformations, I noticed that we were subtracting
a percentage that is usually between 0-100 from 1 instead of 100, and so
were endind up with some surprising results. Confirmed with lucasmbrown-usds
  • Loading branch information
@mattbowen-usds
mattbowen-usds committed Sep 15, 2022
1 parent 7d41acd commit b55e667
Show file tree
Hide file tree
Showing 4 changed files with 32 additions and 32 deletions.
2 changes: 1 addition & 1 deletion data/data-pipeline/data_pipeline/etl/sources/nlcd_nature_deprived/etl.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -69,7 +69,7 @@ def transform(self) -> None:
df_ncld[self.TRACT_ACRES_FIELD_NAME] >= self.TRACT_ACRES_LOWER_BOUND
)
df_ncld[self.TRACT_PERCENT_NON_NATURAL_FIELD_NAME] = (
1 - df_ncld[self.PERCENT_NATURAL_FIELD_NAME]
100 - df_ncld[self.PERCENT_NATURAL_FIELD_NAME]
)

# Assign the final df to the class' output_df for the load method with rename
Expand Down
2 changes: 1 addition & 1 deletion data/data-pipeline/data_pipeline/tests/sources/historic_redlining/test_etl.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,7 @@ def test_extract_produces_valid_data(self, snapshot, mock_etl, mock_paths):

def test_load_base(self, snapshot, mock_etl, mock_paths):
"""Test load method.
We need to run transform here for real to add
We need to run transform here for real to add
the dynamic cols to keep
"""
# setup - input variables
Expand Down
30 changes: 15 additions & 15 deletions data/data-pipeline/data_pipeline/tests/sources/nlcd_nature_deprived/data/output.csv
View file
Original file line number Diff line number Diff line change
@@ -1,16 +1,16 @@
GEOID10_TRACT,Does the tract have at least 35 acres in it?,Share of the tract's land area that is covered by impervious surface as a percent,Share of the tract's land area that is covered by impervious surface or cropland as a percent,Share of the tract's land area that is covered by cropland as a percent
06027000800,True,0.2995360000,-98.6814000000,0.0190559000
06069000802,True,0.1123740000,-98.2782000000,0.6093810000
06061021322,True,2.9274100000,-56.5222000000,39.5504000000
15001021010,True,2.9274100000,-56.5222000000,39.5504000000
15001021101,True,2.9274100000,-56.5222000000,39.5504000000
15007040603,True,2.9274100000,-56.5222000000,39.5504000000
15007040700,True,2.9274100000,-56.5222000000,39.5504000000
15009030100,True,2.9274100000,-56.5222000000,39.5504000000
15009030201,True,2.9274100000,-56.5222000000,39.5504000000
15001021402,True,2.9274100000,-56.5222000000,39.5504000000
15001021800,True,2.9274100000,-56.5222000000,39.5504000000
15009030402,True,2.9274100000,-56.5222000000,39.5504000000
15009030800,True,2.9274100000,-56.5222000000,39.5504000000
15003010201,True,2.9274100000,-56.5222000000,39.5504000000
15007040604,True,2.9274100000,-56.5222000000,39.5504000000
06027000800,True,0.2995360000,0.3186000000,0.0190559000
06069000802,True,0.1123740000,0.7218000000,0.6093810000
06061021322,True,2.9274100000,42.4778000000,39.5504000000
15001021010,True,2.9274100000,42.4778000000,39.5504000000
15001021101,True,2.9274100000,42.4778000000,39.5504000000
15007040603,True,2.9274100000,42.4778000000,39.5504000000
15007040700,True,2.9274100000,42.4778000000,39.5504000000
15009030100,True,2.9274100000,42.4778000000,39.5504000000
15009030201,True,2.9274100000,42.4778000000,39.5504000000
15001021402,True,2.9274100000,42.4778000000,39.5504000000
15001021800,True,2.9274100000,42.4778000000,39.5504000000
15009030402,True,2.9274100000,42.4778000000,39.5504000000
15009030800,True,2.9274100000,42.4778000000,39.5504000000
15003010201,True,2.9274100000,42.4778000000,39.5504000000
15007040604,True,2.9274100000,42.4778000000,39.5504000000
30 changes: 15 additions & 15 deletions data/data-pipeline/data_pipeline/tests/sources/nlcd_nature_deprived/data/transform.csv
View file
Original file line number Diff line number Diff line change
@@ -1,16 +1,16 @@
GEOID10,SF,CF,P200_PFS,CA_LT20,TractAcres,AcresCrops,Share of the tract's land area that is covered by cropland as a percent,Share of the tract's land area that is covered by impervious surface as a percent,PctNatural,PctNat90,ImpOrCrop,LowInAndEd,NatureDep,GEOID10_TRACT,Does the tract have at least 35 acres in it?,Share of the tract's land area that is covered by impervious surface or cropland as a percent
6027000800,California,Inyo County,0.6700000000,1,4374150.0000000000,833.5350000000,0.0190559000,0.2995360000,99.6814000000,0,0,1,0,06027000800,True,-98.6814000000
6069000802,California,San Benito County,0.3700000000,1,738261.0000000000,4498.8200000000,0.6093810000,0.1123740000,99.2782000000,0,0,0,0,06069000802,True,-98.2782000000
6061021322,California,Placer County,0.2500000000,1,63993.3000000000,25309.6000000000,39.5504000000,2.9274100000,57.5222000000,0,1,0,0,06061021322,True,-56.5222000000
15001021010,Hawaii,Hawaii County,0.3700000000,1,63993.3000000000,25309.6000000000,39.5504000000,2.9274100000,57.5222000000,0,1,0,0,15001021010,True,-56.5222000000
15001021101,Hawaii,Hawaii County,0.3700000000,1,63993.3000000000,25309.6000000000,39.5504000000,2.9274100000,57.5222000000,0,1,0,0,15001021101,True,-56.5222000000
15007040603,Hawaii,Hawaii County,0.3700000000,1,63993.3000000000,25309.6000000000,39.5504000000,2.9274100000,57.5222000000,0,1,0,0,15007040603,True,-56.5222000000
15007040700,Hawaii,Hawaii County,0.3700000000,1,63993.3000000000,25309.6000000000,39.5504000000,2.9274100000,57.5222000000,0,1,0,0,15007040700,True,-56.5222000000
15009030100,Hawaii,Hawaii County,0.3700000000,1,63993.3000000000,25309.6000000000,39.5504000000,2.9274100000,57.5222000000,0,1,0,0,15009030100,True,-56.5222000000
15009030201,Hawaii,Hawaii County,0.3700000000,1,63993.3000000000,25309.6000000000,39.5504000000,2.9274100000,57.5222000000,0,1,0,0,15009030201,True,-56.5222000000
15001021402,Hawaii,Hawaii County,0.3700000000,1,63993.3000000000,25309.6000000000,39.5504000000,2.9274100000,57.5222000000,0,1,0,0,15001021402,True,-56.5222000000
15001021800,Hawaii,Hawaii County,0.3700000000,1,63993.3000000000,25309.6000000000,39.5504000000,2.9274100000,57.5222000000,0,1,0,0,15001021800,True,-56.5222000000
15009030402,Hawaii,Hawaii County,0.3700000000,1,63993.3000000000,25309.6000000000,39.5504000000,2.9274100000,57.5222000000,0,1,0,0,15009030402,True,-56.5222000000
15009030800,Hawaii,Hawaii County,0.3700000000,1,63993.3000000000,25309.6000000000,39.5504000000,2.9274100000,57.5222000000,0,1,0,0,15009030800,True,-56.5222000000
15003010201,Hawaii,Hawaii County,0.3700000000,1,63993.3000000000,25309.6000000000,39.5504000000,2.9274100000,57.5222000000,0,1,0,0,15003010201,True,-56.5222000000
15007040604,Hawaii,Hawaii County,0.3700000000,1,63993.3000000000,25309.6000000000,39.5504000000,2.9274100000,57.5222000000,0,1,0,0,15007040604,True,-56.5222000000
6027000800,California,Inyo County,0.6700000000,1,4374150.0000000000,833.5350000000,0.0190559000,0.2995360000,99.6814000000,0,0,1,0,06027000800,True,0.3186000000
6069000802,California,San Benito County,0.3700000000,1,738261.0000000000,4498.8200000000,0.6093810000,0.1123740000,99.2782000000,0,0,0,0,06069000802,True,0.7218000000
6061021322,California,Placer County,0.2500000000,1,63993.3000000000,25309.6000000000,39.5504000000,2.9274100000,57.5222000000,0,1,0,0,06061021322,True,42.4778000000
15001021010,Hawaii,Hawaii County,0.3700000000,1,63993.3000000000,25309.6000000000,39.5504000000,2.9274100000,57.5222000000,0,1,0,0,15001021010,True,42.4778000000
15001021101,Hawaii,Hawaii County,0.3700000000,1,63993.3000000000,25309.6000000000,39.5504000000,2.9274100000,57.5222000000,0,1,0,0,15001021101,True,42.4778000000
15007040603,Hawaii,Hawaii County,0.3700000000,1,63993.3000000000,25309.6000000000,39.5504000000,2.9274100000,57.5222000000,0,1,0,0,15007040603,True,42.4778000000
15007040700,Hawaii,Hawaii County,0.3700000000,1,63993.3000000000,25309.6000000000,39.5504000000,2.9274100000,57.5222000000,0,1,0,0,15007040700,True,42.4778000000
15009030100,Hawaii,Hawaii County,0.3700000000,1,63993.3000000000,25309.6000000000,39.5504000000,2.9274100000,57.5222000000,0,1,0,0,15009030100,True,42.4778000000
15009030201,Hawaii,Hawaii County,0.3700000000,1,63993.3000000000,25309.6000000000,39.5504000000,2.9274100000,57.5222000000,0,1,0,0,15009030201,True,42.4778000000
15001021402,Hawaii,Hawaii County,0.3700000000,1,63993.3000000000,25309.6000000000,39.5504000000,2.9274100000,57.5222000000,0,1,0,0,15001021402,True,42.4778000000
15001021800,Hawaii,Hawaii County,0.3700000000,1,63993.3000000000,25309.6000000000,39.5504000000,2.9274100000,57.5222000000,0,1,0,0,15001021800,True,42.4778000000
15009030402,Hawaii,Hawaii County,0.3700000000,1,63993.3000000000,25309.6000000000,39.5504000000,2.9274100000,57.5222000000,0,1,0,0,15009030402,True,42.4778000000
15009030800,Hawaii,Hawaii County,0.3700000000,1,63993.3000000000,25309.6000000000,39.5504000000,2.9274100000,57.5222000000,0,1,0,0,15009030800,True,42.4778000000
15003010201,Hawaii,Hawaii County,0.3700000000,1,63993.3000000000,25309.6000000000,39.5504000000,2.9274100000,57.5222000000,0,1,0,0,15003010201,True,42.4778000000
15007040604,Hawaii,Hawaii County,0.3700000000,1,63993.3000000000,25309.6000000000,39.5504000000,2.9274100000,57.5222000000,0,1,0,0,15007040604,True,42.4778000000

0 comments on commit b55e667

Please sign in to comment.