Update to WQX ref (ft, in, m) and TADA (mass/mass to mass/volume) ref #522

wokenny13 · 2024-09-03T18:40:06Z

Kevin has updated the WQX ref to reflect the changes. We decided to keep target units to inches, and fixing those 3 rows for the conversion factors.

I've added additional rows in TADA.PriorityCharConvertRef for common mass/mass to mass/volume conversions.

I ran a random testing TADA_DataRetrieval and found a #/100mL unit with no conversions to CFU/100mL, and added on this as a row to TADA.PriorityCharConvertRef in the csv.

Checking the conversion factors for reasonableness will be good for review, and if these additional rows make sense.

hillarymarler

Please add conversions to ug/L and mg/L for other mass/mass units. Once you've added those, I will approve this PR as it is working well, just needs a few additions. Nice work!

hillarymarler · 2024-09-04T17:35:25Z

inst/extdata/TADAPriorityCharConvertRef.csv

+ug/kg,8/29/2024 12:00,mg/L,1.00E-03,0
+#/100mL,8/29/2024 12:00,CFU/100ML,1.00E+00,0
+%,8/29/2024 12:00,ug/L,1.00E-03,0
+%,8/29/2024 12:00,mg/L,1.00E-06,0


In test data sets, I am still seeing a number of mass/mass units that are not converting because conversion factors to common mass/volume units are not included in TADAPriorityCharConvertRef.

For example: ug/g, mg/kg, ng/g, ng/kg

I think looking at the "Concentration Percentage" units in the WQX MeasureUnit.csv can help generate a list of the missing ones. It would be great to include conversion factors for ug/L and mg/L (as you already have for fg/kg, g/g, ug/kg, etc.)

- Updates reference tables and example data - Removes TADA.WQXTargetUnit and adds TADA.Target.ResultMeasure.MeasureUnitCode to required cols - Update ordering of columns and comparable data ID when transform = FALSE - Adds TADA.WQXUnitConversionCoefficient and TADA.WQXResultUnitConversion to column ordering (only relevant when transform = FALSE) - Update documentation

- Allows function to run without autoclean being run first - found that PPM was "Accepted" unit for "Depth, Bottom". Changed this to "Rejected" in WQX validation table. Reviewed many other depth result units as well as part of this update. - Updated example data

…-massvolume---kw' into fix-unit-conversions

cristinamullin · 2024-09-06T20:43:41Z

The unit conversions (output from TADA_ConvertResultUnits) and the TADA version of the WQX unit reference table look good now. However, the output df from TADA_CreateUnitRef still does not reflect the correct conversion factor (TADA.WQXUnitConversionFactor). See test2 output below.

I am sure this can be fixed (ran out of time today), but I was also thinking that it might not be necessary to include the conversion factors and coefficients in the table produced by TADA_CreateUnitRef, since that information is available to users if they want it via TADA_ConvertResultUnits when transform = FALSE. Either way, it looks like TADA_CreateUnitRef needs additional review/updating to finish off this issue.

`
test <- TADA_DataRetrieval(
startDate = "2020-01-01",
endDate = "null",
countrycode = "null",
countycode = "null",
huc = "null",
siteid = "null",
siteType = "null",
characteristicName = "Depth, bottom",
characteristicType = "null",
sampleMedia = "null",
statecode = "null",
organization = "null",
project = "null",
providers = "null",
applyautoclean = FALSE
)

test2 <- TADA_CreateUnitRef(test)

test3 <- TADA_ConvertResultUnits(test, transform = FALSE)

test4 <- TADA_ConvertResultUnits(test)
`

Note: I addressed a few other issues as part of the PR:

Updated TADA_FlagResultUnit so it can now run without autoclean being run first
I found that PPM was "Accepted" unit for "Depth, Bottom". Changed this to "Rejected" in WQX validation table. Reviewed many other depth result units as well & updated the WQX validation table for those as part of this update.
Updated reference tables and example data again
Removed TADA.WQXTargetUnit and added TADA.Target.ResultMeasure.MeasureUnitCode to required cols
Updated ordering of columns and comparable data ID when transform = FALSE
Adds TADA.WQXUnitConversionCoefficient and TADA.WQXResultUnitConversion to column ordering (only relevant when transform = FALSE)
Updated documentation

hillarymarler · 2024-09-09T14:14:32Z

@cristinamullin - the reason I had left the units in the TADA_CreateUnitRef output was for cases where a user might want to convert to a unit that we don't have in the unit refs. But if that scenario is not going to happen (or extremely unlikely), then I agree we can remove those columns as users can access the conversion info through TADA_ConvertResultUnits when transform = FALSE.

@wokenny13 - do you want to work on the updates needed for TADA_CreateUnitRef? Or would you like me to do that?

wokenny13 · 2024-09-09T14:20:10Z

test <- TADA_DataRetrieval(
startDate = "2020-01-01",
endDate = "null",
countrycode = "null",
countycode = "null",
huc = "null",
siteid = "null",
siteType = "null",
characteristicName = "Depth, bottom",
characteristicType = "null",
sampleMedia = "null",
statecode = "null",
organization = "null",
project = "null",
providers = "null",
applyautoclean = FALSE
)

@hillarymarler I can work on making these updates. I will let you know if I have any questions as I catch up on these comments.

Additional rows for other mass/mass to mass/volume Removed the two columns for TADA.WQXUnitConversionFactor and TADA.WQXUnitConversionCoefficient in the function TADA_CreateUnitRef

Changed order of columns to make it more clear on which columns are using the conversion factor

The test_that should re-run the TADA_RandomTestingData pull up to 10 times for testdat1. It will stop and determine if dimensions are the same. if testdat2 is able to run without error, that is if paired parameters in the testdat1 are in its dataframe.

updated comments and styler

wokenny13 · 2024-09-18T13:56:40Z

@hillarymarler @cristinamullin

Updated list of mass/mass to mass/volume TADAPriorityCharConvertRef.csv units to be converted to mg/L and UG/L.
Rearranged columns of output for TADA_CreateUnitRef. Keeping in the Conversion Factor and Coefficient but hope to make it clear which columns are being referenced with this rearrangement.
Included the updates to test-that CriteriaComparison.R in this pull request - see comments in Improve test for TADA_PairForCriteriaCalc #525

hillarymarler

@wokenny13 - the changes you made look good! I like the changes to the CriteriaComparison test.

@cristinamullin - did you want to take another look at any of this PR?

…me---kw

cristinamullin · 2024-10-09T19:58:05Z

These updates look great - thank you @wokenny13! I just merged in updates from develop. Once the checks complete, it can be merged in.

used break to break the loop. defined testdat as a global variable list that contains the two dataframes in the trycatch error function of testdat1 and testdat2

Update to WQX ref (ft, in, m) and TADA (mass/mass to mass/volume) ref

eec2b55

This was linked to issues Sep 3, 2024

ft, m, in unit conversions not working #517

Closed

Unit conversions - mass/mass vs mass/volume #505

Closed

wokenny13 requested a review from hillarymarler September 3, 2024 18:48

hillarymarler requested changes Sep 4, 2024

View reviewed changes

cristinamullin added 3 commits September 6, 2024 15:00

Merge remote-tracking branch 'origin/505-unit-conversions-massmass-vs…

cb21913

…-massvolume---kw' into fix-unit-conversions

wokenny13 added 6 commits September 9, 2024 12:06

TADA_CreateUnitRef and TADAPriorityCharConvertRef update

afde8ea

Additional rows for other mass/mass to mass/volume Removed the two columns for TADA.WQXUnitConversionFactor and TADA.WQXUnitConversionCoefficient in the function TADA_CreateUnitRef

TADA_CreateUnitRef and TADAPriorityCharConvertRef update

0ffd91c

TADAPriorityCharConvertRef

fae664f

Update UnitConversions.R

9e04979

Update UnitConversions.R

4b31207

Changed order of columns to make it more clear on which columns are using the conversion factor

Update test-CriteriaComparison.R

01c40fd

The test_that should re-run the TADA_RandomTestingData pull up to 10 times for testdat1. It will stop and determine if dimensions are the same. if testdat2 is able to run without error, that is if paired parameters in the testdat1 are in its dataframe.

wokenny13 linked an issue Sep 17, 2024 that may be closed by this pull request

Improve test for TADA_PairForCriteriaCalc #525

Closed

Update test-CriteriaComparison.R

7d032b0

updated comments and styler

wokenny13 requested a review from hillarymarler September 18, 2024 13:49

hillarymarler mentioned this pull request Sep 18, 2024

Improve test for TADA_PairForCriteriaCalc #525

Closed

hillarymarler approved these changes Oct 4, 2024

View reviewed changes

Merge branch 'develop' into 505-unit-conversions-massmass-vs-massvolu…

1752cd4

…me---kw

Update test-CriteriaComparison.R

5b155a3

used break to break the loop. defined testdat as a global variable list that contains the two dataframes in the trycatch error function of testdat1 and testdat2

cristinamullin merged commit 752b3ed into develop Oct 15, 2024
7 checks passed

cristinamullin deleted the 505-unit-conversions-massmass-vs-massvolume---kw branch October 15, 2024 18:55

cristinamullin restored the 505-unit-conversions-massmass-vs-massvolume---kw branch November 8, 2024 19:11

cristinamullin deleted the 505-unit-conversions-massmass-vs-massvolume---kw branch November 8, 2024 19:11

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update to WQX ref (ft, in, m) and TADA (mass/mass to mass/volume) ref #522

Update to WQX ref (ft, in, m) and TADA (mass/mass to mass/volume) ref #522

wokenny13 commented Sep 3, 2024

hillarymarler left a comment •

edited

Loading

hillarymarler Sep 4, 2024

cristinamullin commented Sep 6, 2024 •

edited

Loading

hillarymarler commented Sep 9, 2024

wokenny13 commented Sep 9, 2024 •

edited

Loading

wokenny13 commented Sep 18, 2024

hillarymarler left a comment

cristinamullin commented Oct 9, 2024

Update to WQX ref (ft, in, m) and TADA (mass/mass to mass/volume) ref #522

Update to WQX ref (ft, in, m) and TADA (mass/mass to mass/volume) ref #522

Conversation

wokenny13 commented Sep 3, 2024

hillarymarler left a comment • edited Loading

Choose a reason for hiding this comment

hillarymarler Sep 4, 2024

Choose a reason for hiding this comment

cristinamullin commented Sep 6, 2024 • edited Loading

hillarymarler commented Sep 9, 2024

wokenny13 commented Sep 9, 2024 • edited Loading

wokenny13 commented Sep 18, 2024

hillarymarler left a comment

Choose a reason for hiding this comment

cristinamullin commented Oct 9, 2024

hillarymarler left a comment •

edited

Loading

cristinamullin commented Sep 6, 2024 •

edited

Loading

wokenny13 commented Sep 9, 2024 •

edited

Loading