TBOX-332 Address Validation #156

ErinCompaan · 2023-07-11T12:46:22Z

↪️ Pull Request

This adds functionality to use Google Address Validation API with Tamr. Tests are incomplete, but putting this up for early feedback since the plan is to use this with a customer within a couple weeks.

Some notes:

The set-up is very similar to the existing translation enrichment -- the main difference is that the address validation doesn't store sets of similar addresses which can be formatted by rules/cleaning to the same result. The translation dictionary stores sets of phrases like this:

tamr-toolbox/tamr_toolbox/enrichment/dictionary.py

Line 35 in 721fd5a

original_phrases: Set[str] = field(default_factory=lambda: set())

I didn't see much value to adding that for addresses, but I'm open to other thoughts.
There is a class here (!), but it was okay for translation apparently, so I'm hoping it's okay for validation.
I used the scripts I added in the examples folder to set up a SM project/ validation mapping on the toolbox test instance. You can see what it looks like there / how I used transformations to pull in the validation. It would be nice to automate set-up of the additional validation columns in the unified dataset, as well as the transformations to populate them. I would welcome suggestions on how to do that / where it belongs (e.g. just an example script, or more enrichment module functionality).

✔️ PR Todo

Add documentation
Add examples
Ensure you are following the practices in the contributor guide and coding_standards
Added/updated testing for this change (ensure you are testing any changes/new code!)
Update title to link to TBOX issue (i.e. start pull request title with TBOX-xx)
Approval from first code-reviewer
Approval from second code-reviewer (only required if a second reviewer has left comments)
Pass all checks

zbpvarun-tamr · 2023-07-11T13:20:16Z

I see that a couple of checks are failing already. I'm not sure why the enforce test coverage shows your test coverage as 0% at the moment... Will look into it in more detail. For the core-dependencies, pandas is identified as an optional dependency. I don't remember the exact reasoning for why this is still the case... Maybe worth revisiting. You can see this script as an example of how to import pandas.

ErinCompaan · 2023-07-13T15:00:22Z

This is ready for review now -- test coverage for the new code is complete:

pytest --cov=tamr_toolbox/enrichment --cov-report=term-missing tests/enrichment
---------- coverage: platform darwin, python 3.9.13-final-0 ----------
Name                                                            Stmts   Miss  Cover   Missing
---------------------------------------------------------------------------------------------
tamr_toolbox/enrichment/__init__.py                                 7      0   100%
tamr_toolbox/enrichment/address_mapping.py                        121      0   100%
tamr_toolbox/enrichment/address_validation.py                      54      0   100%
tamr_toolbox/enrichment/api_client/__init__.py                      3      0   100%
tamr_toolbox/enrichment/api_client/google_address_validate.py      43      0   100%
tamr_toolbox/enrichment/api_client/google_translate.py             63      0   100%
tamr_toolbox/enrichment/dictionary.py                             123      7    94%   281-285, 380-385
tamr_toolbox/enrichment/enrichment_utils.py                        26      0   100%
tamr_toolbox/enrichment/translate.py                               66      4    94%   154-156, 197
---------------------------------------------------------------------------------------------
TOTAL                                                             506     11    98%

optional_requirements.txt

zbpvarun-tamr · 2023-07-14T19:55:15Z

tamr_toolbox/enrichment/address_validation.py

+        if joined_addr not in addr_mapping.keys():
+            addr_to_validate.append(joined_addr)
+            count_new_addr += 1
+        elif addr_mapping[joined_addr].expiration < str(datetime.now() + expiration_date_buffer):


I was confused by this but I think I get it now... Please confirm if this is the case - basically if the validation expires then it will get deleted. This elif clause catches those that are soon to expire to also be validated again. Is this correct?

How do you differentiate between those records that have been validated but have expired versus those that have not been validated before? Does it matter?

Yes, this is all to make it possible to comply with the google terms of service, which prohibit caching most results for more than 30 days. Regarding the first comment, validation won't be deleted if it expires currently, but if it's a validation that's still in use (e.g. it's an address that is still in the list of addresses of interest), it will be validated again.

I don't differentiate between the two cases currently -- I'm not sure there's a reason to do so.

zbpvarun-tamr · 2023-07-14T20:12:00Z

tamr_toolbox/enrichment/api_client/google_address_validate.py

+
+
+def get_maps_client() -> "googlemaps.Client":
+    """Get GoogleMaps client using the environment variable 'GOOGLEMAPS_API_KEY'


Does this show up in the docs as the name that is required to be set when running address validation?

zbpvarun-tamr · 2023-07-14T20:30:21Z

tests/enrichment/address_validation_mapping.json

@@ -0,0 +1,5 @@
+{"input_address": "66 Church St Cambridge Mass", "validated_formatted_address": "66 Church Street, Cambridge, MA 02138-3733, USA", "expiration": "2023-08-09 11:46:40.408657", "region_code": "US", "postal_code": "02138-3733", "admin_area": "MA", "locality": "Cambridge", "address_lines": ["66 Church St"], "usps_first_address_line": "66 CHURCH ST", "usps_city_state_zip_line": "CAMBRIDGE MA 02138-3733", "usps_city": "CAMBRIDGE", "usps_state": "MA", "usps_zip_code": "02138-3733", "latitude": 42.3739503, "longitude": -71.1211445, "place_id": "ChIJNR2ZIGh344kRNQAj-dh6d00", "input_granularity": "PREMISE", "validation_granularity": "PREMISE", "geocode_granularity": "PREMISE", "has_inferred": true, "has_unconfirmed": false, "has_replaced": false, "address_complete": false}


Do you need these files as part of the PR and testing? They will be added to the package if included in the present form. I know the addresses are not that critical but still, I am wondering if we can test without adding these files to the PR. I don't see equivalent files for translation either...

I may have missed it but where do you use these 2 json files?

These are used to test the from_json functions and from_json error handling for module. The translation doesn't have them because it has data in the test file, which it saves to json and then loads, and it doesn't test the same error handling. We could get rid of them by doing a save + load in the test file and mocking the bad data somehow, but the same data would still be stored somewhere.

zbpvarun-tamr · 2023-07-14T20:34:58Z

examples/scripts/enrichment/validate_addresses_from_file.py

+    path_to_csv_to_validate: str,
+    path_to_validated_csv: str,
+) -> None:
+    """Validate data located on disk and save results to disk.


This example is a good place to specify how the API key needs to be stored.

Also important to specify since your example allows for a config file but your get_maps_client function assumes an environment variable. Worth converting to a config variable?

Yes, perhaps that would be better. I can see about doing that.

zbpvarun-tamr · 2023-07-14T20:54:07Z

tests/enrichment/test_address_validation.py

+from tamr_toolbox.enrichment.api_client.google_address_validate import get_maps_client
+from tamr_toolbox.utils.testing import mock_api
+
+ADDR_VAL_MAPPING_0 = tamr_toolbox.enrichment.address_mapping.AddressValidationMapping(


I would be much happier if we could find a way to test this without putting all these details into the package... But if that does not exist, the tests themselves look good...

What is your concern with storing the data? Depending on the issue, there may be a way around it.

zbpvarun-tamr

Looks good to me. Depending on your conversation with Ravi, if you need to make any changes, I can take another look. But otherwise, feel free to merge when ready.

ErinCompaan added 10 commits July 6, 2023 17:31

initial work on address validation

0cde1e0

progress on API setup

dd3a7e1

clean up address validation file

b9686d3

start on example script

7934909

working example scripts

8cbc051

typo fix

27a0954

add some tests

d5f562d

add more tests

5404c03

fix linting

d7e9d9a

using typing extensions

f8ac159

ErinCompaan added 10 commits July 11, 2023 09:27

make pandas optional

c03d39e

fix path in docs build

a9d95f6

update pandas replace for 3.7 syntax

76f729d

initial tests for address mapping

c6185a2

tests for address_mapping

6fca00d

add docs

cd7d2b0

remove personal file paths

8f421e6

tests for address validation funcs

53da618

set env for test

0cdd3c0

complete tests

48577c6

ErinCompaan marked this pull request as ready for review July 13, 2023 15:00

ErinCompaan requested review from skalish, jun-dao, wpmartin, zbpvarun-tamr, hduckduck and RJSanders as code owners July 13, 2023 15:00

ErinCompaan requested a review from a user July 13, 2023 15:00

ErinCompaan requested a review from Adam-Tamr as a code owner July 13, 2023 15:00

zbpvarun-tamr reviewed Jul 14, 2023

View reviewed changes

ErinCompaan added 2 commits July 17, 2023 08:50

loosen dependency req

20628b7

use config file for API key

9e3380f

zbpvarun-tamr approved these changes Jul 17, 2023

View reviewed changes

ErinCompaan merged commit 15b19b7 into Datatamer:develop Jul 17, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

TBOX-332 Address Validation #156

TBOX-332 Address Validation #156

ErinCompaan commented Jul 11, 2023 •

edited by zbpvarun-tamr

Loading

zbpvarun-tamr commented Jul 11, 2023

ErinCompaan commented Jul 13, 2023

zbpvarun-tamr Jul 14, 2023

zbpvarun-tamr Jul 14, 2023

ErinCompaan Jul 17, 2023 •

edited

Loading

zbpvarun-tamr Jul 14, 2023

zbpvarun-tamr Jul 14, 2023

zbpvarun-tamr Jul 14, 2023

ErinCompaan Jul 17, 2023

zbpvarun-tamr Jul 14, 2023

zbpvarun-tamr Jul 14, 2023

ErinCompaan Jul 17, 2023

zbpvarun-tamr Jul 14, 2023

ErinCompaan Jul 17, 2023

zbpvarun-tamr left a comment



		def get_maps_client() -> "googlemaps.Client":
		"""Get GoogleMaps client using the environment variable 'GOOGLEMAPS_API_KEY'

		@@ -0,0 +1,5 @@
		{"input_address": "66 Church St Cambridge Mass", "validated_formatted_address": "66 Church Street, Cambridge, MA 02138-3733, USA", "expiration": "2023-08-09 11:46:40.408657", "region_code": "US", "postal_code": "02138-3733", "admin_area": "MA", "locality": "Cambridge", "address_lines": ["66 Church St"], "usps_first_address_line": "66 CHURCH ST", "usps_city_state_zip_line": "CAMBRIDGE MA 02138-3733", "usps_city": "CAMBRIDGE", "usps_state": "MA", "usps_zip_code": "02138-3733", "latitude": 42.3739503, "longitude": -71.1211445, "place_id": "ChIJNR2ZIGh344kRNQAj-dh6d00", "input_granularity": "PREMISE", "validation_granularity": "PREMISE", "geocode_granularity": "PREMISE", "has_inferred": true, "has_unconfirmed": false, "has_replaced": false, "address_complete": false}

TBOX-332 Address Validation #156

TBOX-332 Address Validation #156

Conversation

ErinCompaan commented Jul 11, 2023 • edited by zbpvarun-tamr Loading

↪️ Pull Request

✔️ PR Todo

zbpvarun-tamr commented Jul 11, 2023

ErinCompaan commented Jul 13, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ErinCompaan Jul 17, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

zbpvarun-tamr left a comment

Choose a reason for hiding this comment

ErinCompaan commented Jul 11, 2023 •

edited by zbpvarun-tamr

Loading

ErinCompaan Jul 17, 2023 •

edited

Loading