Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Filter out invalid CSV values so they never get ingested #424

Open
mjordan opened this issue May 25, 2022 · 3 comments
Open

Filter out invalid CSV values so they never get ingested #424

mjordan opened this issue May 25, 2022 · 3 comments
Assignees
Labels
enhancement New feature or request

Comments

@mjordan
Copy link
Owner

mjordan commented May 25, 2022

Now that #417 is done, any incoming values that are invalid ETDF, link, geocordinates, etc. should be filtered out. Related issue #387.

@mjordan mjordan self-assigned this May 25, 2022
mjordan added a commit that referenced this issue May 26, 2022
@mjordan
Copy link
Owner Author

mjordan commented May 26, 2022

Work started in f45d49e on SimpleField, GeolocationField, and LinkField classes, with tests in field_tests_values.py. Need to figure out how remove_invalid_values() will work with 1-cardinality creates/updates.

Validating entity reference and typed relation values will likely need a hot Drupal, so we'll need to figure out how to configure host, username, and password in tests.

@mjordan mjordan added the enhancement New feature or request label Aug 10, 2022
@mjordan
Copy link
Owner Author

mjordan commented May 8, 2023

@mjordan
Copy link
Owner Author

mjordan commented May 10, 2023

Reviewing this feature again. For every field type, we want to, in this order:

  1. remove invalid values. We never alter values (e.g. truncate text longer than allowed) except for title; instead, we do not populate the field (or value instance in multivalued fields) and just log that we haven't populated the field.
  2. dedupe values

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant