Remove stack trace from evaluation error #128
Assignees
Comments
ThomasLaPiana
pushed a commit
that referenced
this issue
Aug 17, 2022
* When building mongo queries, have the projection (which specifies fields to return), use top-level fields only to avoid potential collisions when querying nested data. - Update "filter_data_categories", which filters retrieved data to only match the specified data categories on the Policy, to be able to filter nested results. Additionally, take in data_category_fields instead of a DatasetGraph for easier testing. - Update QueryConfig.update_value_map to be able to map nested fields to their masked version. Note: pandas.to_dict seems to return pandas.Timestamp instead of a Datetime. * Add tests confirming mongo collections can be discovered via a nested identity. - Rearrange some of the base mongo test collection, to create a new table with a nested identity. - Consolidate some large similar tests that were making lengthy access requests to verify the filtered data at the end into one test. * Add nested test that mongo queries are generated properly and that when building a traversal, an Edge can be created from the Root identity to a nested identity field. * Allow collections to be linked via nested fields by flattening the nested output from one collection into dot-separate input into a dependent collection (in to_dask_input_data) - Add mongo test collection that has a nested field reference a nested field in another collection. - Update FieldPath.retrieve_from to return None if a level is not found instead of a KeyError - * Update the fidesops postman collection to expand the request body that creates the mongo dataset to include the nested examples supported in our test mongo_db example db. * Add validation that object fields can't have declared data_categories - do this both at the FidesopsDatasetField level (which is used when parsing yaml into a Dataset) and internally at the ObjectField level in case the ObjectField is declared directly (usu in testing). * Extend test_combined_erasure_task to add some simple assertions for erasing nested fields - more comprehensive testing needed. - Improve testing capabilities of nested object erasures: - Update the "field" test util to be able to access a scalar field inside of an ObjectField and adjust all "field" usages to be able to accept a dynamic number of paths. - Fix bug in test fixture combined_mongo_postgresql_graph that nested field unexpectedly. - Add "generate_mongo_specific_records" that can create mongo records that can be deleted as part of the test * More thoroughly annotate the pieces of data_category_field_mapping, modifying the keys to be CollectionAddresses, too, so it is easier to tell what is expected from this resource. - remove other instances of else:/pass - Better type other variables in filter_data_categories * Fix datatype from previous draft. * Unrelated, discovered in testing by Catherine: If the request body doesn't include all the referenced datasets when previewing a query, a 500 was being thrown. Catch this error and raise a 400 instead, and explain how the request body should be formatted instead. * Refactor FieldPath.retrieve_from to use "pydash.objects.get" - this has array support too which will be useful in future iterations. The method is currently only returning the pydash call, but I think I should leave it there as a FieldPath method, because array queries may need more options added. The pydash method also changes the default behavior if no path was specified - now returns None instead of the original data. - Fix filter_data_categories to not expect that returned data contains all of the annotated fields in the dataset. * Require that unflatten_dict can only run on completely flattened dicts - (to avoid unexpected output) and add additional unflatten_dict edge cases to tests.
ThomasLaPiana
pushed a commit
that referenced
this issue
Sep 26, 2022
* When building mongo queries, have the projection (which specifies fields to return), use top-level fields only to avoid potential collisions when querying nested data. - Update "filter_data_categories", which filters retrieved data to only match the specified data categories on the Policy, to be able to filter nested results. Additionally, take in data_category_fields instead of a DatasetGraph for easier testing. - Update QueryConfig.update_value_map to be able to map nested fields to their masked version. Note: pandas.to_dict seems to return pandas.Timestamp instead of a Datetime. * Add tests confirming mongo collections can be discovered via a nested identity. - Rearrange some of the base mongo test collection, to create a new table with a nested identity. - Consolidate some large similar tests that were making lengthy access requests to verify the filtered data at the end into one test. * Add nested test that mongo queries are generated properly and that when building a traversal, an Edge can be created from the Root identity to a nested identity field. * Allow collections to be linked via nested fields by flattening the nested output from one collection into dot-separate input into a dependent collection (in to_dask_input_data) - Add mongo test collection that has a nested field reference a nested field in another collection. - Update FieldPath.retrieve_from to return None if a level is not found instead of a KeyError - * Update the fidesops postman collection to expand the request body that creates the mongo dataset to include the nested examples supported in our test mongo_db example db. * Add validation that object fields can't have declared data_categories - do this both at the FidesopsDatasetField level (which is used when parsing yaml into a Dataset) and internally at the ObjectField level in case the ObjectField is declared directly (usu in testing). * Extend test_combined_erasure_task to add some simple assertions for erasing nested fields - more comprehensive testing needed. - Improve testing capabilities of nested object erasures: - Update the "field" test util to be able to access a scalar field inside of an ObjectField and adjust all "field" usages to be able to accept a dynamic number of paths. - Fix bug in test fixture combined_mongo_postgresql_graph that nested field unexpectedly. - Add "generate_mongo_specific_records" that can create mongo records that can be deleted as part of the test * More thoroughly annotate the pieces of data_category_field_mapping, modifying the keys to be CollectionAddresses, too, so it is easier to tell what is expected from this resource. - remove other instances of else:/pass - Better type other variables in filter_data_categories * Fix datatype from previous draft. * Unrelated, discovered in testing by Catherine: If the request body doesn't include all the referenced datasets when previewing a query, a 500 was being thrown. Catch this error and raise a 400 instead, and explain how the request body should be formatted instead. * Refactor FieldPath.retrieve_from to use "pydash.objects.get" - this has array support too which will be useful in future iterations. The method is currently only returning the pydash call, but I think I should leave it there as a FieldPath method, because array queries may need more options added. The pydash method also changes the default behavior if no path was specified - now returns None instead of the original data. - Fix filter_data_categories to not expect that returned data contains all of the annotated fields in the dataset. * Require that unflatten_dict can only run on completely flattened dicts - (to avoid unexpected output) and add additional unflatten_dict edge cases to tests.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Should be a cleaner message.
Remove stack traces from FidesValidationErrors as well as EvaluationErrors
The text was updated successfully, but these errors were encountered: