Fides rebranding pass #165
Comments
7 tasks
|
Closed by #201 |
ThomasLaPiana
pushed a commit
that referenced
this issue
Aug 17, 2022
* When building mongo queries, have the projection (which specifies fields to return), use top-level fields only to avoid potential collisions when querying nested data. - Update "filter_data_categories", which filters retrieved data to only match the specified data categories on the Policy, to be able to filter nested results. Additionally, take in data_category_fields instead of a DatasetGraph for easier testing. - Update QueryConfig.update_value_map to be able to map nested fields to their masked version. Note: pandas.to_dict seems to return pandas.Timestamp instead of a Datetime. * Add tests confirming mongo collections can be discovered via a nested identity. - Rearrange some of the base mongo test collection, to create a new table with a nested identity. - Consolidate some large similar tests that were making lengthy access requests to verify the filtered data at the end into one test. * Add nested test that mongo queries are generated properly and that when building a traversal, an Edge can be created from the Root identity to a nested identity field. * Allow collections to be linked via nested fields by flattening the nested output from one collection into dot-separate input into a dependent collection (in to_dask_input_data) - Add mongo test collection that has a nested field reference a nested field in another collection. - Update FieldPath.retrieve_from to return None if a level is not found instead of a KeyError - * Update the fidesops postman collection to expand the request body that creates the mongo dataset to include the nested examples supported in our test mongo_db example db. * Add validation that object fields can't have declared data_categories - do this both at the FidesopsDatasetField level (which is used when parsing yaml into a Dataset) and internally at the ObjectField level in case the ObjectField is declared directly (usu in testing). * Extend test_combined_erasure_task to add some simple assertions for erasing nested fields - more comprehensive testing needed. - Improve testing capabilities of nested object erasures: - Update the "field" test util to be able to access a scalar field inside of an ObjectField and adjust all "field" usages to be able to accept a dynamic number of paths. - Fix bug in test fixture combined_mongo_postgresql_graph that nested field unexpectedly. - Add "generate_mongo_specific_records" that can create mongo records that can be deleted as part of the test * More thoroughly annotate the pieces of data_category_field_mapping, modifying the keys to be CollectionAddresses, too, so it is easier to tell what is expected from this resource. - remove other instances of else:/pass - Better type other variables in filter_data_categories * Fix datatype from previous draft. * Unrelated, discovered in testing by Catherine: If the request body doesn't include all the referenced datasets when previewing a query, a 500 was being thrown. Catch this error and raise a 400 instead, and explain how the request body should be formatted instead. * Refactor FieldPath.retrieve_from to use "pydash.objects.get" - this has array support too which will be useful in future iterations. The method is currently only returning the pydash call, but I think I should leave it there as a FieldPath method, because array queries may need more options added. The pydash method also changes the default behavior if no path was specified - now returns None instead of the original data. - Fix filter_data_categories to not expect that returned data contains all of the annotated fields in the dataset. * Require that unflatten_dict can only run on completely flattened dicts - (to avoid unexpected output) and add additional unflatten_dict edge cases to tests.
ThomasLaPiana
pushed a commit
that referenced
this issue
Aug 17, 2022
Allow data to be masked within arrays, as well as data within objects within arrays (Nested object erasure support was already introduced here #165). Worth noting, individual elements within arrays are targeted and masked, not the entire array. Support is for MongoDB only, but the most of this work is done in code, and can be used to help support other database implementations in the future.
ThomasLaPiana
pushed a commit
that referenced
this issue
Sep 26, 2022
* When building mongo queries, have the projection (which specifies fields to return), use top-level fields only to avoid potential collisions when querying nested data. - Update "filter_data_categories", which filters retrieved data to only match the specified data categories on the Policy, to be able to filter nested results. Additionally, take in data_category_fields instead of a DatasetGraph for easier testing. - Update QueryConfig.update_value_map to be able to map nested fields to their masked version. Note: pandas.to_dict seems to return pandas.Timestamp instead of a Datetime. * Add tests confirming mongo collections can be discovered via a nested identity. - Rearrange some of the base mongo test collection, to create a new table with a nested identity. - Consolidate some large similar tests that were making lengthy access requests to verify the filtered data at the end into one test. * Add nested test that mongo queries are generated properly and that when building a traversal, an Edge can be created from the Root identity to a nested identity field. * Allow collections to be linked via nested fields by flattening the nested output from one collection into dot-separate input into a dependent collection (in to_dask_input_data) - Add mongo test collection that has a nested field reference a nested field in another collection. - Update FieldPath.retrieve_from to return None if a level is not found instead of a KeyError - * Update the fidesops postman collection to expand the request body that creates the mongo dataset to include the nested examples supported in our test mongo_db example db. * Add validation that object fields can't have declared data_categories - do this both at the FidesopsDatasetField level (which is used when parsing yaml into a Dataset) and internally at the ObjectField level in case the ObjectField is declared directly (usu in testing). * Extend test_combined_erasure_task to add some simple assertions for erasing nested fields - more comprehensive testing needed. - Improve testing capabilities of nested object erasures: - Update the "field" test util to be able to access a scalar field inside of an ObjectField and adjust all "field" usages to be able to accept a dynamic number of paths. - Fix bug in test fixture combined_mongo_postgresql_graph that nested field unexpectedly. - Add "generate_mongo_specific_records" that can create mongo records that can be deleted as part of the test * More thoroughly annotate the pieces of data_category_field_mapping, modifying the keys to be CollectionAddresses, too, so it is easier to tell what is expected from this resource. - remove other instances of else:/pass - Better type other variables in filter_data_categories * Fix datatype from previous draft. * Unrelated, discovered in testing by Catherine: If the request body doesn't include all the referenced datasets when previewing a query, a 500 was being thrown. Catch this error and raise a 400 instead, and explain how the request body should be formatted instead. * Refactor FieldPath.retrieve_from to use "pydash.objects.get" - this has array support too which will be useful in future iterations. The method is currently only returning the pydash call, but I think I should leave it there as a FieldPath method, because array queries may need more options added. The pydash method also changes the default behavior if no path was specified - now returns None instead of the original data. - Fix filter_data_categories to not expect that returned data contains all of the annotated fields in the dataset. * Require that unflatten_dict can only run on completely flattened dicts - (to avoid unexpected output) and add additional unflatten_dict edge cases to tests.
ThomasLaPiana
pushed a commit
that referenced
this issue
Sep 26, 2022
Allow data to be masked within arrays, as well as data within objects within arrays (Nested object erasure support was already introduced here #165). Worth noting, individual elements within arrays are targeted and masked, not the entire array. Support is for MongoDB only, but the most of this work is done in code, and can be used to help support other database implementations in the future.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
This will be a branding pass on adding the logos to the
Currently waiting on the assets for these and will add to Fides when ready
The text was updated successfully, but these errors were encountered: