Skip Indexing Custom Privacy Request Field Arrays #5127
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…the value for search if the value is a list. This avoids us potentially indexing values that are too large and with the way we are indexing, this is not going to be useful for search, regardless.
|
The latest updates on your projects. Learn more about Vercel for Git ↗︎ 1 Skipped Deployment
|
Passing run #9141 ↗︎
Details:
Review all test suite changes for PR #5127 ↗︎ |
||||||||||||||||||||
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## main #5127 +/- ##
=======================================
Coverage 86.56% 86.56%
=======================================
Files 357 357
Lines 22349 22349
Branches 2955 2954 -1
=======================================
Hits 19347 19347
Misses 2480 2480
Partials 522 522 ☔ View full report in Codecov by Sentry. |
pattisdr
commented
Jul 25, 2024
galvana
approved these changes
Jul 25, 2024
There was a problem hiding this comment.
Looks good Dawn! Thank you for making this change so quickly. Nice test coverage as usual and the sample requests were a nice touch to help with manual testing. I submitted both requests and verified the hashed_value is no longer set for list values.
fides=# select privacy_request_id, field_name, hashed_value, left(encrypted_value, 16) as encrypted_value from custom_privacy_request_field;
privacy_request_id | field_name | hashed_value | encrypted_value
------------------------------------------+--------------+--------------+------------------
pri_71b295b7-cbf6-4621-a855-162fcf07f345 | internal_ids | | 780gDzPwNE/KKhU0
pri_71b295b7-cbf6-4621-a855-162fcf07f345 | evidence_box | | UbSq53Bt1EB4hIV+
pri_3f8a3c78-a9a0-4fab-8f9a-ebc0e6321989 | internal_ids | | 7hF5S/N7hcJMgZo9
pri_3f8a3c78-a9a0-4fab-8f9a-ebc0e6321989 | evidence_box | | r3pw7c4AgQ5eGG/n
(4 rows)
|
Thank you @galvana this was quick in part because your CustomPrivacyRequestField.hashed_value was already nullable ⭐ |
Passing run #9147 ↗︎
Details:
Review all test suite changes for PR #5127 ↗︎ |
||||||||||||||||||||
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Closes #PROD-2359
Description Of Changes
Sending in a large list value for a single custom privacy request field results in an extremely lengthy response time that can time out due to 1) hashing the list and 2) trying to then index that list for search.
We're going to skip hashing/index custom privacy request values for search if the supplied value is an array. This avoids us potentially hashing/indexing too large of a value. Further, indexing arrays don't really serve a purpose given that we're indexing the entire array, there's no full text search or anything of that nature enabled.
Code Changes
Steps to Confirm
This request should be extremely quick locally. Previously, this would hang for several minutes and then fail with a (psycopg2.errors.ProgramLimitExceeded) index row requires 51720 bytes, maximum size is 8191
Or POST http://localhost:8080/api/v1/privacy-request/authenticated
[ { "policy_key":"default_access_policy", "identity":{ "email":"user@example.com" }, "custom_privacy_request_fields":{ "internal_ids":{ "label":"Internal ID", "value":[ "internal1", "internal2", "internal3" ] }, "evidence_box":{ "label":"Evidence Box", "value":[ "box-0", "box-1", "box-2", "box-3", "box-4", "box-5", "box-6", "box-7", "box-8", "box-9", "box-10", "box-11", "box-12", "box-13", "box-14", "box-15", "box-16", "box-17", "box-18", "box-19", "box-20", "box-21", "box-22", "box-23", "box-24", "box-25", "box-26", "box-27", "box-28", "box-29", "box-30", "box-31", "box-32", "box-33", "box-34", "box-35", "box-36", "box-37", "box-38", "box-39", "box-40", "box-41", "box-42", "box-43", "box-44", "box-45", "box-46", "box-47", "box-48", "box-49", "box-50", "box-51", "box-52", "box-53", "box-54", "box-55", "box-56", "box-57", "box-58", "box-59", "box-60", "box-61", "box-62", "box-63", "box-64", "box-65", "box-66", "box-67", "box-68", "box-69", "box-70", "box-71", "box-72", "box-73", "box-74", "box-75", "box-76", "box-77", "box-78", "box-79", "box-80", "box-81", "box-82", "box-83", "box-84", "box-85", "box-86", "box-87", "box-88", "box-89", "box-90", "box-91", "box-92", "box-93", "box-94", "box-95", "box-96", "box-97", "box-98", "box-99", "box-100", "box-101", "box-102", "box-103", "box-104", "box-105", "box-106", "box-107", "box-108", "box-109", "box-110", "box-111", "box-112", "box-113", "box-114", "box-115", "box-116", "box-117", "box-118", "box-119", "box-120", "box-121", "box-122", "box-123", "box-124", "box-125", "box-126", "box-127", "box-128", "box-129", "box-130", "box-131", "box-132", "box-133", "box-134", "box-135", "box-136", "box-137", "box-138", "box-139", "box-140", "box-141", "box-142", "box-143", "box-144", "box-145", "box-146", "box-147", "box-148", "box-149", "box-150", "box-151", "box-152", "box-153", "box-154", "box-155", "box-156", "box-157", "box-158", "box-159", "box-160", "box-161", "box-162", "box-163", "box-164", "box-165", "box-166", "box-167", "box-168", "box-169", "box-170", "box-171", "box-172", "box-173", "box-174", "box-175", "box-176", "box-177", "box-178", "box-179", "box-180", "box-181", "box-182", "box-183", "box-184", "box-185", "box-186", "box-187", "box-188", "box-189", "box-190", "box-191", "box-192", "box-193", "box-194", "box-195", "box-196", "box-197", "box-198", "box-199", "box-200", "box-201", "box-202", "box-203", "box-204", "box-205", "box-206", "box-207", "box-208", "box-209", "box-210", "box-211", "box-212", "box-213", "box-214", "box-215", "box-216", "box-217", "box-218", "box-219", "box-220", "box-221", "box-222", "box-223", "box-224", "box-225", "box-226", "box-227", "box-228", "box-229", "box-230", "box-231", "box-232", "box-233", "box-234", "box-235", "box-236", "box-237", "box-238", "box-239", "box-240", "box-241", "box-242", "box-243", "box-244", "box-245", "box-246", "box-247", "box-248", "box-249", "box-250", "box-251", "box-252", "box-253", "box-254", "box-255", "box-256", "box-257", "box-258", "box-259", "box-260", "box-261", "box-262", "box-263", "box-264", "box-265", "box-266", "box-267", "box-268", "box-269", "box-270", "box-271", "box-272", "box-273", "box-274", "box-275", "box-276", "box-277", "box-278", "box-279", "box-280", "box-281", "box-282", "box-283", "box-284", "box-285", "box-286", "box-287", "box-288", "box-289", "box-290", "box-291", "box-292", "box-293", "box-294", "box-295", "box-296", "box-297", "box-298", "box-299", "box-300", "box-301", "box-302", "box-303", "box-304", "box-305", "box-306", "box-307", "box-308", "box-309", "box-310", "box-311", "box-312", "box-313", "box-314", "box-315", "box-316", "box-317", "box-318", "box-319", "box-320", "box-321", "box-322", "box-323", "box-324", "box-325", "box-326", "box-327", "box-328", "box-329", "box-330", "box-331", "box-332", "box-333", "box-334", "box-335", "box-336", "box-337", "box-338", "box-339", "box-340", "box-341", "box-342", "box-343", "box-344", "box-345", "box-346", "box-347", "box-348", "box-349", "box-350", "box-351", "box-352", "box-353", "box-354", "box-355", "box-356", "box-357", "box-358", "box-359", "box-360", "box-361", "box-362", "box-363", "box-364", "box-365", "box-366", "box-367", "box-368", "box-369", "box-370", "box-371", "box-372", "box-373", "box-374", "box-375", "box-376", "box-377", "box-378", "box-379", "box-380", "box-381", "box-382", "box-383", "box-384", "box-385", "box-386", "box-387", "box-388", "box-389", "box-390", "box-391", "box-392", "box-393", "box-394", "box-395", "box-396", "box-397", "box-398", "box-399", "box-400", "box-401", "box-402", "box-403", "box-404", "box-405", "box-406", "box-407", "box-408", "box-409", "box-410", "box-411", "box-412", "box-413", "box-414", "box-415", "box-416", "box-417", "box-418", "box-419", "box-420", "box-421", "box-422", "box-423", "box-424", "box-425", "box-426", "box-427", "box-428", "box-429", "box-430", "box-431", "box-432", "box-433", "box-434", "box-435", "box-436", "box-437", "box-438", "box-439", "box-440", "box-441", "box-442", "box-443", "box-444", "box-445", "box-446", "box-447", "box-448", "box-449", "box-450", "box-451", "box-452", "box-453", "box-454", "box-455", "box-456", "box-457", "box-458", "box-459", "box-460", "box-461", "box-462", "box-463", "box-464", "box-465", "box-466", "box-467", "box-468", "box-469", "box-470", "box-471", "box-472", "box-473", "box-474", "box-475", "box-476", "box-477", "box-478", "box-479", "box-480", "box-481", "box-482", "box-483", "box-484", "box-485", "box-486", "box-487", "box-488", "box-489", "box-490", "box-491", "box-492", "box-493", "box-494", "box-495", "box-496", "box-497", "box-498", "box-499", "box-500", "box-501", "box-502", "box-503", "box-504", "box-505", "box-506", "box-507", "box-508", "box-509", "box-510", "box-511", "box-512", "box-513", "box-514", "box-515", "box-516", "box-517", "box-518", "box-519", "box-520", "box-521", "box-522", "box-523", "box-524", "box-525", "box-526", "box-527", "box-528", "box-529", "box-530", "box-531", "box-532", "box-533", "box-534", "box-535", "box-536", "box-537", "box-538", "box-539", "box-540", "box-541", "box-542", "box-543", "box-544", "box-545", "box-546", "box-547", "box-548", "box-549", "box-550", "box-551", "box-552", "box-553", "box-554", "box-555", "box-556", "box-557", "box-558", "box-559", "box-560", "box-561", "box-562", "box-563", "box-564", "box-565", "box-566", "box-567", "box-568", "box-569", "box-570", "box-571", "box-572", "box-573", "box-574", "box-575", "box-576", "box-577", "box-578", "box-579", "box-580", "box-581", "box-582", "box-583", "box-584", "box-585", "box-586", "box-587", "box-588", "box-589", "box-590", "box-591", "box-592", "box-593", "box-594", "box-595", "box-596", "box-597", "box-598", "box-599", "box-600", "box-601", "box-602", "box-603", "box-604", "box-605", "box-606", "box-607", "box-608", "box-609", "box-610", "box-611", "box-612", "box-613", "box-614", "box-615", "box-616", "box-617", "box-618", "box-619", "box-620", "box-621", "box-622", "box-623", "box-624", "box-625", "box-626", "box-627", "box-628", "box-629", "box-630", "box-631", "box-632", "box-633", "box-634", "box-635", "box-636", "box-637", "box-638", "box-639", "box-640", "box-641", "box-642", "box-643", "box-644", "box-645", "box-646", "box-647", "box-648", "box-649", "box-650", "box-651", "box-652", "box-653", "box-654", "box-655", "box-656", "box-657", "box-658", "box-659", "box-660", "box-661", "box-662", "box-663", "box-664", "box-665", "box-666", "box-667", "box-668", "box-669", "box-670", "box-671", "box-672", "box-673", "box-674", "box-675", "box-676", "box-677", "box-678", "box-679", "box-680", "box-681", "box-682", "box-683", "box-684", "box-685", "box-686", "box-687", "box-688", "box-689", "box-690", "box-691", "box-692", "box-693", "box-694", "box-695", "box-696", "box-697", "box-698", "box-699", "box-700", "box-701", "box-702", "box-703", "box-704", "box-705", "box-706", "box-707", "box-708", "box-709", "box-710", "box-711", "box-712", "box-713", "box-714", "box-715", "box-716", "box-717", "box-718", "box-719", "box-720", "box-721", "box-722", "box-723", "box-724", "box-725", "box-726", "box-727", "box-728", "box-729", "box-730", "box-731", "box-732", "box-733", "box-734", "box-735", "box-736", "box-737", "box-738", "box-739", "box-740", "box-741", "box-742", "box-743", "box-744", "box-745", "box-746", "box-747", "box-748", "box-749", "box-750", "box-751", "box-752", "box-753", "box-754", "box-755", "box-756", "box-757", "box-758", "box-759", "box-760", "box-761", "box-762", "box-763", "box-764", "box-765", "box-766", "box-767", "box-768", "box-769", "box-770", "box-771", "box-772", "box-773", "box-774", "box-775", "box-776", "box-777", "box-778", "box-779", "box-780", "box-781", "box-782", "box-783", "box-784", "box-785", "box-786", "box-787", "box-788", "box-789", "box-790", "box-791", "box-792", "box-793", "box-794", "box-795", "box-796", "box-797", "box-798", "box-799", "box-800", "box-801", "box-802", "box-803", "box-804", "box-805", "box-806", "box-807", "box-808", "box-809", "box-810", "box-811", "box-812", "box-813", "box-814", "box-815", "box-816", "box-817", "box-818", "box-819", "box-820", "box-821", "box-822", "box-823", "box-824", "box-825", "box-826", "box-827", "box-828", "box-829", "box-830", "box-831", "box-832", "box-833", "box-834", "box-835", "box-836", "box-837", "box-838", "box-839", "box-840", "box-841", "box-842", "box-843", "box-844", "box-845", "box-846", "box-847", "box-848", "box-849", "box-850", "box-851", "box-852", "box-853", "box-854", "box-855", "box-856", "box-857", "box-858", "box-859", "box-860", "box-861", "box-862", "box-863", "box-864", "box-865", "box-866", "box-867", "box-868", "box-869", "box-870", "box-871", "box-872", "box-873", "box-874", "box-875", "box-876", "box-877", "box-878", "box-879", "box-880", "box-881", "box-882", "box-883", "box-884", "box-885", "box-886", "box-887", "box-888", "box-889", "box-890", "box-891", "box-892", "box-893", "box-894", "box-895", "box-896", "box-897", "box-898", "box-899", "box-900", "box-901", "box-902", "box-903", "box-904", "box-905", "box-906", "box-907", "box-908", "box-909", "box-910", "box-911", "box-912", "box-913", "box-914", "box-915", "box-916", "box-917", "box-918", "box-919", "box-920", "box-921", "box-922", "box-923", "box-924", "box-925", "box-926", "box-927", "box-928", "box-929", "box-930", "box-931", "box-932", "box-933", "box-934", "box-935", "box-936", "box-937", "box-938", "box-939", "box-940", "box-941", "box-942", "box-943", "box-944", "box-945", "box-946", "box-947", "box-948", "box-949", "box-950", "box-951", "box-952", "box-953", "box-954", "box-955", "box-956", "box-957", "box-958", "box-959", "box-960", "box-961", "box-962", "box-963", "box-964", "box-965", "box-966", "box-967", "box-968", "box-969", "box-970", "box-971", "box-972", "box-973", "box-974", "box-975", "box-976", "box-977", "box-978", "box-979", "box-980", "box-981", "box-982", "box-983", "box-984", "box-985", "box-986", "box-987", "box-988", "box-989", "box-990", "box-991", "box-992", "box-993", "box-994", "box-995", "box-996", "box-997", "box-998", "box-999" ] } } } ]Pre-Merge Checklist
CHANGELOG.mdmaindowngrade()migration is correct and works