10502 dxox

__TODO-10502.md

@@ -0,0 +1,54 @@
+# TODO
+
+- get rid of case-mapping (pair):
+  - getReadyForTrialCases (confirm finished in a deployed environment, and how to handle locking)
+  - advancedDocumentSearch
+    - Case title, petitioner advanced syntax
+  - Can we remove formatWorkItemResult and formatMessageResult? What about formatDocketEntryResult?
+
+- if we use the trigram search, run on the db CREATE EXTENSION IF NOT EXISTS pg_trgm;
+- Some TODOs are marked with "10502 TODO", so search for this (without the quotes) as needed. (solo, meetup as needed)
+- Make sure that report data for case-mapping reports (above) is correct, and make sure indices are added for efficiency as needed for all case-based reports (pair)
+- Make sure getCasesMetadataByUserId (previously called getCasesForUser) is working. It relies on processPractitionerMappingEntries streaming things into PractitionerOnCaseTable (pair)
+- Fix all of the broken tests (solo)
+- Do we need to implement Case entity locking? E.g., mimic withLocking. If we have removed locks, confirm with court that is ok. (pair)
+- Figure out if there are fields we can get rid of on database types to make things simpler (solo)
+- Remove console.logs
+
+---
+
+# Things to test:
+
+- Be consistent with Case and RawCase (maybe? not a big deal) and with returning undefined vs. throwing a not-found error in persistence methods
+- Places where we createCase/upsertCases sometimes also need to create/upsert rows in related tables (petitionerOnCase, etc.) that used to be children objects on Case dynamo records.
+- We removed duplicative getCountOfConsolidatedCases. Make sure that everything works as expected--Chris H mentioned that consolidated cases have had several refactors, so there might be a reason this existed after all.
+- Are there any unhappy paths in which latency-induced data discrepancies can occur between Postgres and Open Search?
+
+---
+
+# General notes
+
+Current summary:
+• All case (and directly related, i.e., stored on the case) seed data has been moved.
+• After talking with Jim L., it became apparent that petitioners on a case are almost entirely distinct from users. We may want an optional foreign key to a (forthcoming) user table, but the user table should really only be login/role info for petitioners. Their contact info on any particular case can be different from their "user" contact info or the contact info on a separate case.
+• The app runs with the fake data, although case is still missing some stuff.
+
+Continue to use CaseFactory, typing it correctly. I think the pattern should be:
+• We force developers to construct via CaseFactory, e.g., CaseFactory.getCase() (generic), CaseFactory.getFullCase (Case) CaseFactory.getPublicCase (PublicCase), etc.
+• PublicCase, Case, etc. entities should be constructed (at least for business logic--we currently use Case as both a representation of our database as well as a business object) from _create static helpers.
+
+So: CaseFactory.getFullCase(data, user) [which will check that the user actually has permissions for a Case and throw an error otherwise] -> cleans up data -> calls Case._create(data)
+
+This design is meant to solve a couple of issues:
+1) Make sure we are always getting a "safe" subset of data for a user. We have been relying on getCaseInteractor, random formatters, etc. to do this, which is dangerous. We should enforce one way of cleaning up a case regardless of which interactor etc.--hence the factory.
+2) There is one place that determines the permissions of what is and is not seen--the factory.
+3) Ideally, we could do the data obfuscation/clean up in each constructor. However, that requires passing user into each constructor, which makes the class harder to reason about, and it means we have multiple places/duplicate logic for cleaning up certain type-independent data obfuscation. That is why I have chosen to put this shared logic in the factory itself.
+4) Ultimately, this allows a developer to know that they are constructing the right type, getting a narrow type when they know what they want (e.g., a PublicCase from .getPublicCase) or a broad type when they don't (.getCase).
+
+- Update as of 20 December 2024:
+  - The most difficult part of this work so far has been figuring out how to get case data into open search.
+  - The scalable, reliable, "good" approaches -- streaming data from serverless Aurora to Open Search via, e.g., Kinesis -- are higher latency.
+  - The faster approaches -- "directly" writing to Open Search or Kinesis in one way or another -- are less scalable and reliable.
+  - We have more options if we switch to provisioned Aurora, but that opens a different can of worms.
+  - Moreover, since our eventual goal is to move as many queries from Open Search into Postgres anyway, much of the above work will be temporary.
+  - For those reasons, the current goal is to move all case-index Open Search queries to Postgres. Then we don't have to worry about the above issues, and the work we are doing will not need to be undone.

__aurora-streams.md

@@ -0,0 +1,62 @@
+# Goal: Stream AWS Aurora Postgres updates to Open Search
+
+## Constraints:
+1) A low-latency, real-time solution. The user should be able to create/update/etc. X and see the updates as soon as possible afterwards.
+
+## Option 1: Replicate (as far as possible) our existing setup, which is the following:
+
+DynamoDB --> DynamoDB Stream --> Lambda --> Open Search
+
+We would need to replace DynamoDB Stream with some other Change Data Capture (CDC) and streaming tool. Two options I have seen:
+
+1) Persistence function --> Aurora --> Data Migration Service --> Kinesis --> Lambda --> Open Search
+2) Persistence function --> Aurora --> Data Activity Stream --> Kinesis --> Lambda --> Open Search
+
+Both DMS and Data Activity Streams can be used to capture changes. DMS is more agnostic about source and destination databases, but it incurs an additional cost. Data Activity Streams are free, but they are narrower in scope (used for AWS RDS). Kinesis is a new cost regardless of our choice between 1 and 2.
+
+Pros:
+1) This continues our current pattern and keeps DB updates synced with--while, implementation-wise, decoupled from--Open Search
+2) More scalable and event-driven: Kinesis offers buffering to smooth out traffic spikes.
+
+Cons:
+1) Added cost of Kinesis and possibly DMS
+2) More moving parts: now we have to configure streams, kinesis, and format the data for Open Search
+3) Kinesis sometimes has duplicate or out-of-order messages.
+
+Basic argument for this approach: "This keeps us consistent with our current pattern, is more robust, and is more easily scalable. Plus, we might be able to use this event-driven pipeline for other things in the future."
+
+Issues I have run into:
+1) Data Activity Stream is only available for provisioned (not serverless) Aurora. This means
+2) I see conflicting information, but Data Migration Service latency seems like it might not be acceptable for some of our use cases. (Potential delays of minutes.)
+
+## Option 2a: "Directly" index in our persistence functions to stream to Open Search on successful Postgres updates
+
+Rather than streaming database changes at all, we modify our code so that any changes that need to be indexed 1) wait for a successful write to postgres and then 2) kick off an async task (storing a message in a queue, for instance) to index the update. Something like:
+
+Persistence function --> (Contingent upon successful update to Aurora) [--> Queue -->] Lambda --> Open Search
+
+Pros:
+1) Lower cost, probably
+2) Faster, maybe?
+-- With a queue, probably not. SQS, for instance, is poll based, not push-based, which introduces latency.
+-- Without a queue--viz., directly invoking a lambda--probably faster.
+3) Relatively simple infrastructure
+
+Cons:
+1) This breaks our existing pattern and puts the onus on us to update Open Search. Implementation between DB and Open Search is now more coupled.
+2) Without a queue, we would need to handle retry logic. Although maybe this retry logic is as simple as "on failure, send to queue," and then we have a separate lambda polling for failed indexing.
+3) Less scalable. If we had 1000s of writes per second, we could reach lambda concurrency limits.
+4) Less of a paper trail.
+
+Basic argument for this approach: "Look, we don't need to index all that much in open search anyway. Why set up all this infrastructure for streaming when we can just index what we need directly? Plus this incentivizes us to move as little into Open Search as necessary."
+
+## Option 2b: "Directly" invoke a lambda function or Kinesis stream in a postgres trigger
+
+This is like 2a, except rather than triggering in code, with trigger in postgres itself via pg_notify. An overview: we enable logical replication, 
+
+## Option 3: Use a plugin or third-party tool like Debezium as a CDC
+
+Cons:
+1) It seems like most of these tools work better with a provisioned Aurora instance
+
+## Option 4? Wrap getDbWriter in something that then invokes the change