investigate #1351 functionality with compartment searches

[kmbarton423]

Issue #1351 was implemented to provide a means to constrain search options.
Need to confirm behavior with compartment searches.

[lmsurpre]

This one is about "how do we catch overly broad queries BEFORE executing them".
Troy added some stuff under #1351 for saying "you can only search by search parameter x if you ALSO include parameter y"
where the main use case would be "you need to include the patient parameter in your searches" (to scope them to a single patient).
But then, we were like "what if the query is scoped to a single patient by virtue of being a patient compartment search?"
"How do we make that work with what troy added for 1351?"

[michaelwschroeder]

When a tenant has configured search parameters or search parameter combinations using the fhirServer/resources/<resourceType>/searchParameters or fhirServer/resources/<resourceType>/searchParameterCombinations properties in the fhir-server-config.json file, there are potential issues with compartment search. These issues are as follows:

1. If compartment inclusion criteria search parameters are not specified in the fhirServer/resources/<resourceType>/searchParameters list for a given resource type, then the new generated compartment search token (if the fhirServer/search/useStoredCompartmentParam value is set to true - see issue Suboptimal compartment search execution plan on Postgres #1708) or the tokens for the inclusion criteria search parameters, are not created/stored at resource create/update time, and thus that resource will not get returned on a compartment search as it should.

   Example:

   1. fhir-server-config.json only specifies _id and _lastUpdated as valid search parameters for the Procedure resource type, not the patient and performer inclusion criteria search parameters:

      "resources": {
        "Procedure": {
          "searchParameters": {
            "_id": "http://hl7.org/fhir/SearchParameter/Resource-id",
            "_lastUpdated": "http://hl7.org/fhir/SearchParameter/Resource-lastUpdated"
          }
        }
      },
      

   2. A Procedure resource is created which contains the following Patient reference, meaning it should be associated with that Patient compartment:

      "subject": {
        "reference": "Patient/17575"
      },
      

      However, the Patient compartment inclusion criteria search parameters for the Procedure resource type, patient and performer, are not in the fhirServer/resources/<resourceType>/searchParameters list, so are not considered valid search parameters. This means the generated compartment search token (if enabled) or the inclusion criteria search tokens will not be created and indexed when the Procedure resource is stored.

   3. The following compartment search is performed:

      Patient/17575/Procedure
      

      The Procedure created in step ii will not be returned in the search results because the necessary compartment search token(s), which are what's used in the compartment search SQL, were not created and indexed.

2. If a compartment inclusion criteria search parameter is specified in the fhirServer/resources/<resourceType>/searchParameterCombinations list to limit search scope (i.e. patient or performer), then that parameter must be specified on a compartment search even though the compartment search already scopes the query to that search parameter. In addition, the search parameter will be specified twice in the generated SQL.

   Example:

   1. fhir-server-config.json specifies the patient search parameter in its fhirServer/resources/<resourceType>/searchParameterCombinations list for the Procedure resource type:

      "resources": {
        "Procedure": {
          "searchParameterCombinations": ["patient","patient+date","patient+status","patient+code"]
        }
      },
      

   2. The following compartment search is performed:

      Patient/17575/Procedure
      

      It will fail because the patient search parameter was not specified, even though the compartment search already scopes the search to a single Patient.

   3. The following compartment search is performed:

      Patient/17575/Procedure?patient=Patient/17575
      

      It will correctly return the Patient's Procedure resources, but the patient search parameter has to be unnecessarily specified, and the generated SQL will contain functionally duplicate WHERE clauses to scope the results to a single Patient.

Possible options to address issue #1

Option	Pros	Cons
1. Leave it working as-is and document behavior.	Documentation change only.	User may not get expected compartment search results.
2. When storing a resource for which compartment search tokens will not be indexed, return a warning on the POST or PUT request.	Gives user an indication that compartment search will not work for this resource.	User may not get expected compartment search results.
3. When generating the compartment search tokens, assume the inclusion criteria search parameters are always valid, rather than checking against the search parameter list configured for the tenant.	Tokens required for compartment search to work properly will always be indexed.	?

Possible options to address issue #2

Option	Pros	Cons
1. Leave it working as-is	No code changes required.	Requires user to specify superfluous search parameter, and generated SQL includes two WHERE clauses when it only needs one.
2. On compartment searches, just ignore the search parameter configuration properties in fhir-server-config	Doesn't require user to specify superfluous search parameters on compartment searches	Although compartment search currently constrains the search to a single compartment, tenant might actually want to constrain these searches even more.
3. Add new versions of the searchParameters and searchParameterCombinations properties in the fhir-server-config that are compartment-specific	Allows tenant to configure compartment search and normal search independently.	May be confusing to user. May be overkill.
4. Attempt some kind of matching where if a compartment inclusion criteria search parameter is specified in the searchParameters or searchParameterCombinations properties, then we don't require that parameter to be explicitly specified on a compartment search, since the search will already be constrained by that search parameter.	Doesn't require user to specify superfluous search parameters on compartment searches	May be confusing to user. Code would be fairly complex.

[michaelwschroeder]

After discussion with the team, issues will be addressed as follows:

Issue 1: For the short-term, we will go with option 1 to address this - work as-is and document potential conflicts between compartment search and configured search parameters (issue #1780). For the longer term, issue #1777 has been opened to implement option 3 (When generating the compartment search tokens, assume the inclusion criteria search parameters are always valid).

Issue 2: For the short-term, we will go with option 1 to address this - work as-is. For the longer term, issue #1778 has been opened to implement a derivation of option 3 (Add new versions of the searchParameters and searchParameterCombinations properties in the fhir-server-config that are compartment-specific).