Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fhir-notification: Limit handling for 1M #1948

Closed
prb112 opened this issue Feb 16, 2021 · 2 comments
Closed

fhir-notification: Limit handling for 1M #1948

prb112 opened this issue Feb 16, 2021 · 2 comments
Assignees
Labels

Comments

@prb112
Copy link
Contributor

prb112 commented Feb 16, 2021

Is your feature request related to a problem? Please describe.

fhir-notification for Kafka and Nats default to 1M sized message payloads. These messages are included in the Notification Event sent from the IBM FHIR Server to the Nats/Kafka endpoints.

If the messages exceed the limit, the Notification is never sent, where as it should be sent and handled appropriately.

Describe the solution you'd like

The Notification Service should support sending a stub message where the receiver can retrieve the large message payload.
This should be signaled to the user using a flag.

Describe alternatives you've considered
1 - Splitting the Messages (however the messages may be received out of order)
2 - Not sending a message (however the user never knows the data exists)

Acceptance Criteria
At least one acceptance criteria is included.

  1. Verify Kafka or Nats has a Message
    GIVEN Kafka OR Nats is configured as a Notification Service
    AND the Notification Service is Enabled
    WHEN A resource of 10M is created (such as a Binary Resource)
    AND success (201)
    THEN The resource is retrieved from the Location
    AND the Topic contains a message without the 10M resource.

Additional context
Refer to #1758 where the Nats issue was fixed by increasing the max payload size.

@lmsurpre
Copy link
Member

The notification message size should be configurable (default to 1MB). When message size is above that, send the stub instead. Needs clear documentation for consumers of the notification events so they know they might not get the payload.
They need to know that they should look for this flag before assuming there is a resource payload.

We should also provide an option to omit the resource payload on all notifications...the code already supports this but its not exposed via config.

@lmsurpre lmsurpre added the P3 Priority 3 - Nice To Have label Apr 12, 2021
@prb112 prb112 self-assigned this Jun 11, 2021
prb112 added a commit that referenced this issue Jun 15, 2021
- Updated fhir server's users guide to outline new preferences and add
missing fields
- Updated FHIRNotificationUtil to support new max Size behavior and
bytes so that it can be tuned to each use case by subset or omit when it
hits a limit. (There is no spanning/splitting to avoid delivery order
issues)
- Add Synchronous support to Kafka Notification and documentation how to
enable (users can now chose if they want a lossy approach or not)
- Add test coverage for fhir-notification and FHIRNotificationUtil

Signed-off-by: Paul Bastide <pbastide@us.ibm.com>
prb112 added a commit that referenced this issue Jun 17, 2021
fhir-notification: Limit handling for 1M #1948
@kmbarton423
Copy link
Contributor

New configuration properties:
fhirServer/notifications/common/maxNotificationSizeBytes => (default) 1000000
fhirServer/notifications/common/maxNotificationSizeBehavior => (default) "subset" or "omit"

Confirmed behavior of the above configuration properties via Kafka messages.
Tested a variety of settings as well as the defaults.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

3 participants