feat: add idempotency interceptor #4531

mahendraHegde · 2023-10-15T15:37:14Z

What change does this PR introduce?

Adds Idempotency interceptor to provide idempotency to POST and PATCH requests
Updates Docs
Adds tests

Reopens #4472

Why was this change needed?

closes #4469

Other information (Screenshots)

New request

Replayed request

Cliftonz · 2023-10-16T13:59:16Z

@mahendraHegde We are seeing the following error in the pipeline

 ERROR in /home/runner/work/novu/novu/apps/api/src/app/shared/framework/idempotency.interceptor.ts
16:28-50
[tsl] ERROR in /home/runner/work/novu/novu/apps/api/src/app/shared/framework/idempotency.interceptor.ts(16,29)
      TS2307: Cannot find module 'libs/shared/dist/cjs' or its corresponding type declarations.
      ``` 
      Would you know what this means? if not I can ask a team member.

mahendraHegde · 2023-10-16T14:03:52Z

@mahendraHegde We are seeing the following error in the pipeline

 ERROR in /home/runner/work/novu/novu/apps/api/src/app/shared/framework/idempotency.interceptor.ts
16:28-50
[tsl] ERROR in /home/runner/work/novu/novu/apps/api/src/app/shared/framework/idempotency.interceptor.ts(16,29)
      TS2307: Cannot find module 'libs/shared/dist/cjs' or its corresponding type declarations.
      ``` 
      Would you know what this means? if not I can ask a team member.

@Cliftonz i think its a mistake of auto import, let me fix this

mahendraHegde · 2023-10-16T14:09:51Z

@mahendraHegde We are seeing the following error in the pipeline

 ERROR in /home/runner/work/novu/novu/apps/api/src/app/shared/framework/idempotency.interceptor.ts
16:28-50
[tsl] ERROR in /home/runner/work/novu/novu/apps/api/src/app/shared/framework/idempotency.interceptor.ts(16,29)
      TS2307: Cannot find module 'libs/shared/dist/cjs' or its corresponding type declarations.
      ``` 
      Would you know what this means? if not I can ask a team member.

@Cliftonz i think its a mistake of auto import, let me fix this

@Cliftonz can you try now?

apps/api/src/app/shared/framework/idempotency.interceptor.ts

.cspell.json

apps/api/src/app/shared/framework/idempotency.interceptor.ts

apps/api/src/.env.production

apps/api/src/app/shared/framework/idempotency.interceptor.ts

apps/api/e2e/idempotency.e2e.ts

rifont · 2023-10-17T14:39:18Z

Hi @mahendraHegde, thanks for the great work on this PR already.

@Cliftonz and I have discussed a few additions to the idempotency implementation which we think will deliver significant short and long-term benefits to users of this feature, using Stripe API as a guide for these additional implementation details. These are changes that will be potentially breaking to add later, should a client begin to depend on the existing implementation detail.

These additions are summarised below, with rationale in brackets.

Return 503 Service unavailable when a request includes an Idempotency-Key header and the request is not processable (this avoids a situation where a client expecting idempotency has their request processed without it, which is potentially dangerous and reputationally damaging for the client in the event of duplicate triggers, for example)
Remove the error attribute from the response body entirely (the Novu team will seek to formalize a response error format specification before we implement additional attributes. This may involve either response header or body attributes. It would be great to get your input here)
Add an Idempotency-Replayed: true header to the request in the event of a cache hit of the idempotency feature (this is potentially valuable information for the client to know when they produce an idempotency cache hit, we should make this information available in the interest of transparency)
Return 409 Conflict when a response is ongoing with the same Idempotency-Key and request body (we are aligning here with Stripe's API and a number of other leading API implementations, choosing to follow the HTTP specification and acknowledging this will result in an additional error code to handle in client implementations).
Modify the idempotency cache TTL from 6 hours to 24 hours (this is the de facto standard across leading API implementations, and importantly allows server hosts more time to resolve production incidents affecting idempotent requests)

If you could please include these updates in the PR alongside the other suggestions, we can move this through to include in the next release 😃

apps/api/src/app/shared/framework/idempotency.interceptor.ts

mahendraHegde · 2023-10-18T01:03:30Z

@rifont i have fixed comments and added few comments, let me know your thoughts

apps/api/src/app/shared/framework/idempotency.interceptor.ts

apps/api/src/types/env.d.ts

apps/api/src/app/testing/testing.controller.ts

apps/api/src/app/shared/framework/idempotency.interceptor.ts

rifont · 2023-10-24T08:33:15Z

Hey @mahendraHegde, I've added a few last comments on the PR.

Additionally, could you please test with your own API key via curl/Postman to verify the following scenarios and post screenshots in the PR description, as we had issues in practice with API key auth in the previous merge of the PR:

Request with API Key auth and a noted Idempotency-Key to POST /events/trigger results in 201 Created
Request with API Key auth and the same noted Idempotency-Key to POST /events/trigger results in 201 Created with Idempotency-Replated true.

Lastly, we need to update the Idempotency docs PR (novuhq/docs#202) to reflect new behavior.

Thanks again for your great work on this critical reliability feature. We're nearly there 🙌

LetItRock

hey @mahendraHegde 👋 thanks a lot for your contribution to Novu 🙌
I hope that I'm not too late for the party here 😄 left you a few suggestions 😉

apps/api/e2e/idempotency.e2e.ts

apps/api/src/app/shared/framework/idempotency.interceptor.ts

…address review comments

mahendraHegde · 2023-10-26T02:37:33Z

@Cliftonz @rifont @LetItRock I believe i have addressed all review comments let me know if i missed any.

Cliftonz · 2023-10-26T12:24:40Z

@rifont The pipeline is failing due to bad container build all of the unit test have passed for this.
If all the requirements are met for this pr and you have manually tested the functionality, I am good with merging this.

rifont · 2023-10-26T17:36:20Z

I have verified the feature locally:

Testing Snippet

Initial Request with key 12341

>
curl -v -X POST http://localhost:3000/v1/events/trigger \
  -H "Authorization: ApiKey XXXXXXXXXXXXXXXXXX" \
  -H "Idempotency-Key: 12341" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "sample",
    "to": {
      "subscriberId": "XXXXXXXXXXXXXXXXXX",
      "email": "john@doemail.com",
      "firstName": "John",
      "lastName": "Doe"
    },
    "payload": {
      "name": "Hello World",
      "organization": {
        "logo": "https://happycorp.com/logo.png"
      }
    }
  }'
...
> Authorization: ApiKey XXXXXXXXXXXXXXXXXX
...
< HTTP/1.1 201 Created
...
< Idempotency-Key: 12341
...
{"data":{"acknowledged":true,"status":"processed","transactionId":"297b420f-8685-4545-b055-bea3e81e8841"}}%

Replayed Request with key 12341, Idempotency-Replay header present

>
curl -v -X POST http://localhost:3000/v1/events/trigger \
  -H "Authorization: ApiKey XXXXXXXXXXXXXXXXXX" \
  -H "Idempotency-Key: 12341" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "sample",
    "to": {
      "subscriberId": "XXXXXXXXXXXXXXXXXX",
      "email": "john@doemail.com",
      "firstName": "John",
      "lastName": "Doe"
    },
    "payload": {
      "name": "Hello World",
      "organization": {
        "logo": "https://happycorp.com/logo.png"
      }
    }
  }'
...
> Authorization: ApiKey XXXXXXXXXXXXXXXXXX
> Idempotency-Key: 12341
...
< HTTP/1.1 201 Created
...
< Idempotency-Key: 12341
< Idempotency-Replay: true
...
{"data":{"acknowledged":true,"status":"processed","transactionId":"297b420f-8685-4545-b055-bea3e81e8841"}}%

Reused key 12341, different body - 422 Unprocessable Entity

>
curl -v -X POST http://localhost:3000/v1/events/trigger \
  -H "Authorization: ApiKey XXXXXXXXXXXXXXXXXX" \
  -H "Idempotency-Key: 12341" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "sample",
    "to": {
      "subscriberId": "XXXXXXXXXXXXXXXXXX",
      "email": "john@doesmail.com",
      "firstName": "John",
      "lastName": "Doe"
    },
    "payload": {
      "name": "Hello World",
      "organization": {
        "logo": "https://happycorp.com/logo.png"
      }
    }
  }'
...
> Authorization: ApiKey XXXXXXXXXXXXXXXXXX
> Idempotency-Key: 12341
...
< HTTP/1.1 422 Unprocessable Entity
...
< Idempotency-Key: 12341
< Link: docs.novu.co/idempotency
{"message":"Request with key \"12341\" is being reused for a different body","error":"Unprocessable Entity","statusCode":422}%

Initial Request with key 12345 resulting in 400 Bad Request

>
curl -v -X POST http://localhost:3000/v1/events/trigger \
  -H "Authorization: ApiKey XXXXXXXXXXXXXXXXXX" \
  -H "Idempotency-Key: 12345" \
  -H "Content-Type: application/json" \
  -d '{
    "to": {
      "subscriberId": "XXXXXXXXXXXXXXXXXX",
      "email": "john@doesmail.com",
      "firstName": "John",
      "lastName": "Doe"
    },
    "payload": {
      "name": "Hello World",
      "organization": {
        "logo": "https://happycorp.com/logo.png"
      }
    }
  }'
...
> Authorization: ApiKey XXXXXXXXXXXXXXXXXX
> Idempotency-Key: 12345
...
< HTTP/1.1 400 Bad Request
...
< Idempotency-Key: 12345
...
{"message":["name should not be null or undefined","name must be a string"],"error":"Bad Request","statusCode":400}%

Replayed 400 Bad Request with key 12345, Idempotency-Replay header present

>
curl -v -X POST http://localhost:3000/v1/events/trigger \
  -H "Authorization: ApiKey XXXXXXXXXXXXXXXXXX" \
  -H "Idempotency-Key: 12345" \
  -H "Content-Type: application/json" \
  -d '{
    "to": {
      "subscriberId": "XXXXXXXXXXXXXXXXXX",
      "email": "john@doesmail.com",
      "firstName": "John",
      "lastName": "Doe"
    },
    "payload": {
      "name": "Hello World",
      "organization": {
        "logo": "https://happycorp.com/logo.png"
      }
    }
  }'
...
> Authorization: ApiKey XXXXXXXXXXXXXXXXXX
> Idempotency-Key: 12345
.
< HTTP/1.1 400 Bad Request
...
< Idempotency-Key: 12345
< Idempotency-Replay: true
...
{"message":["name should not be null or undefined","name must be a string"],"error":"Bad Request","statusCode":400}%

Superb work on this feature @mahendraHegde ! Thank you for this important contribution.

@LetItRock could you please take a look at the requested changes?

LetItRock · 2023-10-27T09:31:14Z

@mahendraHegde Looks good to me. Excellent job! Thank you! 🙌

github-actions bot added @novu/api community labels Oct 15, 2023

mahendraHegde force-pushed the idempotency-impl branch from 4798d36 to 0b566a6 Compare October 15, 2023 15:57

Cliftonz requested review from scopsy, p-fernandez and rifont October 15, 2023 16:05

mahendraHegde force-pushed the idempotency-impl branch 2 times, most recently from f4bf5fb to 8d253d7 Compare October 16, 2023 14:08

rifont suggested changes Oct 16, 2023

View reviewed changes

rifont suggested changes Oct 17, 2023

View reviewed changes

mahendraHegde force-pushed the idempotency-impl branch from 25eefcf to 9d73164 Compare October 18, 2023 01:01

rifont suggested changes Oct 18, 2023

View reviewed changes

mahendraHegde commented Oct 18, 2023

View reviewed changes

apps/api/src/app/shared/framework/idempotency.interceptor.ts Show resolved Hide resolved

mahendraHegde force-pushed the idempotency-impl branch from 9d73164 to 89601f2 Compare October 21, 2023 00:29

feat: add idempotency interceptor

40e5226

mahendraHegde force-pushed the idempotency-impl branch from 89601f2 to 40e5226 Compare October 21, 2023 00:29

rifont suggested changes Oct 23, 2023

View reviewed changes

apps/api/src/types/env.d.ts Outdated Show resolved Hide resolved

apps/api/src/app/testing/testing.controller.ts Show resolved Hide resolved

apps/api/src/app/shared/framework/idempotency.interceptor.ts Show resolved Hide resolved

LetItRock requested changes Oct 24, 2023

View reviewed changes

Merge branch 'next' into idempotency-impl

4c09d4c

rifont suggested changes Oct 25, 2023

View reviewed changes

apps/api/src/app/shared/framework/idempotency.interceptor.ts Outdated Show resolved Hide resolved

apps/api/src/app/shared/framework/idempotency.interceptor.ts Show resolved Hide resolved

apps/api/src/app/shared/framework/idempotency.interceptor.ts Outdated Show resolved Hide resolved

rifont added 3 commits October 25, 2023 14:57

fix: Idempotency-Replay noun convention

c3feff1

test(api): Fix idempotency header fixture naming

4403ca8

fix: Idempotency interceptor typo

35042d3

rifont reviewed Oct 25, 2023

View reviewed changes

apps/api/src/app/shared/framework/idempotency.interceptor.ts Outdated Show resolved Hide resolved

rifont and others added 9 commits October 25, 2023 15:04

fix: Remove redundant throwError wrapper

1c85adf

Merge branch 'next' into idempotency-impl

4443046

fix(api): few tests were flakey in the ci/cd

8c15f66

feat(shared): tidy up filters operators to reuse in events broadcast

c9ecd3f

Added the Georgian language typescript file

2c0db35

Update ka.ts

d76ec3c

Enhanced SECURITY.md

b431cb2

feat: add idempotency interceptor

3247f37

fix: add link response header test, validate idempotency key length, …

f3c8ee9

…address review comments

github-actions bot added @novu/web @novu/notification-center @novu/shared @novu/worker labels Oct 26, 2023

Merge branch 'next' into idempotency-impl

3a03f70

github-actions bot removed @novu/web @novu/notification-center @novu/shared @novu/worker labels Oct 26, 2023

rifont approved these changes Oct 26, 2023

View reviewed changes

Merge branch 'next' into idempotency-impl

7890302

Cliftonz approved these changes Oct 26, 2023

View reviewed changes

Cliftonz requested a review from LetItRock October 26, 2023 22:37

Merge branch 'next' into idempotency-impl

d0dc15a

LetItRock approved these changes Oct 27, 2023

View reviewed changes

LetItRock merged commit 460708d into novuhq:next Oct 27, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: add idempotency interceptor #4531

feat: add idempotency interceptor #4531

mahendraHegde commented Oct 15, 2023 •

edited

Loading

Cliftonz commented Oct 16, 2023

mahendraHegde commented Oct 16, 2023

mahendraHegde commented Oct 16, 2023

rifont commented Oct 17, 2023 •

edited

Loading

mahendraHegde commented Oct 18, 2023

rifont commented Oct 24, 2023 •

edited

Loading

LetItRock left a comment •

edited

Loading

mahendraHegde commented Oct 26, 2023

Cliftonz commented Oct 26, 2023

rifont commented Oct 26, 2023 •

edited

Loading

LetItRock commented Oct 27, 2023

feat: add idempotency interceptor #4531

feat: add idempotency interceptor #4531

Conversation

mahendraHegde commented Oct 15, 2023 • edited Loading

What change does this PR introduce?

Why was this change needed?

Other information (Screenshots)

Cliftonz commented Oct 16, 2023

mahendraHegde commented Oct 16, 2023

mahendraHegde commented Oct 16, 2023

rifont commented Oct 17, 2023 • edited Loading

mahendraHegde commented Oct 18, 2023

rifont commented Oct 24, 2023 • edited Loading

LetItRock left a comment • edited Loading

Choose a reason for hiding this comment

mahendraHegde commented Oct 26, 2023

Cliftonz commented Oct 26, 2023

rifont commented Oct 26, 2023 • edited Loading

LetItRock commented Oct 27, 2023

mahendraHegde commented Oct 15, 2023 •

edited

Loading

rifont commented Oct 17, 2023 •

edited

Loading

rifont commented Oct 24, 2023 •

edited

Loading

LetItRock left a comment •

edited

Loading

rifont commented Oct 26, 2023 •

edited

Loading