Addons FxA webhook endpoint is returning 403s to FxA

[biancadanforth]

Describe the problem and steps to reproduce it:

What happened?

Starting November 10th, 2022, the prod Addons endpoint (https://addons.mozilla.org/api/auth/fxa-notification) started throwing 403s for webhook delivery requests from FxA.

What did you expect to happen?

200s from this endpoint.

Anything else we should know?

Per @bbangert:

We make the request in a uniform manner to all RP's.

Since we don't see any other RPs (relying parties) returning the same errors, we are guessing it might be an issue validating the request.

┆Issue is synchronized with this Jira Task

[diox]

Need to trigger password change, primary email change and account deletion from FxA on dev once mozilla/addons-server#20201 is deployed, and look into the logs.

[ioanarusiczki]

@diox Right now this is open , waiting to be closed.

[diox]

Sorry, still opened because I haven't got a fix yet, just added more logging to see what's going on. You can try triggering events I noted in #9044 and I'll look at the logs.

[ioanarusiczki]

@diox

From the fxA Profile page (which is Manage Firefox Accounts URL that redirects from AMO's Edit Profile Page to fxA stage)

I changed password and the primary email for rusiczki.ioana+35@gmail.com
Deleted account for rusiczki.ioana+37@gmail. com

[diox]

I don't see anything in the logs, unfortunately this could mean it's not connected correctly for our dev environment... We might need to test on stage (which doesn't have the added logging yet)

[ioanarusiczki]

@diox Done on -stage , same accounts.

I'll try again -dev with fxA stage, there could be a difference in behaviors but I'm not sure yet.

[diox]

The patch to add necessary logging has hit stage yet (needs to be cherry-picked)

[ioanarusiczki]

@diox

Right now on -dev when I changed password and left the edit profile page open in a different tab I an not being logged out. Meanwhile on -stage if I refresh the edit profie page -> I am logged out

When I switch the primary email , on dev the old email is still displayed (only after I log out and login again I will see it changed). On stage the email address changes to the new one at a page refresh (no need to logout)

[ioanarusiczki]

@diox

I see the same at deleting the account.

On -dev I am left with a login.
On -stage I'm automatically logged out from AMO.

P.S. these are older -dev issues. Ok, I'll wait for -stage then.

[diox]

The fact that you are automatically logged out/email is automatically updated on stage means that the error isn't reproducible there, so that's interesting...

[ioanarusiczki]

@diox

I don't see a different behavior on -stage after repeating the testing now.

[diox]

I can see some errors on stage & prod now with the added logging: The token is not yet valid (iat). leeway is 0, since this is not re-using the JWT config we have for our developer API keys.

[sentry-io]

Sentry issue: ADDONS-SERVER-PROD-B5

[sentry-io]

Sentry issue: ADDONS-SERVER-STAGE-35

[diox]

The root of the problem is that FxA is sending iat as floats with millisecond precision. When we upgraded PyJWT, we got this extra check, jpadilla/pyjwt#794, but PyJWT is using the iat provided as is (a float in our case), comparing against the current datetime as an integer. So without a leeway, it fails (almost) everytime.

jpadilla/pyjwt#814 would likely fix this but is not yet deployed. FxA could also send the iat as integer to help, but I think we'll want a short (1 to 5 seconds max - all tracebacks I found didn't need more than 1) leeway just in case.

[diox]

Since QA couldn't find any functional issue, I suspect most of these errors could be caused by notifications of events we don't actually care about... But it'd be good to fix in any case.

[diox]

Fix is on stage.

[ioanarusiczki]

@diox

I made all the necessary operations again. -> it did not change in behavior since yesterday

I've noticed at change password that after I'm logged out of AMO and try to login again, I'm asked to Confirm sign-in on email.

I did not receive the email to confirm the sign-in from the first attempt so I clicked on Resend which works, I'm getting the email. -> this happened 3/3 attempts

[bobsilverberg]

@ioanarusiczki I believe our intention is to push this change (from mozilla/addons-server#20210) to prod today. Has this been verified on stage?

[ioanarusiczki]

@bobsilverberg The results above are from today's checking.
As I said, I did not see changes compared to yesterday's testing on AMO stage.

[KevinMind]

Old Jira Ticket: https://mozilla-hub.atlassian.net/browse/ADDSRV-210