Chronos error on mutation ProcessRecurrence

[rafaelromcar-parabol]

Issue - Bug

• How serious is the bug? P1 / P2?
• OS: macOS / Linux / Windows / ? our Docker images
• Browser: Chrome / Safari / Lynx / ? -
• Node version: node --version v20.11.0
• NPM version: npm --version 10.2.4
• RethinkDB version: rethinkdb --version

Chronos will throw the following error non-stop

Trace: SEND TO SENTRY TIMEOUT {"query":"\n        mutation ProcessRecurrence{\n          pro"} undefined                                                                                                           
    at sendToSentry (/home/node/parabol/dist/chronos.js:17369:11)                                                                                                                                                  
    at publishWebhookGQL (/home/node/parabol/dist/chronos.js:17340:28)

(See Datadog for that date. We only have 15 days of history in Datadog, and it has been happening for at least those 15 days.

Acceptance Criteria (optional)

Chronos does not throw the error non-stop. And I suppose recurrence works.

Triage is performed to find the root cause of the bug, timeboxed to ~1 hour.

Estimated effort: 1 hour to triage. More if root cause is already identified.

[Dschoordsch]

If chronos cannot publish to the gql executor, it looks like it lost the connection to redis 🤔

[rafaelromcar-parabol]

Ah! Not clear enough on the logs IMHO. I can restart Chronos, even if it has been restarted multiple times in the last two weeks and we are getting that error message 😿

That said, Chronos should retry connecting because Redis is up and running as the other components are using it. And the error message should state that it can't connect to Redis 🤔

[Dschoordsch]

Might be something different then. I will check the logs.

[Dschoordsch]

I checked, it's not a real issue, just a wrong timeout. The logs 🔒 show that the mutation takes around 15s but the default query timeout is 10s. Nothing bad happens atm., but I think we wanted to use the 10m timeout instead, but lost it at some point.

[rafaelromcar-parabol]

Good that it is working then 😸 Still, the error message should be fixed to help catching real errors then.