Missing `in_progress` checkin with Cron Monitoring

[SoerenWeber]

How do you use Sentry?

Sentry Saas (sentry.io)

Version

2.8.0

Steps to Reproduce

I use Cron Monitoring to watch the execution of scheduled, long-running jobs, managed by RQ. The monitoring is done by using the monitor context manager:

with monitor(monitor_slug=settings.BACKUP_MONITOR_SLUG):
  pass

To mitigate potential connection problems, keep_alive is enabled in the sentry_sdk.init call. The same project runs in multiple instances on DigitalOcean infrastructure.

Expected Result

The monitor should send a check-in both for the start and end of execution. These check-ins should be visible on sentry.io.

Actual Result

The in_progress check-ins are pretty consistently missed, resulting in monitor failure. After the job ran, the ok check-in is sent successfully and resolves the failed monitor.

After enabling debug in sentry_sdk.init, it seems that the in_progress check-in just isn't sent:

Jul 11 02:00:32 [worker] {"event": "default: project.tasks.create_external_backup() (a85d1596-974d-4b9b-8acb-01c4ce58e693)"}
 Jul 11 02:00:33 [worker] [sentry] DEBUG: [Tracing] Discarding <queue.task.rq> transaction <project.tasks.create_external_backup> because traces_sample_rate is set to 0
 Jul 11 02:00:33 [worker] Backup started.
 Jul 11 02:50:00 [worker] Backup done.
 Jul 11 02:50:00 [worker] [sentry] DEBUG: Sending envelope [envelope with 1 items (monitor)] project:[redacted].ingest.us.sentry.io
 Jul 11 02:50:00 [worker] {"event": "default: Job OK (a85d1596-974d-4b9b-8acb-01c4ce58e693)"}
 Jul 11 02:50:00 [worker] {"event": "Result will never expire, clean up result key manually"}
 Jul 11 02:50:00 [worker] [sentry] DEBUG: Discarding transaction because sampled = False
 Jul 11 02:50:00 [worker] [sentry] DEBUG: Flushing HTTP transport
 Jul 11 02:50:00 [worker] [sentry] DEBUG: background worker got flush request
 Jul 11 02:50:00 [worker] [sentry] DEBUG: Sending envelope [envelope with 1 items (monitor)] project:[redacted].ingest.us.sentry.io
 Jul 11 02:50:00 [worker] [sentry] DEBUG: Sending envelope [envelope with 1 items (internal)] project:[redacted].ingest.us.sentry.io
 Jul 11 02:50:00 [worker] [sentry] DEBUG: background worker flushed

Interestingly, there does seem to be some kind of networking issue because a bit earlier I found these entries, unrelated to the job mentioned above:

Jul 11 01:35:33 [worker] [sentry] DEBUG: Discarding transaction because sampled = False
 Jul 11 01:35:33 [worker] [sentry] DEBUG: Flushing HTTP transport
 Jul 11 01:35:33 [worker] [sentry] DEBUG: background worker got flush request
 Jul 11 01:35:33 [worker] [sentry] DEBUG: 2 event(s) pending on flush
 Jul 11 01:35:35 [worker] [sentry] ERROR: flush timed out, dropped 2 events
 Jul 11 01:35:35 [worker] [sentry] DEBUG: background worker flushed

Any guidance would be much appreciated! 🙏

[szokeasaurusrex]

@SoerenWeber, thanks for reporting this issue!

Has this ever worked properly for you, perhaps in a previous SDK version? If so, which SDK version was this still working with?

[SoerenWeber]

@szokeasaurusrex, thank you for your fast reaction!

I never got this working reliably, unfortunately. It seemed that adding the keep_alive flag improved the reliability slightly a few months ago, but the results were inconclusive (some check-ins seemed to succeed, but I could not find a pattern there).

[szokeasaurusrex]

@SoerenWeber, you're welcome! Are you able to share your source code repo with me (if it is public), or do you have a minimal reproduction you are able to share?

If not, that is okay; in that case, I can see whether I can reproduce the bug from the information you have already provided.

[SoerenWeber]

@szokeasaurusrex, I know it is a hassle, but I cannot share the code with you and do not have an MRE at the moment, sorry. I will try to provide one in the coming weeks, though.

I would greatly appreciate you looking into this, if you have the capacity. Otherwise, I will update this issue once I was able to narrow down the problem space.