feat: Heatmaps ingestion

[benjackwhite]

Problem

Ingestion part of #21487

Related API PR #21623
Related JS PR PostHog/posthog-js#1131

Changes

• Adds a new step for parsing the special $heatmaps_data property
• Wrapped in try catch to make sure it doesn't affect overall event ingestion if it fails
• Converts to new kafka events and returns them as delayed acks
• One outstanding question about what to do if it errors

👉 Stay up-to-date with PostHog coding conventions for a smoother review.

Does this work well for both Cloud and self-hosted?

How did you test this code?

[pauldambra]

didn't run it but LGTM

[bretthoerner]

Makes sense to me, the fact that it's in a try/catch and uses the existing ack system should mean it's safe. 👍

One thing I'm curious about: is this part of a JS SDK change that we'll just flip on for all autocapture events or something? The test heatmap data looks kind of large, is that representative? Can they be larger? Is there a limit or any protection in the SDK? I'm mostly wondering about data volume changes flowing through capture and multiple Kafka topics and whether we need to consider that all.

I don't have anything particular to worry about, it just got my spidey sense tingling.

[benjackwhite]

Makes sense to me, the fact that it's in a try/catch and uses the existing ack system should mean it's safe. 👍

One thing I'm curious about: is this part of a JS SDK change that we'll just flip on for all autocapture events or something? The test heatmap data looks kind of large, is that representative? Can they be larger? Is there a limit or any protection in the SDK? I'm mostly wondering about data volume changes flowing through capture and multiple Kafka topics and whether we need to consider that all.

I don't have anything particular to worry about, it just got my spidey sense tingling.

maybe possibly. We have it as a JS change and it could indeed end up being a half decent amount of data. It is however client side flagged currently so we will be testing it just for our team for starters so we can get a better sense of how the data flow goes.

I think I'll add a server-side setting for it as well so that we can control it remotely to give us a little more reaction if we do end up seeing worrying sizes.