feat(connector): single events table across namespaces

[hekike]

Single events table across namespaces.

ClickHouse recommendation is only to have one insert per second.
See: ClickHouse/ClickHouse#34170

This PR batches all inserts into a single monthly partition across all namespaces as:

A thumb rule , the insert should insert into only one partition. Number of inserts is up to 1 insert per second PER INSTANCE. (NOT PER TABLE).

This PR stores all namespace events into a single table partitioned by month:

You should never use too granular partitioning. Don't partition your data by client identifiers or names (instead, make client identifier or name the first column in the ORDER BY expression).

Breaking Change:

This PR changes the query behind materialized views moving the meters.
As altering the query of MaterializedViews backed by AggregatingMergeTree is currently unimplemented in ClickHouse, there is no straightforward path forward to migrate.

We recommend creating new meters. Old meters remain queriable.

Migration:

If you MUST migrate follow:

As pre-1.0.0 release, this breaking change requires manual migration.

1. Stop sink-worker or Kafka Connect

This stop ingestion to the ClickHouse DB. Events will queue up in Kafka topic, so you won't lose new data just delay meter aggregates.

2. Update Events Table in ClickHouse

Run the following command to update your ClickHouse DB:

curl -u default:default -d "RENAME DATABASE openmeter.om_default_events TO openmeter.om_events" -X POST http://localhost:8123
curl -u default:default -d "ALTER TABLE openmeter.om_events ADD COLUMN namespace String FIRST" -X POST http://localhost:8123
curl -u default:default -d "ALTER TABLE openmeter.om_events UPDATE namespace = 'default' WHERE empty(namespace);" -X POST http://localhost:8123

3. Start OpenMeter With Recreate Config

Add the following to your config.yaml temporary.

aggregation:
  createOrReplaceMeter: true
  populateMeter: true

Starting OpenMeter with these settings can take longer.

4. Restart sink-worker

Restart sink-worker and remove createOrReplaceMeter and populateMeter from your config.

If you are stuck with migration, reach out on our Discord, and we will assist you.