Chore: This and that

.github/workflows/main.yml

@@ -20,6 +20,8 @@ jobs:
  env:
  OS: ${{ matrix.os }}
  PYTHON: ${{ matrix.python-version }}
+ # Do not tear down Testcontainers
+ TC_KEEPALIVE: true
 
  name: Python ${{ matrix.python-version }} on OS ${{ matrix.os }}
  steps:

doc/backlog.md

@@ -1,17 +1,72 @@
 # Backlog
 
 ## Iteration +1
-- Add additional check if data table(s) exists, or not.
-- Dissolve JOIN-based retention task gathering, because, when the application
- does not discover any retention policy job, it can not discriminate between
- "no retention policy" and "no data", and thus, it is not able to report about
- it correspondingly.
+- CLI: Use proper exit codes.
+- Make `--cutoff-day` optional, use `today()` as default.
 - Refactor "partition"-based strategies into subfamily/category, in order to
  make room for other types of strategies not necessarily using partitioned
  tables.
 
+## Iteration +1.5
+- CI: Nightly builds, to verify regressions on CrateDB
+- CI: Also build OCI images for ARM, maybe only on PRs to `main`, and releases?
+- CI: Add code coverage tracking and reporting.
+- More subcommands, like `list-policies` (`list`) and `check-policies` (`check`).
+- Improve testing for the `reallocate` strategy.
+- Provide components for emulating materialized views
+ - https://en.wikipedia.org/wiki/Materialized_view
+ - https://github.com/crate/crate/issues/10661
+ - https://github.com/crate/crate/issues/8806
+ - https://github.com/crate/crate/issues/10803
+ - https://github.com/crate/crate/issues/14377
+ - Example:
+ ```shell
+ cratedb-retention create-materialized-view doc.raw_metrics \
+ --as='SELECT * FROM <table_name>;'
+ cratedb-retention refresh-materialized-view doc.raw_metrics
+ ```
+
+## Iteration +1.75
+Add two non-partition-based strategies. Category: `timerange`.
+
+- Add a shortcut interface for adding policies.
+ - Provide a TTL-like interface.
+ - https://iotdb.apache.org/UserGuide/V1.1.x/Delete-Data/TTL.html
+ - https://iotdb.apache.org/UserGuide/V1.1.x/Query-Data/Continuous-Query.html#downsampling-and-data-retention
+ - Rename `retention period` to `duration`. It is shorter, and aligns with InfluxDB.
+ - https://docs.influxdata.com/influxdb/v1.8/query_language/manage-database/#create-retention-policies-with-create-retention-policy
+ - https://docs.influxdata.com/influxdb/v1.8/query_language/spec/#durations
+ - Example:
+ ```shell
+ cratedb-retention set-ttl doc.raw_metrics \
+ --strategy=delete --duration=1w
+ ```
+- Provide a solid (considering best-practices, DWIM) cascaded/multi-level
+ downsampling implementation/toolkit, similar to RRDtool or Munin.
+
+ - https://bostik.iki.fi/aivoituksia/projects/influxdb-cascaded-downsampling.html
+ - https://community.openhab.org/t/influxdb-in-rrd-style/88395
+ - https://github.com/influxdata/influxdb/issues/23108
+ - https://forums.percona.com/t/data-retention-archive-some-metrics-like-rrdtool-with-1d-resolution/21437
+- Naming things: Generalize to `supertask`.
+ - Examples
+ ```shell
+ # Example for a shortcut form of `supertask create-retention-policy`.
+ # <TABLE> <STRATEGY>:<TARGET> <DURATION>
+ st ttl doc.sensor_readings snapshot:export_cold 365d
+ ```
+ When using partition-based retention, previously using the `--partition-column=time_month`
+ option, that syntax might be suitable:
+ ```shell
+ # <TABLE> <PARTCOL> <STRATEGY>:<TARGET> <DURATION>
+ st ttl doc.sensor_readings:time_month snapshot:export_cold 4w
+ ```
+
 ## Iteration +2
-- Recurrent queries via scheduling.
+- Periodic/recurrent queries via scheduling.
+ - https://github.com/crate/crate/issues/11182
+ - https://github.com/crate/crate-insights/issues/75
+
  Either use classic cron, or systemd-timers, or use one of `APScheduler`,
  `schedule`, or `scheduler`.
 
@@ -30,23 +85,13 @@
  - https://github.com/agronholm/apscheduler
  - https://github.com/dbader/schedule
  - https://gitlab.com/DigonIO/scheduler
-- Document "Docker Compose" setup variant
+- Document complete "Docker Compose" setup variant, using both CrateDB and `cratedb-retention`
 - Generalize from `cutoff_day` to `cutoff_date`?
-- Refactor SQL queries once more, introducing comment-stripping, and renaming the files.
-- Make all tests work completely.
- The `snapshot` and `reallocate` scenarios are currently untested.
-- Battle testing.
-- More subcommands, like `list-policies` (`list`) and `check-policies` (`check`).
-- Improve how to create a policy, see README and `examples/basic.py`
-- Remedy the need to do a `run_sql` step by introducing a subcommand `add-policy`.
-- Provide a solid (considering best-practices, DWIM) cascaded/multi-level
- downsampling implementation/toolkit, similar to RRDtool or Munin.
-
- - https://bostik.iki.fi/aivoituksia/projects/influxdb-cascaded-downsampling.html
- - https://community.openhab.org/t/influxdb-in-rrd-style/88395
- - https://github.com/influxdata/influxdb/issues/23108
- - https://forums.percona.com/t/data-retention-archive-some-metrics-like-rrdtool-with-1d-resolution/21437
-- OCI: Also build for ARM, maybe only on PRs to `main`, and releases?
+ For example, use `ms`. See https://iotdb.apache.org/UserGuide/V1.1.x/Delete-Data/TTL.html.
+- More battle testing, in sandboxes and on production systems.
+- Use storage classes
+ - https://github.com/crate/crate/issues/14298
+ - https://github.com/crate/crate/pull/14346
 
 ## Iteration +3
 - Review SQL queries: What about details like `ORDER BY 5 ASC`?
@@ -95,3 +140,15 @@
 - Document compact invocation, after applying an alias and exporting an
  environment variable: `cratedb-retention rm --tags=baz`
 - Default value for `"${CRATEDB_URI}"` aka. `dburi` argument
+- Add additional check if data table(s) exists, or not.
+- Dissolve JOIN-based retention task gathering, because, when the application
+ does not discover any retention policy job, it can not discriminate between
+ "no retention policy" and "no data", and thus, it is not able to report about
+ it correspondingly.
+- CLI: Provide `--dry-run` option
+- Docs: Before running the examples, need to invoke `cratedb-retention setup --schema=examples`
+- For testing the snapshot strategy, provide an embedded MinIO S3 instance to the test suite.
+- Improve SNAPSHOT testing: Microsoft Azure Blob Storage
+ - https://learn.microsoft.com/en-us/azure/storage/common/storage-use-azurite
+ - https://learn.microsoft.com/en-us/azure/storage/blobs/use-azurite-to-run-automated-tests
+- Improve SNAPSHOT testing: Filesystem

pyproject.toml

@@ -206,7 +206,7 @@ extend-exclude = [
 ]
 
 [tool.ruff.per-file-ignores]
-"tests/*" = ["S101", "T201"] # Allow use of `assert`, and `print`.
+"tests/*" = ["S101"] # Allow use of `assert`, and `print`.
 "examples/*" = ["T201"] # Allow `print`
 "cratedb_retention/cli.py" = ["T201"] # Allow `print`
 

tests/conftest.py

@@ -135,9 +135,11 @@ def raw_metrics(cratedb, settings, store):
  Populate the `raw_metrics` table.
  """
 
+ tablename_full = f'"{TESTDRIVE_DATA_SCHEMA}"."raw_metrics"'
+
  database_url = cratedb.get_connection_url()
  ddl = f"""
- CREATE TABLE "{TESTDRIVE_DATA_SCHEMA}"."raw_metrics" (
+ CREATE TABLE {tablename_full} (
  "variable" TEXT,
  "timestamp" TIMESTAMP WITH TIME ZONE,
  "ts_day" TIMESTAMP GENERATED ALWAYS AS date_trunc('day', "timestamp"),
@@ -151,7 +153,7 @@ def raw_metrics(cratedb, settings, store):
  """
 
  dml = f"""
- INSERT INTO "{TESTDRIVE_DATA_SCHEMA}"."raw_metrics"
+ INSERT INTO {tablename_full}
  (variable, timestamp, value, quality)
  SELECT
  'temperature' AS variable,
@@ -163,7 +165,9 @@ def raw_metrics(cratedb, settings, store):
 
  run_sql(database_url, ddl)
  run_sql(database_url, dml)
- run_sql(database_url, f'REFRESH TABLE "{TESTDRIVE_DATA_SCHEMA}"."raw_metrics";')
+ run_sql(database_url, f"REFRESH TABLE {tablename_full};")
+
+ return tablename_full
 
 
 @pytest.fixture(scope="function")
@@ -172,9 +176,11 @@ def sensor_readings(cratedb, settings, store):
  Populate the `sensor_readings` table.
  """
 
+ tablename_full = f'"{TESTDRIVE_DATA_SCHEMA}"."sensor_readings"'
+
  database_url = cratedb.get_connection_url()
  ddl = f"""
- CREATE TABLE "{TESTDRIVE_DATA_SCHEMA}"."sensor_readings" (
+ CREATE TABLE {tablename_full} (
  time TIMESTAMP WITH TIME ZONE NOT NULL,
  time_month TIMESTAMP WITH TIME ZONE GENERATED ALWAYS AS DATE_TRUNC('month', "time"),
  sensor_id TEXT NOT NULL,
@@ -186,7 +192,7 @@ def sensor_readings(cratedb, settings, store):
  """
 
  dml = f"""
- INSERT INTO "{TESTDRIVE_DATA_SCHEMA}"."sensor_readings"
+ INSERT INTO {tablename_full}
  (time, sensor_id, battery_level, battery_status, battery_temperature)
  SELECT
  generate_series AS time,
@@ -203,7 +209,9 @@ def sensor_readings(cratedb, settings, store):
 
  run_sql(database_url, ddl)
  run_sql(database_url, dml)
- run_sql(database_url, f'REFRESH TABLE "{TESTDRIVE_DATA_SCHEMA}"."sensor_readings";')
+ run_sql(database_url, f"REFRESH TABLE {tablename_full};")
+
+ return tablename_full
 
 
 @pytest.fixture(scope="function")

tests/test_cli.py

@@ -146,6 +146,9 @@ def test_run_delete_basic(store, database, raw_metrics, policies):
  database_url = store.database.dburi
  runner = CliRunner()
 
+ # Check number of records in database.
+ assert database.count_records(raw_metrics) == 6
+
  # Invoke data retention through CLI interface.
  result = runner.invoke(
  cli,
@@ -154,8 +157,8 @@ def test_run_delete_basic(store, database, raw_metrics, policies):
  )
  assert result.exit_code == 0
 
- # Verify that records have been deleted.
- assert database.count_records(f'"{TESTDRIVE_DATA_SCHEMA}"."raw_metrics"') == 0
+ # Check number of records in database.
+ assert database.count_records(raw_metrics) == 0
 
 
 def test_run_delete_dryrun(caplog, store, database, raw_metrics, policies):
@@ -166,19 +169,21 @@ def test_run_delete_dryrun(caplog, store, database, raw_metrics, policies):
  database_url = store.database.dburi
  runner = CliRunner()
 
+ # Check number of records in database.
+ assert database.count_records(raw_metrics) == 6
+
  # Invoke data retention through CLI interface.
  result = runner.invoke(
  cli,
  args=f'run --cutoff-day=2024-12-31 --strategy=delete --dry-run "{database_url}"',
  catch_exceptions=False,
  )
  assert result.exit_code == 0
-
- # Verify that records have been deleted.
- assert database.count_records(f'"{TESTDRIVE_DATA_SCHEMA}"."raw_metrics"') == 6
-
  assert "Pretending to execute SQL statement" in caplog.text
 
+ # Check number of records in database.
+ assert database.count_records(raw_metrics) == 6
+
 
 def test_run_delete_with_tags_match(store, database, sensor_readings, policies):
  """
@@ -188,6 +193,9 @@ def test_run_delete_with_tags_match(store, database, sensor_readings, policies):
  database_url = store.database.dburi
  runner = CliRunner()
 
+ # Check number of records in database.
+ assert database.count_records(sensor_readings) == 9
+
  # Invoke data retention through CLI interface.
  result = runner.invoke(
  cli,
@@ -208,6 +216,9 @@ def test_run_delete_with_tags_unknown(caplog, store, database, sensor_readings,
  database_url = store.database.dburi
  runner = CliRunner()
 
+ # Check number of records in database.
+ assert database.count_records(sensor_readings) == 9
+
  # Invoke data retention through CLI interface.
  result = runner.invoke(
  cli,
@@ -216,11 +227,12 @@ def test_run_delete_with_tags_unknown(caplog, store, database, sensor_readings,
  )
  assert result.exit_code == 0
 
- # Verify that records have not been deleted, because the tags did not match.
- assert database.count_records(f'"{TESTDRIVE_DATA_SCHEMA}"."sensor_readings"') == 9
-
  assert "No retention policies found with tags: ['foo', 'unknown']" in caplog.messages
 
+ # Check number of records in database.
+ # Records have not been deleted, because the tags did not match.
+ assert database.count_records(sensor_readings) == 9
+
 
 def test_run_reallocate(store, database, raw_metrics, raw_metrics_reallocate_policy):
  """
@@ -230,6 +242,9 @@ def test_run_reallocate(store, database, raw_metrics, raw_metrics_reallocate_pol
  database_url = store.database.dburi
  runner = CliRunner()
 
+ # Check number of records in database.
+ assert database.count_records(raw_metrics) == 6
+
  # Invoke data retention through CLI interface.
  result = runner.invoke(
  cli,
@@ -238,11 +253,11 @@ def test_run_reallocate(store, database, raw_metrics, raw_metrics_reallocate_pol
  )
  assert result.exit_code == 0
 
- # Verify that records have been deleted.
+ # Check number of records in database.
  # FIXME: Currently, the test for this strategy apparently does not remove any records.
  # The reason is because the scenario can't easily be simulated on a single-node
  # cluster. The suite would need to orchestrate at least two nodes.
- assert database.count_records(f'"{TESTDRIVE_DATA_SCHEMA}"."raw_metrics"') == 6
+ assert database.count_records(raw_metrics) == 6
 
 
 def test_run_snapshot(caplog, store, database, sensor_readings, sensor_readings_snapshot_policy):