querylog migration failing on 0006_sorting_key_change

[chipzzz]

self-hosted/ helm-chart Sentry Version 23.12.1
externalclickhouse version: 23.4.1
singleNode=False
SNUBA_SETTINGS=distributed
SENTRY_DISTRIBUTED_CLICKHOUSE_TABLES = True

{"module": "snuba.migrations.runner", "event": "Running migration: 0006_sorting_key_change", "severity": "info", "timestamp": "2024-08-07T21:50:55.799090Z"}
Traceback (most recent call last):
  File "/usr/local/bin/snuba", line 33, in <module>
    sys.exit(load_entry_point('snuba', 'console_scripts', 'snuba')())
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/snuba/snuba/cli/migrations.py", line 114, in migrate
    runner.run_all(
  File "/usr/src/snuba/snuba/migrations/runner.py", line 219, in run_all
    self._run_migration_impl(
  File "/usr/src/snuba/snuba/migrations/runner.py", line 299, in _run_migration_impl
    migration.forwards(context, dry_run)
  File "/usr/src/snuba/snuba/migrations/migration.py", line 95, in forwards
    op.execute(logger)
  File "/usr/src/snuba/snuba/migrations/operations.py", line 539, in execute
    self.__func(logger)
  File "/usr/src/snuba/snuba/snuba_migrations/querylog/0006_sorting_key_change.py", line 96, in forwards
    update_querylog_table(connection, database)
  File "/usr/src/snuba/snuba/snuba_migrations/querylog/0006_sorting_key_change.py", line 41, in update_querylog_table
    assert new_create_table_statement.count(new_timestamp_type) == 0

[chipzzz]

@dbanda

[chipzzz]

Upgrading Clickhouse to 23.8.3(highest I can go with my helm chart not go past 23.8.11) helped but getting another issue - ClickHouse/ClickHouse#35518 - I can manually intervene by adjusting the script for now just painful.

[chipzzz]

Disabling
<internal_replication>false</internal_replication>
<compression>false</compression>
Seems to help in the version I was having trouble with