clickhouse broken after upgrade to 24.5.1 from 23.6.2 (from 22.1.0)

[maxp1256]

Self-Hosted Version

23.6.2

CPU Architecture

x86_64

Docker Version

24.0.5

Docker Compose Version

2.27.1

Steps to Reproduce

I've a self hosted sentry instance with version 22.1.0 and want to migrate it to current 24.5.1
After (a successful) upgrade to the required intermediate step 23.6.2 it breaks the clickhouse container with the following errors:

clickhouse-1 | 2024.06.06 21:25:11.570376 [ 1 ] {} Application: Caught exception while loading metadata: Code: 48. DB::Exception: Data directory for system database exists, but metadata file does not. Probably you are trying to upgrade from version older than 20.7. If so, you should upgrade through intermediate version. (NOT_IMPLEMENTED), Stack trace (when copying this message, always include the lines below):

Are there any undocumented intermediated steps to upgrade 23.6.2 to latest?

Expected Result

Working 24.5.1 instance

Actual Result

clickhouse-1 | 2024.06.06 21:25:11.563935 [ 1 ] {} Application: Listen [::]:9009 failed: Poco::Exception. Code: 1000, e.code() = 0, DNS error: EAI: Address family for hostname not supported (version 23.8.11.29.altinitystable (altinity build)). If it is an IPv6 or IPv4 address and your host has disabled IPv6 or IPv4, then consider to specify not disabled IPv4 or IPv6 address to listen in <listen_host> element of configuration file. Example for disabled IPv6: <listen_host>0.0.0.0</listen_host> . Example for disabled IPv4: <listen_host>::</listen_host>
clickhouse-1 | 2024.06.06 21:25:11.570376 [ 1 ] {} Application: Caught exception while loading metadata: Code: 48. DB::Exception: Data directory for system database exists, but metadata file does not. Probably you are trying to upgrade from version older than 20.7. If so, you should upgrade through intermediate version. (NOT_IMPLEMENTED), Stack trace (when copying this message, always include the lines below):
clickhouse-1 |
clickhouse-1 | 0. DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000c61ff37 in /usr/bin/clickhouse
clickhouse-1 | 1. DB::Exception::Exception<String const&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x00000000074a9bcd in /usr/bin/clickhouse
clickhouse-1 | 2. DB::checkUnsupportedVersion(std::shared_ptrDB::Context, String const&) @ 0x00000000122dd350 in /usr/bin/clickhouse
clickhouse-1 | 3. DB::loadSystemDatabaseImpl(std::shared_ptrDB::Context, String const&, String const&) @ 0x00000000122e540b in /usr/bin/clickhouse
clickhouse-1 | 4. DB::loadMetadataSystem(std::shared_ptrDB::Context) @ 0x00000000122e43de in /usr/bin/clickhouse
clickhouse-1 | 5. DB::Server::main(std::vector<String, std::allocator> const&) @ 0x000000000c6b7f1d in /usr/bin/clickhouse
clickhouse-1 | 6. Poco::Util::Application::run() @ 0x0000000015b50a66 in /usr/bin/clickhouse
clickhouse-1 | 7. DB::Server::run() @ 0x000000000c6a6dde in /usr/bin/clickhouse
clickhouse-1 | 8. Poco::Util::ServerApplication::run(int, char**) @ 0x0000000015b5fb79 in /usr/bin/clickhouse
clickhouse-1 | 9. mainEntryClickHouseServer(int, char**) @ 0x000000000c6a3baa in /usr/bin/clickhouse
clickhouse-1 | 10. main @ 0x000000000712c42f in /usr/bin/clickhouse
clickhouse-1 | 11. ? @ 0x00007fa85a1cd083 in ?
clickhouse-1 | 12. _start @ 0x000000000635186e in /usr/bin/clickhouse
clickhouse-1 | (version 23.8.11.29.altinitystable (altinity build))

Event ID

No response

[hubertdeng123]

Hmmm, that is probably because 23.6.2 is on clickhouse 20.3 still. Could you try to upgrade to 23.11.0 before proceeding to 24.5.1?
https://github.com/getsentry/self-hosted/releases/tag/23.11.0

That release includes a bump to clickhouse 21.8

[martijnbolhuis]

Hello,

I had exactly the same problem upgrading from 22.9.0 -> 23.6.2 -> 24.5.1. I resolved this by reverting & restoring a backup to 22.9.0. Subsequently, I did the upgrade again and added an intermediate upgrade to 24.4.2 so: 22.9.0 -> 23.6.2 -> 24.4.2 -> 24.5.1. According to the changelog clickhouse was upgraded in 24.5.0 so that is why I added the version before that (24.4.2) as an intermediate stop.

I think 24.4.2 should be added as a "hard stop" in the upgrade documentation?

[maxp1256]

Hello,
@hubertdeng123 I can confirm, that the upgrade to 23.11.0 was successful, but after upgrading to 24.4.2 as recommended from @martijnbolhuis it's broken again.

I don't know if this output is useful, but it seems that something is wrong with clickhouse or related components?
logoutput.txt

[hubertdeng123]

@martijnbolhuis I think it's more accurate to have a hard stop added at 23.11.0, since that is where the first bump to clickhouse 21.8 occurred. I'd prefer not to use 24.4.2 since that includes an upgrade to postgres 14.11-alpine, which we have reverted back to using the debian image.

@maxp1256 It is concerning that your writes are being skipped in the logs. What part of self-hosted are you seeing broken here? For the clickhouse errors, you can follow along in this issue:
getsentry/snuba#5707

[maxp1256]

@hubertdeng123 That's a good question. I didn't try to open the Webif after seeing the stacktraces at the console. Indeed it seems to be working. Currently I'm at 24.4.2 which is not recommended according to your last post. So I'm trying to upgrade again during next week. Is there any ETA when the snuba-problem will be fixed?

[hubertdeng123]

Yep, if you're using 24.4.2, you should try to change the postgres-14.11-alpine image in your docker-compose.yml file to postgres-14.11. Unfortunately, there is no ETA for the snuba problem.

[spiffytortoise]

There is a upgrade-clickhouse.sh script, but it only runs for certain versions. I think it got skipped in our install since we were on a older clickhouse version.

I manually ran the clickhouse upgrades:

docker compose build --build-arg BASE_IMAGE=altinity/clickhouse-server:21.8.13.1.altinitystable clickhouse
docker compose up clickhouse
# wait then ^C
docker compose build --build-arg BASE_IMAGE=altinity/clickhouse-server:22.8.15.25.altinitystable clickhouse
docker compose up clickhouse
# wait then ^C
docker compose build --build-arg BASE_IMAGE=altinity/clickhouse-server:23.3.19.33.altinitystable clickhouse
docker compose up clickhouse
# wait then ^C
docker compose build --build-arg BASE_IMAGE=altinity/clickhouse-server:23.8.11.29.altinitystable clickhouse
docker compose up clickhouse
# wait then ^C
docker compose up -d

[envy]

Hi,

I experienced the same problem. I had an installation on 23.8.0 and seeing no hard stops in the documentation went to 24.6.0 and then got a failing clickhouse container.

After reading the comments here, I first "downgraded" to 24.4.1 (not 24.4.2) which went without issues. Then I upgraded to 24.6.0 again and everything worked.

So my chain was: 23.8.0 -> 24.4.1 -> 24.6.0

[Kashemir001]

+1, same as above was trying to upgrade 23.8.0 -> 24.6.0, was only able to do so with a hard stop on 23.11.0

[hubertdeng123]

Got it, thanks for the info everyone. We'll add notes related to this.