distributed table hung - test draft showing problem

[filimonov]

I hereby agree to the terms of the CLA available at: https://yandex.ru/legal/cla/?lang=en

Changelog category (leave one):

• Other

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):

...

Detailed description / Documentation draft:

...

By adding documentation, you'll allow users to try your new feature immediately, not when someone else will have time to document it later. Documentation is necessary for all features that affect user experience in any way. You can add brief documentation draft above, or add documentation right into your patch as Markdown files in docs folder.

If you are doing this for the first time, it's recommended to read the lightweight Contributing to ClickHouse Documentation guide first.

[akuzm]

This part is expected -- you're pausing the container, so the server process is suspended. The kernel still accepts the connection, so the connection timeout doesn't fire. But the paused server can't reply to hello packet, hence the initiator waits for receive_timeout.

[filimonov]

I understand the mechanics. But i'm not satisfied with the result.

It need to work better. We have another healthy replica.

[akuzm]

This won't help, because if the server can be suddenly paused before hello packet, it can also be paused after it, with the same result.

[filimonov]

heartbeats

[akuzm]

Adding more kinds of packets to the protocol doesn't help -- they are all subject to the same problem, heartbeat packets included. The problem is not in the protocol but in the fact that we work with replicas synchronously. Anyway, we have some understanding of what can be done about this case, AFAIR @tavplubix wanted to make a prototype but haven't got to it yet.

[filimonov]

Yeah, it should be async.

[Avogar]

Fixed in #19291.

[filimonov]

@Avogar did you add that test?

[Avogar]

@Avogar did you add that test?

Yes, I added a similar test:

ClickHouse/tests/integration/test_hedged_requests/test.py

Lines 86 to 105 in 2e7f756

	def test_stuck_replica(started_cluster):
	cluster.pause_container("node_1")
	check_query(expected_replica="node_2")
	result = NODES['node'].query("SELECT slowdowns_count FROM system.clusters WHERE cluster='test_cluster' and host_name='node_1'")
	assert TSV(result) == TSV("1")
	result = NODES['node'].query("SELECT hostName(), id FROM distributed ORDER BY id LIMIT 1");
	assert TSV(result) == TSV("node_2\t0")
	# Check that we didn't choose node_1 first again and slowdowns_count didn't increase.
	result = NODES['node'].query("SELECT slowdowns_count FROM system.clusters WHERE cluster='test_cluster' and host_name='node_1'")
	assert TSV(result) == TSV("1")
	cluster.unpause_container("node_1")