bug: stuck when dropping sink

[fuyufjh]

Describe the bug

Create sink (nexmark_q4), wait for a while, and then drop it. The client was stuck then.

CREATE SINK nexmark_q4
AS
SELECT Q.category,
       AVG(Q.final) as avg
FROM (SELECT MAX(B.price) AS final,
             A.category
      FROM auction A,
           bid B
      WHERE A.id = B.auction
        AND B.date_time BETWEEN A.date_time AND A.expires
      GROUP BY A.id, A.category) Q
GROUP BY Q.category
WITH ( connector = 'blackhole' );

Observations:

1. The CPU usage is zero.
2. async stack trace (verbose):

3. Barriers:

To Reproduce

Failed to reproduce it in my environment...

Expected behavior

No response

Additional context

[BugenZhao]

Looks like we're stuck in sync_epoch. Does it mean that there's a deadlock in the Hummock event loop? Note that dropping streaming jobs triggers the de-registration of the compaction groups and bloom filter catalog for internal tables, and it's easy to get it wrong.

Will #7637 be related?

[fuyufjh]

Will #7637 be related?

I guess not? #7637 was a deterministic bug while this one looks accidental.

Does it mean that there's a deadlock in the Hummock event loop? Note that dropping streaming jobs triggers the de-registration of the compaction groups and bloom filter catalog for internal tables, and it's easy to get it wrong.

Can you give some hint on how to investigate on this? I am trying to enable verbose_async_trace now.

[yezizp2012]

Will #7637 be related?

Guess not +1. #7637 only fix the metadata of sink catalog in catalog manager. Compaction group related registration and un-registration are already handled properly via TableFragments.

[BugenZhao]

I am trying to enable verbose_async_trace now.

async_stack_trace seems to be set to Verbose by default on the testing cluster as "store_await_sync" is already a verbose span. 🤔

I've quickly checked the code of the HummockEventHandler and there's even no await while processing an event. I've also run the gdb -batch -ex 'thread apply all bt' -p 1 to check the backtrace of all threads and find nothing interesting like parking-lot lock parking. 🤔

[fuyufjh]

async_stack_trace seems to be set to Verbose by default on the testing cluster as "store_await_sync" is already a verbose span. 🤔

Haha, that screenshot was updated by me after I set aync_stack_trace to "verbose"

[wenym1]

I've tried to reproduce the bug both in our local environment and in the CI environment, but failed.

The CI environment used to be able to reproduce the bug, but now we have run under the same benchmark setting for many times, but still cannot reproduce it.

I suspect that the bug has been fixed by some PR by the way, and I have added necessary logs to trace the progress of sync. If there is some pending sync task, we can see from the log.

I think we can hold this issue and observe whether the bug happens again in the future.

[fuyufjh]

It seems that this bug has not been seen for several months?

[wenym1]

It seems that this bug has not been seen for several months?

I think so. Maybe we can close this issue?

[fuyufjh]

Agree. Let's close it now.