Queues not terminating gracefully

[brechtvl]

Description

We are running into the issue where a Gitea restart happens during merge conflict checking, PRs remain stuck in conflict state forever. I think there's two distinct issues here, one is that these don't get unstuck on new pushes and perhaps that's best left for another report.

However the reason things get into this state in the first place seems to be a problem in shutdown terminating all the queues immediately instead of respecting the hammer time.

When starting ./gitea web and then do killall gitea , I see the following in the log:

2023/02/21 19:43:26 ...eful/manager_unix.go:149:handleSignals() [W] [63f510cd-4] PID 1500309. Received SIGTERM. Shutting down...
2023/02/21 19:43:26 cmd/web.go:271:listen() [I] [63f510cd-48] HTTP Listener: 0.0.0.0:3000 Closed
2023/02/21 19:43:26 ...eue/queue_channel.go:127:func1() [W] ChannelQueue: task-channel Terminated before completed flushing
2023/02/21 19:43:26 .../graceful/manager.go:205:doHammerTime() [W] Setting Hammer condition
2023/02/21 19:43:26 ...eue/queue_channel.go:127:func1() [W] ChannelQueue: notification-service-channel Terminated before completed flushing
2023/02/21 19:43:26 ...eue/queue_channel.go:127:func1() [W] ChannelQueue: push_update-channel Terminated before completed flushing
2023/02/21 19:43:26 ...eful/server_hooks.go:46:doShutdown() [I] [63f510cd-48] PID: 1500309 Listener ([::]:3000) closed.
2023/02/21 19:43:27 .../graceful/manager.go:224:doTerminate() [W] Terminating
2023/02/21 19:43:27 ...er/issues/indexer.go:201:2() [I] PID: 1500309 Issue Indexer closed
2023/02/21 19:43:27 ...eful/manager_unix.go:157:handleSignals() [W] PID: 1500309. Background context for manager closed - context canceled - Shutting down...
2023/02/21 19:43:27 cmd/web.go:183:runWeb() [I] PID: 1500309 Gitea Web Finished

This log is from my local test instance with configuration set to defaults as much as possible.

On our production instance that Terminated before completed flushing is leading to a lot of different errors as workers get terminated in the middle of what they're doing.

I can provide more detailed logs if needed, but maybe this is easy to redo on any instance.

Gitea Version

main (43405c3)

Can you reproduce the bug on the Gitea demo site?

No

Log Gist

No response

Screenshots

No response

Git Version

No response

Operating System

No response

How are you running Gitea?

Own build from main.

Database

None

[techknowlogick]

For debugging purposes: [W] [63f40381] cannot get cache context when getting data: &{context.Background.WithCancel.WithCancel.WithValue(type pprof.labelContextKey, val {"graceful-lifecycle":"with-hammer"}) was also found in the logs, which traces back to the following function:
https://sourcegraph.com/github.com/go-gitea/gitea/-/blob/modules/cache/context.go?L58

Edit: this is also showing up on gitea.com, although not during shutdowns.

[brechtvl]

I'm also not seeing "graceful-lifecycle":"with-hammer" messages when simply starting and stopping gitea. But when I for example do a push that causes conflict checking workers to be run for PRs, then they start showing up.

So I guess there is some problem with cache context in the workers?

[zeripath]

If I had to guess this is going to be a misreport and there's going to be a non-deterministic hierarchy of cancelled contexts - the wrong one of which is being picked and reported. I'd need to spend a good few hours looking through the code thinking about the concurrency again which I won't have time to look at until the weekend.

[lunny]

The context cache should be used in http request level currently. Do you have a redid cache server?

[techknowlogick]

@lunny they are using memcached for cache

[lunny]

@lunny they are using memcached for cache

This maybe occurs when Gitea start up.

[brechtvl]

I see these Terminated before completed flushing warning messages also in v1.16.9 (had some problems testing earlier). So I'm not sure if this is a problem that has existed for a while, or if these messages are not as serious as they sound and the problem is elsewhere.

Here is the part of the log from the production server where it got the shutdown signal and then ran into errors, if it helps:
https://gist.github.com/brechtvl/e4778293c1c919a4ec896a31e418677c

[techknowlogick]

@lunny the same ctx error can be seen in Gitea.com logs and occurs at times other than startup and shutdown

[lunny]

I have sent a PR #23054, but I don't think that will fix this problem in fact. It will remove the unnecessary warning logs.

[lunny]

I see these Terminated before completed flushing warning messages also in v1.16.9 (had some problems testing earlier). So I'm not sure if this is a problem that has existed for a while, or if these messages are not as serious as they sound and the problem is elsewhere.

Here is the part of the log from the production server where it got the shutdown signal and then ran into errors, if it helps: https://gist.github.com/brechtvl/e4778293c1c919a4ec896a31e418677c

From the logs, looks like it's a normal terminal process.

[brechtvl]

Everything seems to shut down within 1s while GRACEFUL_HAMMER_TIME was left to the default 60s. I was expecting it to wait up to 60 seconds for any ongoing work to complete. Or is that only meant for requests, not queues?

If it's not meant for queues, is the mechanism that work gets added back to the queue on such errors, to be redone after restart? It wasn't doing that as far as I could tell, but I may well have missed something.

[lunny]

I think the logic should be if all workers shut down less than 60s, then it will exit immediately. If some workers shutting down time out(address 60s), then hammer shutdown will be executed and the program will exit at that time.