Graceful Shutdown

[rhafer]

Describe the bug

When running in single binary mode, sending a SIGTERM (or SIGINT or SIGQUIT) to the ocis process will trigger a signal handlers for each reva service (https://github.com/cs3org/reva/blob/master/cmd/revad/internal/grace/grace.go#L265) where the first Handler that finishes with just call os.Exit() and end the complete ocis process.

We need to find a way to cleanly shutdown an ocis, finishing all in-flight request in order to prevent things like: https://github.com/owncloud/enterprise/issues/5783 from happening during regluar operation.

[rhafer]

Interesting read on this topic: https://www.rudderstack.com/blog/implementing-graceful-shutdown-in-go/

[rhafer]

We should probably first tackle this issue for the case when we're not using the ocis runtime. (I.e. fix the case when running the service as separated processes. This should at least avoid the race condition with the signal handlers of the different reva services.

We then just need to make the graceful shutdown behavior, which is currently triggered with a SIGQUIT, the default behavior for SIGTERM and SIGINT. (maybe based on a config setting). And get rid of those os.Exit() calls.

In second step we could then figure out how to disable the signal handlers in reva and trigger a proper reva shutdown from within the ocis runtime.

[micbar]

@rhafer I agree, we should focus on the single service runtime first.

IMO we could even look only on the storage users and storage system services for now.

Another idea might be to gather some information how severe the problem shows up in real life. We could run the inspector on the load test instance on the VP

@wkloucek Is that system still there?

[rhafer]

To scan all personal and project spaces this might be helpful:

for dir in $(find  /var/lib/ocis/storage/users/spaces  -mindepth 2 -maxdepth 2); do
    idpath=${dir#/var/lib/ocis/storage/users/spaces}
    id=${idpath///}
    ocis decomposedfs check-treesize -r /var/lib/ocis/storage/users/ -n $id
done

[wkloucek]

@rhafer I agree, we should focus on the single service runtime first.

IMO we could even look only on the storage users and storage system services for now.

There are also more reasons to NOT invest too much in the runtime: #6819

[rhafer]

There are also more reasons to NOT invest too much in the runtime: #6819

The main problem here is the reva runtime on the ocis one.

[rhafer]

#6840 together with cs3org/reva#4072 should allow a more graceful shutdown of the storage-users service, when running it standalone. It might make sense to introduce the *_GRACEFUL_SHUTDOWN_TIMEOUT setting to other reva-backed services as well (e.g. storage-system)

To do a proper shutdown when running everything in one process, we'll need to get rid of the exit calls inside reva and initiate a proper shutdown from ocis in a different way. (e.g. using the mechanism outlined in the above referenced blog post).