runc-facade squashes rare runc errors

[utam0k]

Is your feature request related to a problem? Please describe

This issue came from #9247 (comment)

Rarely runc will cause an error due to something (probably seccomp notify). It is very likely that these errors are difficult to resolve at the root. Therefore, runc-facade is used to retry and squash these errors.

This problem still happens recently, the gcp log

Solution

Please give it a retry when runc returns any error around here

gitpod/components/docker-up/runc-facade/main.go

Lines 89 to 92 in 52848f6

	err = syscall.Exec(runcPath, os.Args, os.Environ())
	if err != nil {
	return xerrors.Errorf("exec %s: %w", runcPath, err)
	}

How to reproduce

Open this repository
https://github.com/spearki/gitpod-runc-issue-repro

Describe the behaviour you'd like

Wrote above

NOTE

• Container startup time is small. Image pull is irrelevant.
• now, we don't have a good way to distinguish between runc error or user application error Some questions about error handling on runc, crun or other runtime opencontainers/runtime-spec#1061

[kylos101]

Thanks @utam0k!

runc-facade is used to retry and squash these errors.

How long, or how many times should runc-facade do retry before it gives up?

Are there any scenarios where we would NOT want to retry, because retrying would be disruptive to the user experience?

[utam0k]

At least one time. Even so, it should be effective enough.

How long, or how many times should runc-facade do retry before it gives up?

Unfortunately, when the user application gets failed in starting a container, runc-facade tried to retry. Ideally, we have to distinguish between the error from runc or a user application, but we can't now. So in case of user application failure, we will have to retry in vain.
Fortunately, runc creating time is very fast, I recommend, don't worry to retry if the user application gets failed
https://github.com/containers/youki#motivation

Are there any scenarios where we would NOT want to retry, because retrying would be disruptive to the user experience?

[kylos101]

@utam0k I removed this from breakdown, but, as you shared it will only take 30m, please feel free to assign yourself and set status toIn-Progress.

If it ends up taking more time, please move back to our inbox (no status).

[jenting]

Therefore, runc-facade is used to retry and squash these errors.

@utam0k
Could you please point me to where is the runc-facade retry code?