Env ulimit/prlimit not inherited by Gramine process

[thomasknauth]

Description of the problem

Gramine has some support for get/setrlimit(). Usually, the limits are inherited by a child process. Changing the limit, e.g., a common one is increasing the maximum allowed open files, in a shell session and subsequently invoking Gramine will not have the desired effect.

Steps to reproduce

Using the bash example from CI-Examples, invoke ulimit -n 4096 ; gramine-direct ./bash -c 'ulimit -n'.

Expected results

Gramine should pick up the limits from the env. In the above example, it should report 4096.

Actual results

This will report 900 (DEFAULT_MAX_FDS).

Gramine commit hash

gramine/jammy,now 1.5 amd64

[dimakuv]

The fact that Gramine just uses some built-in default is definitely a bug.

However, I'm not sure that Gramine must get the limits from the untrusted environment. I think a better solution would be to introduce new manifest options, which the app/manifest developer can set to known-to-be-good values, or use a passthrough if the developer wants to pick up the limits from the env.

I think we even discussed this with @mkow at some point.

The proposed manifest syntax may look like this:

loader.rlimit.[NAME] = "[VALUE]"
or
loader.rlimit.[NAME] = { value = "[VALUE]" }
or
loader.rlimit.[NAME] = { passthrough = true }

For example, in @thomasknauth case, the manifest will specify:

loader.rlimit.RLIMIT_NOFILE = { passthrough = true }

This is the same to how Gramine specifies environment variables.

References:

• https://github.com/gramineproject/gramine/blob/a60dcf78e9b5d9c4f1d315c63dd087592668b8e3/libos/src/sys/libos_getrlimit.c -- current Gramine implementation
• https://gramine.readthedocs.io/en/stable/manifest-syntax.html#environment-variables -- similar Gramine syntax for environment variables
• https://man7.org/linux/man-pages/man2/getrlimit.2.html

[mkow]

Gramine should pick up the limits from the env.

What do you mean? Env variables? That's not how it works on Linux?

The proposed manifest syntax may look like this:

loader.rlimit.[NAME] = "[VALUE]"
or
loader.rlimit.[NAME] = { value = "[VALUE]" }
or
loader.rlimit.[NAME] = { passthrough = true }

We could do that, but why is this needed?

Also, the original issue here mixes two independent problems:

• Whether limits are inherited between in-Gramine processes.
• What are the default/initial values for them.

[dimakuv]

Whether limits are inherited between in-Gramine processes.

The limits are inherited between in-Gramine processes, notice the attribute_migratable keyword:

gramine/libos/src/sys/libos_getrlimit.c

Line 37 in a60dcf7

static struct __kernel_rlimit64 __rlim[RLIM_NLIMITS] __attribute_migratable = {

Also, I don't think this was an original question.

What are the default/initial values for them.

I think this was the original question.

Gramine should pick up the limits from the env.
What do you mean? Env variables? That's not how it works on Linux?

No, I meant "env" = "the host environment". In case of rlimits, the host information should be queried via the getrlimit() host syscall.

We could do that, but why is this needed?

Sometimes you need to adjust your rlimits on your system, for example, for web servers that have a large number of clients (i.e., large number of opened FDs). On a normal system, you won't rely on the application to do that, instead you as a sysadmin increase the rlimit via e.g. ulimit -n 4096.

Now run this application under Gramine. It will fail because Gramine will refuse to open an FD, since the in-Gramine rlimit is reached. And the application doesn't know/understand that it needs to increase this limit itself, because typically the apps just rely on the sysadmin to do the job.

In Gramine case, we currently don't have this "sysadmin will do this job" (i.e., via passthrough or via value = 4096 in the manifest). That's the current problem.

[thomasknauth]

Even setting the limit from within the application is not guaranteed to work. By the time the app's main() function is reached and a new limit may be set, the app may have already cloned/forked a bunch of processes/threads which will still have the default hard-coded Gramine limit.

[dimakuv]

By the time the app's main() function is reached and a new limit may be set, the app may have already cloned/forked a bunch of processes/threads which will still have the default hard-coded Gramine limit.

@thomasknauth I can't understand how this flow is possible.

Why would the app clone/fork before the main() function? C/C++/Rust/Go applications do not do that. Are you using some language runtime/framework that does this? Or maybe you use some LD_PRELOAD trick with a constructor function, that is invoked before main()?

[thomasknauth]

Pretty much. In our case, it is Rust code using the tokio async runtime.

[dimakuv]

For other rlimits, see these issues:

• [Encrypted FS] Use pre-allocacted free list instead of calloc/free for file nodes #1714
• Nginx performance degradation in Gramine-SGX attributed to SHA256 hashing #1712

We should also move sys.stack.size and sys.brk_size (and maybe other manifest options) under this rlimits syntax.

[thomasknauth]

Since it came up again recently, a work around is to set limits and then exec into the actual binary. If this binary now forks/spawns threads before its main function, the limits will already be set to the desired value (ht @haraldh).