Can't create child processes with node.js inside gramine

[adnanjam]

Description of the problem

I am trying to run a simple node.js program inside gramine, based on the nodejs.manifest.template.

The enclave crashes whenever I start any child process through node (which uses fork) and I am not sure what the cause is for this. Would really appreciate if someone has more knowledge what the issue may be.

Steps to reproduce

Here is my manifest file, minimal changes from the above example:

# Node.js manifest file example

loader.preload = "file:{{ gramine.libos }}"  # for compatibility with v1.0

loader.entrypoint = "file:{{ gramine.libos }}"
libos.entrypoint = "{{ nodejs_dir }}/nodejs"

loader.log_level = "all"
loader.log_file = "gramine.log"

loader.env.LD_LIBRARY_PATH = "/lib:{{ arch_libdir }}:/usr/{{ arch_libdir }}"

loader.insecure__use_cmdline_argv = true
loader.insecure__use_host_env = true

# Node.js requires eventfd2() emulation otherwise fails on `(uv_loop_init(&tracing_loop_)) == (0)'
sys.insecure__allow_eventfd = true

fs.mount.nodejs.type = "chroot"
fs.mount.nodejs.path = "{{ nodejs_dir }}/nodejs"
fs.mount.nodejs.uri = "file:{{ nodejs_dir }}/nodejs"

fs.mount.lib.type = "chroot"
fs.mount.lib.path = "/lib"
fs.mount.lib.uri = "file:{{ gramine.runtimedir() }}"

fs.mount.lib2.type = "chroot"
fs.mount.lib2.path = "{{ arch_libdir }}"
fs.mount.lib2.uri = "file:{{ arch_libdir }}"

fs.mount.lib3.type = "chroot"
fs.mount.lib3.path = "/usr/{{ arch_libdir }}"
fs.mount.lib3.uri = "file:/usr/{{ arch_libdir }}"

fs.mount.usr.type = "chroot"
fs.mount.usr.path = "/user-dir"
fs.mount.usr.uri = "file:user-dir"

sgx.nonpie_binary = true
# Node.js expects around 1.7GB of heap on startup, see https://github.com/nodejs/node/issues/13018
sgx.enclave_size = "4G"
sgx.thread_num = 64

sgx.trusted_files = [
  "file:{{ gramine.libos }}",
  "file:{{ nodejs_dir }}/nodejs",
  "file:{{ gramine.runtimedir() }}/",
  "file:{{ arch_libdir }}/",
  "file:/usr/{{ arch_libdir }}/",
  "file:main.js",
  "file:test.js",
  "file:user-dir/",
]

Here is the code I am trying to run using the command: gramine-sgx nodejs test.js:

const child_process = require('child_process');
const child =  child_process.execFile(
    "./fibonacci",
    [],
    {
        cwd: __dirname,
        env: process.env,
    }
);

child.on("message", function(m){
    console.log("Message from child: ", m);
});

child.on('error', (err) => {
    console.log(err)
}); 

fibonacci.js is just a simple function that computes fibonacci number up to n, which is hardcoded in the file.

Expected results

I expected the code to work and give me the Fibonacci number as it does when I run It on my machine or using gramine-direct nodejs test.js

Actual results

I am getting this error when I try to run the module, seems like node.js can't be spawned:

error: Failed to initialize child process: -24

Additional information

Actually, the code I am trying to run is just a sandbox example for testing child processes in nodejs, in reality I want to be able to execute a shell command using child_process.exec("pwd") for example. I tried that first and also added the /bin/sh binaries to gramine but that has the same issue as described above.

Gramine commit hash

[dimakuv]

So it looks like gramine-direct (non-SGX) version works, but gramine-sgx fails.

I suspect that your machine is running out of physical memory (RAM + swap file). Some references to better understand this issue:

• https://gramine.readthedocs.io/en/latest/manifest-syntax.html#enclave-size
• https://gramine.readthedocs.io/en/latest/devel/performance.html#multi-process-workloads
• https://gramine.readthedocs.io/en/latest/devel/performance.html#glibc-malloc-tuning

So I would like to ask you to root cause the problem a bit further:

1. Please show us your machine's physical memory size; you can paste the output of free -h here.
2. When you run gramine-sgx nodejs test.js, please copy the whole output, not just the last line (Failed to initialize...). There is probably more info in the output.
3. When you run gramine-sgx nodejs test.js, please check the Linux system log using dmesg. Does it contain something like Invoking the OOM killer ... in the last lines? This indicates that Linux decided to kill the child Gramine process due to lack of physical memory.

If it turns out that indeed your machine has too-small RAM + swapfile, and that's why Gramine can't create a child process, then you have several choices:

1. Try to decrease sgx.enclave_size -- but in case of NodeJS, I don't think you can push the size down any further
2. Move to another machine with bigger RAM
3. Buy more RAM into this machine
4. Increase the swap file (https://www.linuxtechi.com/extend-swap-space-using-swap-file-in-linux/) -- but this will be slooooow

[adnanjam]

Thanks for the quick response @dimakuv. Indeed it was an issue with resource limitation, which I already suspected since the virtual machine was also terminating the Vscode SSH server when I started the gramine container. I have added a swapfile to the VM to extend the memory available although as you predicted, it is performing really slow. The ideal solution would be to increase the memory available on the VM.