Running OpenMP inside Gramine Shielded Containers

[fwoodruff]

I have a C++ microservice I am looking to run inside an SGX enclave. I am able to produce a Docker image that works fine outside Gramine. Using GSC, I am able to sign and build my executable however I am getting the runtime error:

OMP: Error #179: Function Can't open SHM2 failed:
OMP: System error #2: No such file or directory

I believe this error occurs because the microservice relies on OpenMP; the C++ makefile contains the -lomp linker flag.

I have tried installing libgomp inside the docker file.

sudo apt-get install -y libgmp-dev libmpfr-dev libmpc-dev libisl-dev

This has no effect on the error.

This looked interesting so I tried changing the -lomp linker flag to -lsgx_omp which Make would no longer compile so this didn't seem the right route.

I tried installing the binary at
https://download.01.org/intel-sgx/sgx-linux/2.23/distro/ubuntu22.04-server/sgx_linux_x64_sdk_2.23.100.2.bin inside the container, without much luck there either.

How do I run a program that relies on OpenMP inside a GSC 'Graminised' Docker container?

[kailun-qin]

@fwoodruff Thanks for the question!

OMP: Error #179: Function Can't open SHM2 failed:
OMP: System error #2: No such file or directory

For this POSIX shared memory to work, I think you may want to try mounting /dev/shm (e.g. fs.mounts = [ ... { type = "untrusted_shm", path = "/dev/shm", uri = "dev:/dev/shm" }, ...]) and allowing specific files in it (e.g. sgx.allowed_files = [ "dev:/dev/shm/(xxx)", ]) in your (GSC) Gramine manifest -- pls see https://gramine.readthedocs.io/en/stable/manifest-syntax.html#untrusted-shared-memory for details.

[dimakuv]

@fwoodruff This is a known issue in some versions of libomp (the LLVM implementation of OpenMP), and I think also in libiomp (Intel implementation of OpenMP).

You can read about this bug here:

• llvm/llvm-project@dafebd5
• llvm/llvm-project@102d864

IIUC, it was completely fixed in LLVM OpenMP v18.1.0. I don't know whether and how it was fixed in libiomp.

I would dissuade you from trying @kailun-qin workaround of enabling untrusted_shm. Instead, you can do this simple workaround:

fs.mounts = [
    ...,
    { type = "tmpfs", path = "/dev/shm" },
]

This way you are guaranteed that whatever files the OpenMP library creates in /dev/shm/, these files will not be visible on the host. So security of your application remains intact.

intel/linux-sgx#826 looked interesting so I tried changing the -lomp linker flag to -lsgx_omp which Make would no longer compile so this didn't seem the right route.

Gramine doesn't share any code/logic/tricks with Intel SGX SDK. So please don't try to use Intel SGX SDK tricks with Gramine.

I believe this error occurs because the microservice relies on OpenMP; the C++ makefile contains the -lomp linker flag.

I believe your microservice is built using the LLVM stack? Or your base OS in the Docker image uses the LLVM stack? In this case, -lomp leads to using libomp instead of a more traditional libgomp (yeah, welcome to the hell of at least three popular OpenMP implementations with names differing in one letter).

[fwoodruff]

I think I tried the above. I am running:
./gsc build --no-cache -b debug image-name path/to/gramine.manifest
My error message ends with:

(libos_parser.c:1658:buf_write_all) [P1:T1:zkProver] trace: ---- openat(AT_FDCWD, "glibc-hwcaps/x86-64-v2/libmemkind.so", O_RDONLY|0x80000, 0000) = -2
(libos_parser.c:1658:buf_write_all) [P1:T1:zkProver] trace: ---- openat(AT_FDCWD, "libmemkind.so", O_RDONLY|0x80000, 0000) = -2
(libos_parser.c:1658:buf_write_all) [P1:T1:zkProver] trace: ---- openat(AT_FDCWD, "/gramine/meson_build_output/lib/libmemkind.so", O_RDONLY|0x80000, 0000) = -2
(libos_parser.c:1658:buf_write_all) [P1:T1:zkProver] trace: ---- openat(AT_FDCWD, "/gramine/meson_build_output/lib/x86_64-linux-gnu/gramine/runtime/glibc/libmemkind.so", O_RDONLY|0x80000, 0000) = -2
(libos_parser.c:1658:buf_write_all) [P1:T1:zkProver] trace: ---- getpid() = 0x1
(libos_parser.c:1658:buf_write_all) [P1:T1:zkProver] trace: ---- getuid() = 0x0
(libos_parser.c:1658:buf_write_all) [P1:T1:zkProver] trace: ---- gettimeofday(0xdba4e9d0, 0) = 0x0
(libos_parser.c:1658:buf_write_all) [P1:T1:zkProver] trace: ---- gettimeofday(0xdba4e9d0, 0) = 0x0
(libos_parser.c:1658:buf_write_all) [P1:T1:zkProver] trace: ---- gettimeofday(0xdba4e9e8, 0) = 0x0
(libos_parser.c:1658:buf_write_all) [P1:T1:zkProver] trace: ---- openat(AT_FDCWD, "/dev/shm/__KMP_REGISTERED_LIB_1_0", O_RDWR|O_CREAT|O_EXCL|0xa0000, 0666) = -2
OMP: Error #179: Function Can't open SHM2 failed:
OMP: System error #2: No such file or directory
(libos_parser.c:1658:buf_write_all) [P1:T1:zkProver] trace: ---- write(2, 0xdba4e4a0, 0x62) ...
(libos_parser.c:1658:buf_write_all) [P1:T1:zkProver] trace: ---- return from write(...) = 0x62
(libos_parser.c:1658:buf_write_all) [P1:T1:zkProver] trace: ---- sched_yield() = 0x0
(libos_parser.c:1658:buf_write_all) [P1:T1:zkProver] trace: ---- getpid() = 0x1
(libos_parser.c:1658:buf_write_all) [P1:T1:zkProver] trace: ---- getuid() = 0x0
(libos_parser.c:1658:buf_write_all) [P1:T1:zkProver] trace: ---- openat(AT_FDCWD, "/dev/shm/__KMP_REGISTERED_LIB_1_0", O_RDONLY|0xa0000, 0000) = -2
(libos_parser.c:1658:buf_write_all) [P1:T1:zkProver] trace: ---- rt_sigprocmask(UNBLOCK, [SIGABRT,], NULL, 0x8) = 0x0
(libos_parser.c:1658:buf_write_all) [P1:T1:zkProver] trace: ---- gettid() = 0x1
(libos_parser.c:1658:buf_write_all) [P1:T1:zkProver] trace: ---- getpid() = 0x1
(libos_parser.c:1658:buf_write_all) [P1:T1:zkProver] trace: ---- tgkill(1, 1, [SIGABRT]) = 0x0
(libos_signal.c:58:sighandler_kill) [P1:T1:zkProver] debug: killed by signal 6
(libos_fs_lock.c:785:file_lock_clear_pid) [P1:T1:zkProver] debug: clearing file (POSIX) locks for pid 1
(libos_sync_client.c:331:shutdown_sync_client) [P1:T1:zkProver] debug: sync client shutdown: closing handles
(libos_sync_client.c:346:shutdown_sync_client) [P1:T1:zkProver] debug: sync client shutdown: waiting for confirmation
(libos_sync_client.c:359:shutdown_sync_client) [P1:T1:zkProver] debug: sync client shutdown: finished
(libos_ipc_worker.c:285:ipc_worker_main) [P1:libos] debug: IPC worker: exiting worker thread
(libos_exit.c:58:libos_clean_and_exit) [P1:T1:zkProver] debug: process 1 exited with status 134
(pal_process.c:248:_PalProcessExit) debug: PalProcessExit: Returning exit code 134

and my manifest file looks like:

sys.enable_sigterm_injection = true
sys.experimental__enable_flock = true

sys.stack.size = "16M"
sys.enable_extra_runtime_domain_names_conf = true
sgx.nonpie_binary = true
sgx.enclave_size = "4G"
sgx.edmm_enable = false
sgx.max_threads = 32

sys.insecure__allowed_eventfd = true

sgx.trusted_files = [
  "file:/etc/host.conf",
  "file:/etc/hosts",
  "file:/etc/nsswitch.conf",
  "file:/etc/resolv.conf",
  "file:/entrypoint.sh",
  "file:/config/",
]

fs.mount = [
  { type = "tmpfs", path = "/dev/shm" },
  { type = "tmpfs", path = "/tmp" },
]

sgx.allowed_files = [
  "dev:/dev/shm/__KMP_REGISTERED_LIB_1_0",
  "file:/tmp",
  "dev:/dev/shm"
]

I also tried a few variants on the allowed files, changing 'dev' to 'file' and allowing whole folders rather than just the __KMP device.

What should I try next? The non-Graminised image builds and runs fine.

[dimakuv]

You have a typo here :(

fs.mount

It should be fs.mounts, with an s at the end (i.e. in plural form). That's why GSC doesn't use the tmpfs mounts.

[dimakuv]

Btw, we are currently working on a tool to detect such typos: #1726