OpenMP in Enclave

[hallojs]

I would like to use OpenMP in an SGX Enclave. I have built a minimal example based on the SampleEnclave and added -fopenmp as compiler option and -lsgx_pthread and -lsgx_omp as enclave linking options in the Makefile. Unfortunately I get the following error:

/usr/bin/ld: /home/jonas/sgxsdk/lib64/libsgx_pthread.a(pthread.o): in function `_pthread_wakeup(unsigned long) [clone .part.0]':
pthread.cpp:(.text._Z15_pthread_wakeupm.part.0+0x2d): undefined reference to `pthread_wakeup_ocall'
/usr/bin/ld: /home/jonas/sgxsdk/lib64/libsgx_pthread.a(pthread.o): in function `pthread_create':
pthread.cpp:(.text.pthread_create+0x117): undefined reference to `pthread_create_ocall'
/usr/bin/ld: pthread.cpp:(.text.pthread_create+0x155): undefined reference to `pthread_wait_timeout_ocall'
/usr/bin/ld: /home/jonas/sgxsdk/lib64/libsgx_pthread.a(pthread.o): in function `pthread_join':
pthread.cpp:(.text.pthread_join+0xdf): undefined reference to `pthread_wait_timeout_ocall'
collect2: error: ld returned 1 exit status
make[1]: *** [Makefile:255: enclave.so] Error 1
make[1]: Leaving directory '/home/jonas/Repositories/omp_intel_sgx'
make: *** [Makefile:180: all] Error 2

Did I forget a necessary step to use omp in the enclave? Thank you!

What are the steps to reproduce the bug?

Add the code below to the sample enclave.

Can you reproduce the bug using the latest master?

Not tested so far.

What CPU, platform, operating system/distribution, and SGX driver are you running? The more specific, the better.

• 2x Intel(R) Xeon(R) Gold 6342
• Ubuntu 20.04 Server
• In-Kernel Driver (Self-compiled 5.11.0)

Makefile

• line 132:

Enclave_Cpp_Flags := $(Enclave_C_Flags) -nostdinc++ -fopenmp

• line 144:

Enclave_Link_Flags := $(MITIGATION_LDFLAGS) $(Enclave_Security_Link_Flags) \
    -Wl,--no-undefined -nostdlib -nodefaultlibs -nostartfiles -L$(SGX_TRUSTED_LIBRARY_PATH) \
	-Wl,--whole-archive -l$(Trts_Library_Name) -Wl,--no-whole-archive \
	-Wl,--start-group -lsgx_tstdc -lsgx_pthread -lsgx_omp -lsgx_tcxx -l$(Crypto_Library_Name) -l$(Service_Library_Name) -Wl,--end-group \
	-Wl,-Bstatic -Wl,-Bsymbolic -Wl,--no-undefined \
	-Wl,-pie,-eenclave_entry -Wl,--export-dynamic  \
	-Wl,--defsym,__ImageBase=0 -Wl,--gc-sections   \
	-Wl,--version-script=Enclave/Enclave.lds

Enclave.cpp:

...
void ecall_do_omp_stuff(int* values, size_t size){
#pragma omp parallel for
    for (size_t i = 0; i < size; i++)
        values[i] = values[i] * values[i];
}
...

App.cpp:

...
int SGX_CDECL main()
{
    /* Initialize the enclave */
    initialize_enclave();
 
    size_t size = 10;
    int* values = (int*)malloc(sizeof(int) * size);
    for(size_t i=0; i < size; ++i)
        values[i] = i;

    ecall_do_omp_stuff(global_eid, values, size);

    for (size_t i = 0; i < size; ++i)
        printf("%d\n", values[i]);

    /* Destroy the enclave */
    sgx_destroy_enclave(global_eid);
    return 0;
}
...

Enclave.edl

...
    trusted {
        public void ecall_do_omp_stuff([in, out, count=size] int* a, size_t size);
    };
...

[llly]

edl should import pthread ocalls like

linux-sgx/SampleCode/SampleDNNL/Enclave/Enclave.edl

Line 37 in e1a37bd

from "sgx_pthread.edl" import *;

[hallojs]

@llly thank you! Now it works :)

I am unsure, but maybe it would be a good idea to add this tip to the Developer Reference? On page 145 it only says:

To enable Intel® SGX OpenMP library in an enclave, follow the steps below to update the Makefile:

1. Add '-fopenmp' option to the enclave compiling options.
2. Add '-lsgx_omp' to the enclave linking options.

[lzha101]

@llly thank you! Now it works :)

I am unsure, but maybe it would be a good idea to add this tip to the Developer Reference? On page 145 it only says:

To enable Intel® SGX OpenMP library in an enclave, follow the steps below to update the Makefile:

1. Add '-fopenmp' option to the enclave compiling options.
2. Add '-lsgx_omp' to the enclave linking options.

Thanks for the advice. SGX OpenMP depends on the SGX pthread lib and requires the SGX pthread EDL file as well. We can add this dependency to the OpenMP description in the Developer Reference.