Secret Provisioning servers running inside SGX enclave failed to verify client embedded SGX quote

[yanghongsing]

I am using example of "ra-tls-secret-prov". It is working fine when I run server-dcap outside SGX enclave. It failed to verify client embedded SGX quote when server_dcap ran in SGX enclave. It gives the error message below:
--- Starting the Secret Provisioning server on port 4433 ---
ra_tls_verify_callback: sgx_qv_verify_quote failed: 57369
client_connection: Secret Provisioning failed during mbedtls_ssl_handshake with error -12288
client_connection: ra_tls_verify_callback_results:
attestation_scheme=2, err_loc=3, client_connection: dcap.func_verify_quote_result=0xe019, dcap.quote_verification_result=0xa006

Above error usual happens when environment variables of RA_TLS_ALLOW_DEBUG_ENCLAVE_INSECURE=1, RA_TLS_ALLOW_OUTDATED_TCB_INSECURE=1 are not set to 1.

Below is server.manifest.template

Secret Provisioning manifest file example (server)

loader.entrypoint = "file:{{ gramine.libos }}"
libos.entrypoint = "/server"

loader.log_level = "{{ log_level }}"

loader.insecure__use_host_env = true
loader.env.LD_LIBRARY_PATH = "/lib:{{ arch_libdir }}:/usr{{ arch_libdir }}:/usr/lib"
loader.env.RA_TLS_ALLOW_DEBUG_ENCLAVE_INSECURE = "1"
loader.env.RA_TLS_ALLOW_OUTDATED_TCB_INSECURE = "1"
loader.env.RA_TLS_ALLOW_HW_CONFIG_NEEDED = "1"
loader.env.RA_TLS_ALLOW_SW_HARDENING_NEEDED = "1"
loader.env.AZDCAP_DEBUG_LOG_LEVEL = '0'

fs.mounts = [
{ path = "/lib", uri = "file:{{ gramine.runtimedir() }}" },
{ path = "{{ arch_libdir }}", uri = "file:{{ arch_libdir }}" },
{ path = "/usr{{ arch_libdir }}", uri = "file:/usr{{ arch_libdir }}" },
{ path = "/usr/lib/libdcap_quoteprov.so", uri = "file:/usr/lib/libdcap_quoteprov.so" },
{ path = "/server", uri = "file:server" },
{ path = "/ssl/", uri = "file:../ssl/" },
{ path = "/etc/hosts", uri = "file:../helper-files/hosts" },
]
sys.enable_extra_runtime_domain_names_conf = true
sgx.enclave_size = "2G"
sgx.debug = false
sgx.edmm_enable = {{ 'true' if env.get('EDMM', '0') == '1' else 'false' }}

sgx.remote_attestation = "{{ ra_type }}"
sgx.ra_client_spid = "{{ ra_client_spid }}"
sgx.ra_client_linkable = {{ 'true' if ra_client_linkable == '1' else 'false' }}

sgx.max_threads = 32

sgx.trusted_files = [
"file:{{ gramine.libos }}",
"file:server",
"file:{{ gramine.runtimedir() }}/",
"file:{{ arch_libdir }}/",
"file:/usr{{ arch_libdir }}/",
"file:/usr/lib/libdcap_quoteprov.so",
"file:../ssl/",
"file:../helper-files/",
]

Makefile
ARCH_LIBDIR ?= /lib/$(shell $(CC) -dumpmachine)

ifeq ($(DEBUG),1)
GRAMINE_LOG_LEVEL = debug
CFLAGS += -O0 -ggdb3
else
GRAMINE_LOG_LEVEL = error
CFLAGS += -O2
endif

CFLAGS += -fPIE
LDFLAGS += -pie

RA_TYPE ?= dcap
RA_CLIENT_SPID ?=
RA_CLIENT_LINKABLE ?= 0

.PHONY: cs
cs: ssl/server.crt server/server secret_prov/client
secret_prov/client.manifest.sgx secret_prov/client.sig
server/server.manifest.sgx server/server.sig

############################# SSL DATA DEPENDENCY #############################

SSL data: key and x.509 self-signed certificate

ssl/server.crt: ssl/ca_config.conf
openssl genrsa -out ssl/ca.key 2048
openssl req -x509 -new -nodes -key ssl/ca.key -sha256 -days 1024 -out ssl/ca.crt -config ssl/ca_config.conf
openssl genrsa -out ssl/server.key 2048
openssl req -new -key ssl/server.key -out ssl/server.csr -config ssl/ca_config.conf
openssl x509 -req -days 360 -in ssl/server.csr -CA ssl/ca.crt -CAkey ssl/ca.key -CAcreateserial -out ssl/server.crt

######################### CLIENT/SERVER EXECUTABLES ###########################

CFLAGS += -Wall -std=c11 $(shell pkg-config --cflags secret_prov_gramine)
LDFLAGS += -Wl,--enable-new-dtags $(shell pkg-config --libs secret_prov_gramine)

%/server_epid: %/server.c
$(CC) $< $(CFLAGS) $(LDFLAGS) -lsecret_prov_verify_epid -pthread -o $@

linker option --no-as-needed is required because SGX DCAP library (libsgx_dcap_quoteverify.so)

does dlopen() instead of directly linking against libsgx_urts.so, and without this option

compilers remove the "seemingly unused" libsgx_urts.so

secret_prov/client: secret_prov/client.c
$(CC) $< $(CFLAGS) $(LDFLAGS) -lsecret_prov_attest -o $@
server/server: server/server.c
$(CC) $< $(CFLAGS) $(LDFLAGS) -Wl,--no-as-needed -lsgx_urts -lsecret_prov_verify_dcap -pthread -o $@

############################### SERVER MANIFEST ###############################

server/server.manifest: server/server.manifest.template
cd server &&
gramine-manifest
-Dlog_level=$(GRAMINE_LOG_LEVEL)
-Darch_libdir=$(ARCH_LIBDIR)
-Dra_type=$(RA_TYPE)
-Dra_client_spid=$(RA_CLIENT_SPID)
-Dra_client_linkable=$(RA_CLIENT_LINKABLE)
$(notdir $<) > $(notdir $@)

server/server.manifest.sgx server/server.sig: sgx_sign_secret_prov_server
@:

.INTERMEDIATE: sgx_sign_secret_prov_server
sgx_sign_secret_prov_server: server/server.manifest server/server
cd server &&
gramine-sgx-sign
--manifest $(notdir $<)
--output $(notdir $<.sgx)
############################### CLIENT MANIFEST ###############################

secret_prov/client.manifest: secret_prov/client.manifest.template
cd secret_prov &&
gramine-manifest
-Dlog_level=$(GRAMINE_LOG_LEVEL)
-Darch_libdir=$(ARCH_LIBDIR)
-Dra_type=$(RA_TYPE)
-Dra_client_spid=$(RA_CLIENT_SPID)
-Dra_client_linkable=$(RA_CLIENT_LINKABLE)
$(notdir $<) > $(notdir $@)

secret_prov/client.manifest.sgx secret_prov/client.sig: sgx_sign_secret_prov_client
@:

.INTERMEDIATE: sgx_sign_secret_prov_client
sgx_sign_secret_prov_client: secret_prov/client.manifest secret_prov/client
cd secret_prov &&
gramine-sgx-sign
--manifest $(notdir $<)
--output $(notdir $<.sgx)

################################## CLEANUP ####################################

.PHONY: clean
clean:
$(RM) OUTPUT *.token *.sig .manifest.sgx .manifest src/dcap
cd secret_prov_minimal; $(RM) client server *.token *.sig *.manifest.sgx .manifest
cd secret_prov; $(RM) client server_ *.token *.sig *.manifest.sgx .manifest
cd secret_prov_pf; $(RM) client server_ *.token *.sig *.manifest.sgx .manifest
cd server; $(RM) server_ *.token *.sig *.manifest.sgx *.manifest

.PHONY: distclean
distclean: clean
$(RM) -r secret_prov_pf/wrap_key secret_prov_pf/enc_files/input.txt ssl/ca.* ssl/server.*

Steps to reproduce:

1. Create a directory of "server" under the directory of "ra-tls-secret-prov"
2. Copy "server.manifest.template" to "server"
3. Copy server.c from "secret_prov" to "server"
4. Copy Makefile to the directory of "ra-tls-secret-prov"
5. Run "make cs" under "ra-tls-secret-prov"
6. Run "gramine-sgx ./server &" under "server"
7. Run "gramine-sgx ./client" under "secret_prov"

[kailun-qin]

@yanghongsing Thanks for the report!

attestation_scheme=2, err_loc=3, client_connection: dcap.func_verify_quote_result=0xe019, dcap.quote_verification_result=0xa006

0xe019 indicated a SGX_QL_NETWORK_ERROR (pls see https://github.com/intel/SGXDataCenterAttestationPrimitives/blob/6882afad8644c27db162b40994402c8ad2a7fb32/QuoteGeneration/quote_wrapper/common/inc/sgx_ql_lib_common.h#L76).

It is working fine when I run server-dcap outside SGX enclave. It failed to verify client embedded SGX quote when server_dcap ran in SGX enclave.

For server-dcap (i.e. quote verification) to run inside SGX enclave, you'll need to mount and allow /etc/sgx_default_qcnl.conf in your server-dcap manifest (pls see the client manifest of our ra-tls-mbedtls example as a reference).

[yanghongsing]

@kailun-qin Thank you for quick response.
It seems that allowing /etc/sgx_default_qcnl.conf take no effect, instead of allowing /etc/ssl/certs/ca-certificates.crt solves this problem.
I would like to close this issue.