Issue with Azure & DCAP (but I propose a solution)

[enedellec]

Hi,

As a new comer, I started by reading the documentation and I quickly arrived to the Cloud Deployment page which was exactly what I wanted to do : quickly testing gramine in Azure.
I managed to run the "hello world" sample, but when I wanted to try one sample which relies on DCAP, it did not work.

To be honest, it took me a while to make it work, and I think it could be worthwhile to add dedicated information on that part in the Cloud Deployment page. I will be happy to propose a PR if you think that brings some value.

To sum up, the issue was more on the Azure side, as in Azure the context is different regarding DCAP : you are invited to use the Azure Data Center Attestation Primitives (DCAP) Client provided by Microsoft.
I just struggled (a lot) in order to find a suitable combo between what I could read in gramine documentation, in Microsoft documentation, in different forums or in other solutions like Open Enclave & al.

I finally managed to make it work, thanks to the docker image available on DockerHub, but I also wanted to have my own bash script file in order to make some quick tests, without using docker. So, I looked at the history of the docker image, in order to find the right commands to make.

I created the following script which works for me with the gramine stable version (v1.2 today), and Ubuntu 20.04 :

#!/bin/bash

set -e

##################################################################
# Install gramine
# Adapted from the docker image history : https://hub.docker.com/r/gramineproject/gramine
# and from the Cloud Deployment gramine doc : https://gramine.readthedocs.io/en/stable/cloud-deployment.html
##################################################################

sudo DEBIAN_FRONTEND=noninteractive apt-get install -y binutils

sudo curl -fsSLo /usr/share/keyrings/gramine-keyring.gpg https://packages.gramineproject.io/gramine-keyring.gpg
echo 'deb [arch=amd64 signed-by=/usr/share/keyrings/gramine-keyring.gpg] https://packages.gramineproject.io/ stable main' | sudo tee /etc/apt/sources.list.d/gramine.list

curl -fsSL https://download.01.org/intel-sgx/sgx_repo/ubuntu/intel-sgx-deb.key | sudo apt-key add -
echo 'deb [arch=amd64] https://download.01.org/intel-sgx/sgx_repo/ubuntu focal main' | sudo tee /etc/apt/sources.list.d/intel-sgx.list

sudo apt-get update
sudo apt-get install -y gramine sgx-aesm-service libsgx-aesm-launch-plugin libsgx-aesm-epid-plugin libsgx-aesm-quote-ex-plugin libsgx-aesm-ecdsa-plugin libsgx-dcap-quote-verify

sudo mkdir -p /var/run/aesmd/

# creation of the "restart_aesm.sh" file
cat > ./restart_aesm.sh << EOF
#!/bin/bash

set -e

killall -q aesm_service || true
AESM_PATH=/opt/intel/sgx-aesm-service/aesm LD_LIBRARY_PATH=/opt/intel/sgx-aesm-service/aesm exec /opt/intel/sgx-aesm-service/aesm/aesm_service --no-syslog
EOF
chmod +x ./restart_aesm.sh
sudo cp ./restart_aesm.sh /

gramine-sgx-gen-private-key

##################################################################
# ECDSA/DCAP attestation in MS Azure cloud
# Adapted from the following source : https://hub.docker.com/r/gramineproject/gramine
##################################################################

# enable Microsoft software repository
echo "deb [arch=amd64] https://packages.microsoft.com/ubuntu/20.04/prod focal main" | sudo tee /etc/apt/sources.list.d/msprod.list
wget -qO - https://packages.microsoft.com/keys/microsoft.asc | sudo apt-key add -

# install Azure DCAP library
sudo apt-get update
sudo apt-get install -y az-dcap-client

# restart the AESM service
sudo /restart_aesm.sh

And I also make a script file for testing the Secret Provisioning Minimal Examples :

#!/bin/bash

set -e

##################################################################
# Adapted from the following source : https://github.com/gramineproject/gramine/tree/master/CI-Examples/ra-tls-secret-prov
##################################################################

# the stable release was v1.2 when I wrote that script
git clone --branch v1.2 https://github.com/gramineproject/gramine.git ~/gramine
cd gramine
git switch -c test

sudo apt-get update
sudo apt-get install -y gcc make pkg-config

cd CI-Examples/ra-tls-secret-prov
make app dcap files/input.txt RA_TYPE=dcap

RA_TLS_ALLOW_DEBUG_ENCLAVE_INSECURE=1 RA_TLS_ALLOW_OUTDATED_TCB_INSECURE=1 ./secret_prov_server_dcap files/wrap-key &

# test minimal client
gramine-sgx ./secret_prov_min_client

# test feature-rich client
gramine-sgx ./secret_prov_client

# test encrypted-files client
gramine-sgx ./secret_prov_pf_client

I am not sure those scripts are optimal, specifically regarding the dependencies, but they work in my context : Azure + Ubuntu 20.04 + DCAP + gramine 1.2.

I noticed in different posts that several people would have been happy to have a kind of starter kit on that specific topic, which is not so specific in my opinion because when you start to evaluate confidential computing, you quickly need to put remote attestations in place.
Anyway, I am sharing what I did here, in order to help people who faced similar issues. And I am opened to enhance the official documentation if that makes sense.

[dimakuv]

@enedellec Thanks for the great summary!

We already have the sources for our official Dockerhub Gramine image. The sources can be found here: https://github.com/gramineproject/gramine/tree/master/packaging/docker. They include the minimal Dockerfile to install Gramine and all required dependencies. (Plus a minimal script to restart the SGX-specific aesmd service.)

In addition to these sources, we have the README for our official Dockerhub Gramine image. We talk about the MS Azure deployment in this README, under the section ECDSA/DCAP attestation in MS Azure cloud.

Finally, we also have some quick documentation on this Gramine image here: https://gramine.readthedocs.io/en/latest/container-integration.html#gramine-docker-image

I believe our sources + README already cover almost everything that you wrote.

The script for Secret Provisioning examples is good. I'm unsure though if we want to put this script somewhere in Gramine documentation, because it more or less duplicates our README for Secret Prov: https://github.com/gramineproject/gramine/blob/v1.2/CI-Examples/ra-tls-secret-prov/README.md

Given this, what do you think we (Gramine developers) should add? I guess we could add the link in this doc page to the sources of the Docker image (the Dockerfile)?

[enedellec]

Thank you @dimakuv for your prompt and detailed reply.

I agree with you that documentation already exists in the docker context, and it is a good point.

To sum up my experience, I started by the hello world program, and when I wanted to go a step further with DCAP, it did not work. It took me a while to understand why it did not work, and to find a solution, just because I tried to improve the script by myself, with the information that I found. For example, if you look at Azure key word on the website, there are only results which refers to the Cloud Deployment page, so I started to look outside of the gramine documentation. I do not know if I am representative, but that was my experience :-).

As you suggested, it does not make sense to integrate my scripts somewhere as the information already exists in the docker context. But IMHO, it could be worthwhile to mention that an official docker image exists earlier in the documentation, in the Quick Start page for example, and to invite people to look at the docker context if they want to use DCAP in the Azure context. If they want to make a bash script like me, they will have all information to make it.