ra-tls-secret-prov/secret_prov_pf on Kubernetes not decrypting file (no errors apparently)

[tiagorvmartins]

Hello, I am trying to run the example ra-tls-secret-prov/secret_prov_pf on a kubernetes environment using volumes with encrypted mounts on gsc manifest - sorry if its the wrong place to make this question.

The client and the server were ported to a docker image, to make it easier to run on kubernetes.
The client image was graminized with the following manifest:

sys.enable_sigterm_injection = true

sgx.remote_attestation = "dcap"

sys.stack.size = "16M"
sys.enable_extra_runtime_domain_names_conf = true
sys.experimental__enable_flock = true
sgx.nonpie_binary = true
sgx.enclave_size = "512M"
sgx.edmm_enable = false
sgx.max_threads = 32

fs.mounts = [
  { path = "/files/", uri = "file:/files/",  type = "encrypted" }
]

sgx.trusted_files = [
  "file:/app/client",
  "file:/etc/host.conf",
  "file:/etc/hosts",
  "file:/etc/nsswitch.conf",
  "file:/etc/resolv.conf",
  "file:/serv_prov/ssl/ca.crt"
]

loader.env.TZ = ":/etc/localtime"
loader.env.SGX_AESM_ADDR="1"
loader.env.LD_PRELOAD = "libsecret_prov_attest.so"
loader.env.SECRET_PROVISION_CONSTRUCTOR = "1"
loader.env.SECRET_PROVISION_SET_KEY = "default"
loader.env.SECRET_PROVISION_CA_CHAIN_PATH = "/serv_prov/ssl/ca.crt"
loader.env.SECRET_PROVISION_SERVERS = "secretprovisioning:4433"

Here is the kubernetes yml file, where I encrypt the file (on a init container called encrypt-data) using the volume and then use the same encrypted file on the main container called client - attaching the same volume.

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: sgx-client
  namespace: testing
  labels:
    app: sgx-client
spec:
  serviceName: sgx-client
  replicas: 1
  revisionHistoryLimit: 1
  selector:
    matchLabels:
      app: sgx-client
  template:
    metadata:
      labels:
        app: sgx-client
    spec:
      containers:
      - name: client
        resources:
          limits:
            sgx.intel.com/epc: "1Mi"
        image: tiagorvmartins/sgx-client-test
        imagePullPolicy: Always
        volumeMounts:
        - name: data
          mountPath: /files/        
        - name: aesmd
          mountPath: /var/run/aesmd        
      initContainers:
      - name: encrypt-data
        image: gramineproject/gramine
        command: ["/bin/sh"]
        args:
        - "-c"
        - |
          if [ -d /files/input.txt ]
          then
            echo 'Data already encrypted.'
            exit 0;
          else            
            echo 'Hi! this is a test!' > /files/input.txt
            echo 'Cat on file content:'
            cat /files/input.txt            
            echo -n $ENCRYPT_KEY | base64 -d > wrap_key            
            gramine-sgx-pf-crypt encrypt -w wrap_key -i /files/input.txt -o /files/input.txt            
            exit 0;
          fi;          
        env:
        - name: ENCRYPT_KEY
          valueFrom:
            secretKeyRef:
              name: secretprovisioning
              key: encrypt_key
        volumeMounts:
        - name: data
          mountPath: /files/
      volumes:  
        - name: aesmd
          hostPath:
            path: /var/run/aesmd 
      nodeSelector:
        kubernetes.azure.com/agentpool: testing
  volumeClaimTemplates:
  - metadata:
      name: data
    spec:
      accessModes: [ "ReadWriteOnce" ]
      storageClassName: default
      resources:
        requests:
          storage: 500Gi

And here are the logs of the server that provides the secret:

--- Reading the master key for encrypted files from '/server/wrap_key' ---
--- Starting the Secret Provisioning server on port 4433 ---Azure Quote Provider: libdcap_quoteprov.so [ERROR]: Could not retrieve environment variable for 'AZDCAP_DEBUG_LOG_LEVEL'
WARNING: The collateral is out of date.
Received the following measurements from the client:  - MRENCLAVE:   082f2c9dafa3c2833119c6719b46d863046624ee409ab15d648176f213e88d80
  - MRSIGNER:    3d83d1e4a46aeaa9c0958026cdb402628e026fd6c08a0bb57c98445a423cc22a
  - ISV_PROD_ID: 0
  - ISV_SVN:     0
[ WARNING: In reality, you would want to compare against expected values! ]

And here is the logs of the client container:

Gramine is starting. Parsing TOML manifest file, this may take some time...
-----------------------------------------------------------------------------------------------------------------------
Gramine detected the following insecure configurations:

  - loader.insecure__use_cmdline_argv = true   (forwarding command-line args from untrusted host to the app)
  - sys.experimental__enable_flock = true      (flock syscall is enabled; still under development and may contain bugs)

Gramine will continue application execution, but this configuration must not be used in production!
-----------------------------------------------------------------------------------------------------------------------

--- [main] Re-starting myself using execvp() ---
--- [child] Read from protected file: 'GRAFS_PF' ---
--- [parent] Read from protected file: 'GRAFS_PF' ---

Since it shows 'GRAFS_PF' its not actually decrypting the file, it is instead the encrypted file itself...
I thought this was a transparent operation, since the manifest marked this folder as encrypted.
I was expecting to read: "Hi! this is a test!"

I guess the problem is related to how kubernetes handles this kind of volumes.
Any thoughts on what I might be doing wrong?
Let me know if you need any extra information.

Thank you in advance!

[dimakuv]

@tiagorvmartins Looks like Gramine for some reason doesn't recognize the /files/ directory (the attached-by-Kubernetes mount) as the encrypted directory. Even though you explicitly specified it in the manifest file.

Please add loader.log_level = "all" in the manifest and re-run and show us the Gramine log. This will contain enough information to debug the issue further.

[tiagorvmartins]

Hi @dimakuv thank you for your reply.
Does this raises any hint on what might be happening?
Logs here
Thank you.

[dimakuv]

Snippet from the shared log:

(libos_fs_encrypted.c:141:cb_debug) [P2:T2:client] debug: ipf_open: handle: 0, path: '/files/input.txt', real size: 12288, mode: 0x3
(libos_fs_encrypted.c:141:cb_debug) [P2:T2:client] debug: ipf_init_existing_file: data size 4096
(libos_fs_encrypted.c:141:cb_debug) [P2:T2:client] debug: ipf_open: OK (data size 4096)
(libos_fs_encrypted.c:141:cb_debug) [P2:T2:client] debug: ipf_internal_flush: no need to write
(libos_fs_encrypted.c:141:cb_debug) [P2:T2:client] debug: ipf_open: handle: 0, path: '/files/input.txt', real size: 12288, mode: 0x3
(libos_fs_encrypted.c:141:cb_debug) [P2:T2:client] debug: ipf_init_existing_file: data size 4096
(libos_fs_encrypted.c:141:cb_debug) [P2:T2:client] debug: ipf_open: OK (data size 4096)
(libos_parser.c:1653:buf_write_all) [P2:T2:client] trace: ---- openat(AT_FDCWD, "/files/input.txt", O_RDONLY, 0000) = 0x3

This indicates that Gramine correctly understands that /files/input.txt is indeed an encrypted files and decrypts it with a key (that was provisioned before correctly, which can also be seen from the log, see connect() lines).

So I think the problem is that you got double-encryption of the file /files/input.txt. Two reasons I can imagine:

1. Your Kubernetes startup script is executed twice, or some steps in the Kubernetes script are executed twice (including the encryption of the file).
2. The line gramine-sgx-pf-crypt encrypt -w wrap_key -i /files/input.txt -o /files/input.txt is suspicious to me. I'm not sure our tool supports in-place encryption (when it writes to the same file it opened). This may be a bug in our tool. Please replace with smth like this:

gramine-sgx-pf-crypt encrypt -w wrap_key -i /files/input.txt -o /files/enc_input.txt
mv /files/enc_input.txt /files/input.txt

[tiagorvmartins]

Hi @dimakuv your proposal resulted in this error, because I believe my client is expecting to decrypt /files/input.txt and doing "-o /files/enc_input.txt" will generate different metadata file, as you explained in one of the previous discussions I opened.

(libos_parser.c:1653:buf_write_all) [P2:T2:client] trace: ---- set_robust_list(0x1d500a20, 0x18) = 0x0
(libos_parser.c:1653:buf_write_all) [P1:T1:client] trace: ---- return from clone(...) = 0x2
(libos_parser.c:1653:buf_write_all) [P1:T1:client] trace: ---- wait4(-1, 0, 0, 0) ...
(libos_fs_encrypted.c:141:cb_debug) [P2:T2:client] debug: ipf_open: handle: 0, path: '/files/input.txt', real size: 4096, mode: 0x3
(libos_fs_encrypted.c:141:cb_debug) [P2:T2:client] debug: ipf_init_existing_file: data size 19
(libos_fs_encrypted.c:141:cb_debug) [P2:T2:client] debug: ipf_open: failed: -8
(libos_fs_encrypted.c:203:encrypted_file_internal_open) [P2:T2:client] warning: pf_open failed: Invalid path
(libos_parser.c:1653:buf_write_all) [P2:T2:client] trace: ---- openat(AT_FDCWD, "/files/input.txt", O_RDONLY, 0000) = -13
(libos_parser.c:1653:buf_write_all) [P2:T2:client] trace: ---- write(2, 0x1d64b058, 0x27) ...
[error] cannot open '/files/input.txt'
(libos_parser.c:1653:buf_write_all) [P2:T2:client] trace: ---- return from write(...) = 0x27
(libos_parser.c:1653:buf_write_all) [P2:T2:client] trace: ---- write(2, 0x1d64b140, 0x28) ...
child could not read the protected file

However your suspicious were right, once I adapted the generation of the encrypted file to instead the following:

echo 'Hi! this is a test' > /files/plain_input.txt
gramine-sgx-pf-crypt encrypt -w wrap_key -i /files/plain_input.txt -o /files/input.txt

I can see now the following output:

--- [child] Read from protected file: 'Hi! this is a test'
--- [parent] Read from protected file: 'Hi! this is a test'

Thank you for the help understanding the problem, been a few days on this without any clues.
Apparently input and output to the same path is a problem when using gramine-sgx-pf-crypt
We can close this discussion,
Thanks again!

[dimakuv]

Hi @dimakuv your proposal resulted in this error, because I believe my client is expecting to decrypt /files/input.txt and doing "-o /files/enc_input.txt" will generate different metadata file, as you explained in one of the previous discussions I opened.

Yes, you're correct! My mistake. But your new solution is perfect.

Apparently input and output to the same path is a problem when using gramine-sgx-pf-crypt

We really need to add a new command-line option to gramine-sgx-pf-crypt to allow specifying arbitrary filenames, so that deployers have easier time. I added it in my TODO list.