DCAP attestation feature design

[ameba23]

As part of our tofino design, the entropy threshold signature server is going to be run in a trusted execution environment. This is a proposed feature for attestation that validators have correctly set up a trusted domain and are running our release VM image with the current release of the entropy-tss binary,

This comes from my notes which were originally going to be incorporated into the tofino-spec. But since it has become a bit implementation-oriented i am putting it here as a feature design issue. Once we are happy with it i'll make a PR to the spec repo with a more high-level description.

pallet-attestation

The attestation pallet will be responsible for requesting attestations via the propagation pallet, and verifying the resulting quotes submitted by TS servers using the attest extrinsic.

Quote verification will involve:

1. Verifying the ECDSA P-256 signature.
2. Checking the attestation key with the associated PCK (Provisioning Certification Key) certificate. The PCK needs to be submitted by the validator when they join, and will be handled by the staking extension pallet. The x509-parser crate might be useful here.
3. Checking the build-time measurement register (MRTD) value matches the current release of the threshold server virtual machine image. I think this value should be hard-coded into the chain runtime and so that it can only be updated with a runtime update.
4. Checking the REPORTDATA for the block number associated with the attestation request and additional details described below. In the case that this data structure exceeds 64 bytes it can be hashed, and the hashes compared. This data structure will need to live in entropy-shared as is will be needed by both the attestation pallet and the threshold server.

If verification fails, the validator should be removed from the validator pool on the next session via the staking extension. If they are a current member of the signing committee, a reshare should be initiated to exclude them.

Since quote verification happens in a pallet, we probably cant use one of intel's crates with bindings to the C quote verification library (eg: intel-tee-quote-verification-sys). This means we will probably have to use a pure rust library. I've put together tdx-quote for this. The quote format is well documented in appendix 3 of the Intel TDX DCAP Quoting Library API reference.

As for when or how often to do on-chain attestation - i am not sure. But it should definitely be triggered when a new validator initially calls the staking extension's validate extrinsic, and the account ID and x25519 public key of the new validator should not be stored until the quote is successfully verified. At the point of initiating network jump start would also be a good idea, as well as before a key reshare.

The threshold server /attest endpoint

Threshold servers will have an /attest http endpoint which is called by the propagation pallet. The request body will contain the current block number, a random nonce and/or other contextual data.

The handler for the /attest endpoint will:

• Prepare the input data for the quote.
• Create the TD quote using filesystem operations as described below.
• Submit an attest extrinsic to the attestation pallet with the quote, which will be around 700 bytes.

Because of the cost and complexity of setting up TDX, we want to be able to run a test network without it, but still be able to test the code which requests and validates quotes from threshold servers. When running the the threshold server in test mode the process of generating the TD quote will be mocked. Similarly when the chain is running with a development chainspec, we will verify mock TD quotes.

Attestation and quote verification could potentially also be done by other threshold servers - but for now i propose to keep it simple and have only on-chain attestation.

Creating the TD quote

Creating the quote itself can be done using the linux configfs-tsm filesystem interface. This exists since linux 6.7 and provides a vendor-agnostic way to get attestations for confidential computing.

This lets us create quotes by reading and writing files in /sys/kernel/config/tsm. For example a report can generated with:

mkdir -p /sys/kernel/config/tsm/report/testreport0
cp my-64-byte-report-input-data /sys/kernel/config/tsm/report/testreport0/inblob

The report output data (the TD quote) can be read by reading the file /sys/kernel/config/tsm/report/testreport0/outblob. There are a few related attributes but basically that's it.

Google's go-configfs-tsm shows an example of how to do this in Go, and writing something similar in rust should not be too complicated.

Update: It might be better to use intel's tdx-guest crate for this. This would be more direct but personally i find the documentation for it pretty confusing.

Secure channel establishment

For performance reasons, we probably don't want to do attestation at the point of signing. But if we can be sure that the x25519 public key of the threshold server was generated in the trust domain, we know that the signature request and signing protocol session messages can only be decrypted by the trust domain.

A way to do this is described in section 9.3 'Secure Channel Establishment' of Intel TDX Demystified which in turn quotes Integrating Remote Attestation with Transport Layer Security section 3.1 'Binding Key to Enclave'.

When creating a TDREPORT, which is the basis of the quote, a user-provided 64-Byte REPORTDATA is used as input and included in the quote and covered by the quote signature. By including the account ID and public x25515 key of the threshold server in this field, we can check them when verifying the quote, and be sure they came from the trusted domain.

Since protocol sessions are also encrypted / authenticated using these keys, we can be sure that protocol sessions are communicating with the trusted domain.

The great thing about this is it is reduces how many rounds of communication we need between the chain and threshold server when running protocols. If we want to do attestation before running the network DKG or reshare protocols, we would normally need that the chain requests a quote using the propagation pallet to call an threshold server endpoint, then receives a response as an extrinsic, and then initiates the protocol by calling another HTTP endpoint, and finally after the protocol the threshold server submits an extrinsic to confirm. If we can attest beforehand that the threshold server's details come from the trusted domain, i think we don't need these extra steps.

[ameba23]

Stuff from review of #1003 :

• In the TS server's /attest endpoint handler, check that the OcwMessageAttestationRequest is correct by making a chain query: https://github.com/entropyxyz/entropy-core/pull/1003/files#r1723953705 No longer applies as offchain worker no longer used
• In the attestation pallet's AttestationRequests storage map, account ids should be represented as T::AccountId rather than Vec<u8>. Doing this may also mean making OcwMessageAttestationRequest generic for chain config. https://github.com/entropyxyz/entropy-core/pull/1003/files#r1723987989
• Add a mechanism by which TS servers can be punished if they do not respond to an attestation request within a certain amount of time. See comment for Jesse's proposal for this: https://github.com/entropyxyz/entropy-core/pull/1003/files#r1723960860
• Store block number when last successful attestation was made for each TS server. (Jesse suggested this, i can't find the comment though). I think whether it makes sense to do this depends on when/how often we initiate attestations requests, which is still an open design decision.

Generally still to do following that PR:

• Implement getting quotes in production (currently only mock quotes are made).
• Handle PCK certificates, and check the public key with them during quote verification.
• Verify build or run time measurement details during quote verification.
• Initiate attestation requests, eg: on validator joining, or joining the signing committee, or just regularly.
• When an attestation fails, or an attestation request is not responded to, do something about it (could be part of slashing feature).
• Possibly change how TS server mnemonics are handled - see TSS account mnemonics and TDX attestation #1015

[ameba23]

Revisiting this following progress made, this is what i still think we need:

• Generate mnemonic internally in entropy-tss #1128 to be merged. This includes a way for the TSS node operator to get its account id and x25519 public key when they start the node, which they will need for the validate extrinsic and to be able to fund it's account.

• A TSS server HTTP route which the node operater can call to retrieve a quote from the TSS, in order to use it in the validate, change_endpoint or change_tss_accounts extrinsics. The handler should internally submit a request_attestation extrinsic from the attestation pallet to retrieve a nonce for the quote, and return the quote in the response body.

• Clarity on encryption / integrity protection of the TDX guest storage volume as this effects a lot of the assumptions on which the design of this feature is based

• Clarity on whether MRTD values can really give us guarantees that our release image is being run unmodified (there was some confusion about this in the last call we had with Poetic). This is tracked in Ensuring the MRTD register value corresponds to an enclave running the entropy-tss binary tdx-build-system#6

There are other things related to devops - this is only stuff related to this feature in entropy-core.