Azure Key Vault signatures fail verification

[vdamle]

Bug Report

• github.com/Azure/azure-sdk-for-go/services/keyvault/v7.0/keyvault
• v39.0.0+incompatible
• go version go1.16.15 linux/amd64

Problem description

This issue is being opened at the request of Azure Keyvault Support team as a follow up to a Sev 1 ticket opened against Azure Key Vault 2204280030001993. We use azure-sdk-for-go to send Sign requests to Azure keyvault to sign Ethereum transactions with Keys of Curve EC and type P-256K. Most of the signed payloads that are returned by Azure Keyvault can be verified and the public key extracted from the signature matches the public key of the keypair. However, there are a small number of requests that fail signature verification. Note that Azure KeyVault returns a signed payload in these cases (no error is returned by Azure keyvault), but when the public key is extracted from the signature, it does not match the public key of the keypair.

As part of the investigation for the ticket referenced above by Azure Key Vault support where the engineer helped analyze logs on the Key Vault side when such failures are encountered. He has arrived at the conclusion that he is unable to see such Sign() requests being processed by Azure KeyVault (request never arrived at Key Vault). It is quite surprising to me how this is possible when:

• Communication is occuring over a TLS session (backed by TCP)
• Azure Key Vault is returning a response to the Sign() request! If, as observed by Mr. Odom, the requests are not being received and processed by Azure Key Vault, I am at a loss to understand who is returning these responses to the Sign() requests.
• In order to eliminate any external aspects, we reproduced the issue to Azure support by running the client which uses the SDK in Azure cloud. It was initially suggested that the problem may be occurring because the client was running outside Azure cloud (which again, appears unlikely on the face of it).

Azure Keyvault support has come to the conclusion that the issue is in the Azure GO SDK, which is causing requests to get LOST (I'd be really interested to find out who is returning the responses, though!). Azure Key Vault support team also indicated that there is NO logging on the Key Vault side that can identify the base-64 encoded payload that is part of the KeySignParameters. We provided Azure support the sign requests, timestamps and the base-64 encoded payload string that is sent by the SDK to Azure Key Vault with the hope that they would be able to correlate it to request logs they see on their end.

We have a docker image with the client program that can launch a bunch of Sign() requests to any key vault and can demonstrat and log the failures to verify the signature. We would like to request you to reach out to Mr. Odom so he can provide further details of the investigation they conducted, so that you can investigate the bug in Azure Go SDK.

[ghost]

Thank you for your feedback. This has been routed to the support team for assistance.

[ghost]

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @RandalliLama, @schaabs, @jlichwa.

Issue Details

---

Bug Report

• github.com/Azure/azure-sdk-for-go/services/keyvault/v7.0/keyvault
• v39.0.0+incompatible
• go version go1.16.15 linux/amd64

Problem description

This issue is being opened at the request of Azure Keyvault Support team as a follow up to a Sev 1 ticket opened against Azure Key Vault 2204280030001993. We use azure-sdk-for-go to send Sign requests to Azure keyvault to sign Ethereum transactions with Keys of Curve EC and type P-256K. Most of the signed payloads that are returned by Azure Keyvault can be verified and the public key extracted from the signature matches the public key of the keypair. However, there are a small number of requests that fail signature verification. Note that Azure KeyVault returns a signed payload in these cases (no error is returned by Azure keyvault), but when the public key is extracted from the signature, it does not match the public key of the keypair.

As part of the investigation for the ticket referenced above by Azure Key Vault support where the engineer helped analyze logs on the Key Vault side when such failures are encountered. He has arrived at the conclusion that he is unable to see such Sign() requests being processed by Azure KeyVault (request never arrived at Key Vault). It is quite surprising to me how this is possible when:

• Communication is occuring over a TLS session (backed by TCP)
• Azure Key Vault is returning a response to the Sign() request! If, as observed by Mr. Odom, the requests are not being received and processed by Azure Key Vault, I am at a loss to understand who is returning these responses to the Sign() requests.
• In order to eliminate any external aspects, we reproduced the issue to Azure support by running the client which uses the SDK in Azure cloud. It was initially suggested that the problem may be occurring because the client was running outside Azure cloud (which again, appears unlikely on the face of it).

Azure Keyvault support has come to the conclusion that the issue is in the Azure GO SDK, which is causing requests to get LOST (I'd be really interested to find out who is returning the responses, though!). Azure Key Vault support team also indicated that there is NO logging on the Key Vault side that can identify the base-64 encoded payload that is part of the KeySignParameters. We provided Azure support the sign requests, timestamps and the base-64 encoded payload string that is sent by the SDK to Azure Key Vault with the hope that they would be able to correlate it to request logs they see on their end.

We have a docker image with the client program that can launch a bunch of Sign() requests to any key vault and can demonstrat and log the failures to verify the signature. We would like to request you to reach out to Mr. Odom so he can provide further details of the investigation they conducted, so that you can investigate the bug in Azure Go SDK.

Author:	vdamle
Assignees:	-
Labels:	question, KeyVault, Service Attention, Client, customer-reported, needs-team-attention
Milestone:	-

[navba-MSFT]

Adding the Service team to look into this.

@RandalliLama, @schaabs, @jlichwa Could you please look into this on priority and provide an update on this ? Awaiting your reply.

[chlowell]

Thank you for the sample application, I am able to reproduce the problem. To be specific, the scenario here is local verification of a Key Vault ES256K signature: the application creates a digest, sends it to Key Vault for signing, then locally decodes the signature and verifies it with the go-ethereum package. The problem is that the application fails to verify a minority of these signatures, raising the question of whether this is due to a bug in the SDK or Key Vault.

I didn't find a bug in the SDK. The client sends the correct digest to Key Vault and returns the correct signature to the application i.e., values in the application match what went on the wire. As for Key Vault, I believe the signatures are correct because I was able to verify a problem signature with both Key Vault and Python's cryptography library.

So, I believe the problem is in either the application code or go-ethereum. I'm not familiar enough with the signature format or go-ethereum to say which. If I were to continue debugging this, I would first take a closer look at the application's assumptions about the format as applied to one of the problem signatures.

[ghost]

Hi @vdamle. Thank you for opening this issue and giving us the opportunity to assist. We believe that this has been addressed. If you feel that further discussion is needed, please add a comment with the text “/unresolve” to remove the “issue-addressed” label and continue the conversation.

[ghost]

Hi @vdamle. Thank you for opening this issue and giving us the opportunity to assist. We believe that this has been addressed. If you feel that further discussion is needed, please add a comment with the text “/unresolve” to remove the “issue-addressed” label and continue the conversation.

[ghost]

Hi @vdamle, since you haven’t asked that we “/unresolve” the issue, we’ll close this out. If you believe further discussion is needed, please add a comment “/unresolve” to reopen the issue.