azcopy sync failed due to server 403

[ranweiler]

azcopy failed to sync an inputs container, after receiving a 403 response from the storage service.

Logs:

2021/10/24 06:27:21 AzcopyVersion  10.12.2
2021/10/24 06:27:21 OS-Environment  windows
2021/10/24 06:27:21 OS-Architecture  amd64
2021/10/24 06:27:21 Log times are in UTC. Local time is 24 Oct 2021 06:27:21
2021/10/24 06:27:21 ==> REQUEST/RESPONSE (Try=1/35.5247ms, OpTime=551.3589ms) -- RESPONSE STATUS CODE ERROR
   HEAD https://<REDACTED>/oft-inputs-<REDACTED>?se=2021-11-21t20%3A03%3A29z&sig=-REDACTED-&sp=rwl&sr=c&st=2021-10-22t13%3A48%3A29z&sv=2020-06-12&timeout=901
   User-Agent: [AzCopy/10.12.2 Azure-Storage/0.14 (go1.16; Windows_NT)]
   X-Ms-Client-Request-Id: [<REDACTED>]
   X-Ms-Version: [2019-12-12]
   --------------------------------------------------------------------------------
   RESPONSE Status: 403 Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.
   Date: [Sun, 24 Oct 2021 06:27:21 GMT]
   Server: [Microsoft-HTTPAPI/2.0]
   X-Ms-Error-Code: [AuthenticationFailed]
   X-Ms-Request-Id: [<REDACTED>]

Note the validity period in the query params:

st (start) = 2021-10-22t13:48:29z
se (expiry) = 2021-11-21t20:03:29z

But the server time is given as 24 Oct 2021 06:27:21 GMT, so this doesn't seem to be about a stale SAS URL.

AB#35954

[mgreisen]

Possible related issue: Azure/azure-storage-azcopy#1122

[jagunter]

Found similar error while fuzzing on internal instance. Job id: d9456762-ef4b-453b-a96e-7b59dd1e554d

##[error][2022-06-30T00:50:22.153Z] HTTP error status (status: 403, body: "\u{feff}<?xml version=\"1.0\" encoding=\"utf-8\"?><Error><Code>AuthenticationFailed</Code><Message>Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.\nRequestId:de5f0982-201e-002a-151b-8cc536000000\nTime:2022-06-30T00:50:22.1653706Z</Message><AuthenticationErrorDetail>Signed expiry time [Thu, 30 Jun 2022 00:49:42 GMT] must be after signed start time [Thu, 30 Jun 2022 00:50:22 GMT]</AuthenticationErrorDetail></Error>")

[stishkin]

@jagunter - the added comment is not related to the original bug. But the bug that you are describing is fixed this PR:
#2116