SSLPeerUnverifiedException on S3 actions #1786
Comments
inssein
added
bug
|
@inssein did some investigation and I believe this is related to a regression found in Apache's We will work in upgrading the |
debora-ito
added
dependencies
|
@debora-ito ahh thats good to know, thanks. |
|
@debora-ito rather late update, but I just tried pinning one of my services to 4.5.12 (same version used in dropwizard), and I am still getting the same error. |
|
@inssein The exactly same error? According to the Jira issue I linked above it was fixed in 4.5.10. |
|
Yup, exactly the same error. I have ran The only thing I am going to do next is try the url-connection http builder and see if it resolves the issues. |
|
I have the code running with the url connection http service, and everything seems to be running smooth, which points to an issue with the apache http client. @debora-ito it does seem like it should be fixed in the new version, but |
|
Can you share your dependency tree? Just the part regarding Apache httpcomponents - |
|
|
This is very interesting, I'm also getting the same exact issue with 4.5.12. I saw some promising mods that people have done, but nothing fixes this problem, even replacing the trust store to trust all certs. But I can issue my REST request via Postman and in the Firefox 76.0 browser with no problems. |
|
Yup, I gave up in the end and just used the URLConnectionClient for now as the service didn't require high performance, but have a todo to switch it out once this dependency is upgraded. |
|
Hi @debora-ito , I've observed endpoints .s3.amazonaws.com and .s3.us-east-1.amazonaws.com return different certificates. Global endpoint: $ true | openssl s_client -connect some-bucket.s3.amazonaws.com:443 2>/dev/null | openssl x509 -noout -text
Certificate:
Data:
Version: 3 (0x2)
Serial Number:
08:2d:f6:8e:e9:c6:93:15:be:bf:72:07:9b:38:10:fd
Signature Algorithm: sha256WithRSAEncryption
Issuer: C=US, O=DigiCert Inc, OU=www.digicert.com, CN=DigiCert Baltimore CA-2 G2
Validity
Not Before: Nov 9 00:00:00 2019 GMT
Not After : Mar 12 12:00:00 2021 GMT
Subject: C=US, ST=Washington, L=Seattle, O=Amazon.com, Inc., CN=*.s3.amazonaws.com
Subject Public Key Info:
Public Key Algorithm: rsaEncryption
...
X509v3 extensions:
X509v3 Subject Alternative Name:
DNS:*.s3.amazonaws.com, DNS:s3.amazonaws.com
...Regional endpoint: $ true | openssl s_client -connect some-bucket.s3.us-east-1.amazonaws.com:443 2>/dev/null | openssl x509 -noout -text
Certificate:
Data:
Version: 3 (0x2)
Serial Number:
0d:64:50:6b:45:f3:0c:e3:5a:6c:2d:df:2c:18:b4:37
Signature Algorithm: sha256WithRSAEncryption
Issuer: C=US, O=DigiCert Inc, OU=www.digicert.com, CN=DigiCert Baltimore CA-2 G2
Validity
Not Before: Aug 4 00:00:00 2020 GMT
Not After : Aug 9 12:00:00 2021 GMT
Subject: C=US, ST=Washington, L=Seattle, O=Amazon.com, Inc., CN=s3.amazonaws.com
Subject Public Key Info:
Public Key Algorithm: rsaEncryption
...
X509v3 extensions:
X509v3 Subject Alternative Name:
DNS:s3.amazonaws.com, DNS:*.s3.amazonaws.com, DNS:*.s3.dualstack.us-east-1.amazonaws.com, DNS:s3.dualstack.us-east-1.amazonaws.com, DNS:*.s3.us-east-1.amazonaws.com, DNS:s3.us-east-1.amazonaws.com, DNS:*.s3-control.us-east-1.amazonaws.com, DNS:s3-control.us-east-1.amazonaws.com, DNS:*.s3-control.dualstack.us-east-1.amazonaws.com, DNS:s3-control.dualstack.us-east-1.amazonaws.com, DNS:*.s3-accesspoint.us-east-1.amazonaws.com, DNS:*.s3-accesspoint.dualstack.us-east-1.amazonaws.com, DNS:*.s3.us-east-1.vpce.amazonaws.com
....So, this code: import software.amazon.awssdk.regions.Region
import software.amazon.awssdk.services.s3.S3Client
import software.amazon.awssdk.services.s3.model.DeleteObjectRequest
import software.amazon.awssdk.services.s3.model.DeleteObjectsRequest
import software.amazon.awssdk.services.s3.model.GetObjectRequest
import software.amazon.awssdk.services.s3.model.ListObjectsV2Request
import software.amazon.awssdk.services.s3.model.ObjectIdentifier
import software.amazon.awssdk.services.s3.model.S3Object
import software.amazon.awssdk.services.s3.presigner.S3Presigner
fun main() {
val bucket = awsConfiguration.bucket
val prefix = "foo/bar"
val listObjectsV2PaginatorResult = s3Client.listObjectsV2Paginator(
ListObjectsV2Request
.builder()
.bucket(bucket)
.prefix(prefix)
.build()
)
val keys: List<String> = listObjectsV2PaginatorResult
.contents()
.stream()
.map { it.key() }
.toList()
logger.info { "==================>>>>> KEYS: $keys" }
}Worked in EU-CENTRAL-1 , but returned SdkClientException: Unable to execute HTTP request: Certificate for <some-bucket.s3.amazonaws.com> doesn't match any of the subject alternative names: [*.s3.amazonaws.com, s3.amazonaws.com] is US-EAST-1. Full stack trace: Setting AWS_S3_US_EAST_1_REGIONAL_ENDPOINT to regional fixed it. But I believe this is a bug because the exception is raised even if the region is being explicitly passed to the S3Client builder: |
|
@raonitimo that is the expected behavior when providing |
|
I'm having the exact same issue as described here. I have got it working using the @debora-ito, @raonitimo would this be the correct way to get the SDK to pick up the
For a Spring app with a bean configuration like: This is my stack trace: The S3 bucket I'm getting the above error from is in US East (N. Virginia). I also have made sure that the apache client i've installed is the version stated as having the fix: |
|
Is anyone still encountering this with the latest SDK and Apache version? We're not able to reproduce in our testing. |
|
Same issue here on us-east-1 using java SDK version 2. The only affected region is us-east-1:
Setting the AWS_S3_US_EAST_1_REGIONAL_ENDPOINT to 'regional' didn't work. Can anybody please help with this? |
|
Ya I never resolved this issue either. We still use v1 for S3 because of this
Get Outlook for Android<https://aka.ms/ghei36>
…________________________________
From: Fran ***@***.***>
Sent: Wednesday, July 28, 2021 6:15:35 PM
To: aws/aws-sdk-java-v2 ***@***.***>
Cc: dlavelle7 ***@***.***>; Comment ***@***.***>
Subject: Re: [aws/aws-sdk-java-v2] SSLPeerUnverifiedException on S3 actions (#1786)
Same issue here on us-east-1 using java SDK version 2. The only affected region is us-east-1:
Caused by: javax.net.ssl.SSLPeerUnverifiedException: Certificate for <some-dashed-string-us-east-1.s3.amazonaws.com> doesn't match any of the subject alternative nam es: [*.s3.amazonaws.com, s3.amazonaws.com]
Setting the AWS_S3_US_EAST_1_REGIONAL_ENDPOINT to 'regional' didn't work. Can anybody please help with this?
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub<#1786 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ABE372LNGSFCNNQLHSLBU3LT2A3LPANCNFSM4MNBZPAA>.
|
|
In the same boat. One workaround is to force a very specific Apache version: 4.5.10. A dozen of our customers are hitting this issue. Based on our reports, it seems that the customers hitting this issue can always reproduce it. Unfortunately, the problem is not reproducible on our side, and often, a customer cannot reproduce this on a local machine but only on a CI pipeline. As people noted here, the issue happens only when using recent Apache versions. That is, 4.5.10 works but 4.5.12 or 4.5.13 fails. Our customers don't use Java SDK at all, so the issue is fundamentally unrelated to the SDK. So one workaround would be to force Apache 4.5.10. However, do note that using an older version prior to 4.5.10 may fail with the same error, so make sure you enforce the right version. I've been debugging and testing the Apache code for some time, but I haven't found anything that can go wrong. Now I start to suspect that this might be an issue of Amazon where some regional factor plays a role. It seems only Amazon S3 that has this problem. And I think using |
…52d753d4f Pull request: release <- staging/05e3418b-4cd6-4bf8-8edc-17152d753d4f
|
Same issue appears: Unable to execute HTTP request: Certificate for <x.x.x.x.x.x.x.s3.amazonaws.com> doesn't match any of the subject alternative names: [*.s3.amazonaws.com, s3.amazonaws.com] @chanseokoh thanks for the suggesting, will try explicitly downgrading lib version. |
|
Hi @BMalaichik did it work downgrading lib version 🤔 |
|
@dieggoluis for me it didn't From docs
So I was able to change bucket naming convention to avoid any issues in future |
|
Also facing this issue, in |
I vaguely remember dots in bucket names definitely cause trouble verifying certificates. You shouldn't use dots. However, I just want to make it clear that some customers do hit this issue even though their S3 bucket names don't contain dots. |
|
I think this might be due to the fact that https://www.publicsuffix.org/list/public_suffix_list.dat contains "s3.amazonaws.com" which means any certificate with "*.s3.amazonaws.com" will be considered overly broad |
|
I've just had this very issue and can confirm that from the errors it's quite obvious that it is linked to dots in the S3 bucket name. It is a standard practice to set your S3 bucket name like a reverse DNS name to ensure no collisions (due to the scope of S3 bucket names being global)... e.g. This is causing this very error for me right now, and the error string makes it clear it can't match the list of peer certificates, and this is not surprising since wildcard certificates only support a single level of sub-domains. The simplest solution, therefore, is to create another bucket where the dots are hyphens which is also unlikely to clash with another global S3 bucket name. |
|
Incase it helps anyone else, I was hitting this error as part of a spark job using org.apache.hadoop.fs.s3a.S3AFileSystem. Setting these fixed the problem: |
|
I have persistent following problem: My configuration:
The code I use to build the S3Client: private S3Client getClient() {
AwsCredentialsProvider credentialProvider = getCredentialProvider();
S3ClientBuilder builder =
S3Client.builder()
.credentialsProvider(credentialProvider)
.region(getRegion(config.getRegion()));
if (StringUtils.isNotBlank(config.getEndpointUrl())) {
LOG.debug("Using custom endpoint: {}", config.getEndpointUrl());
builder.endpointOverride(URI.create(config.getEndpointUrl()));
}What surprises me:
Why I think the bug is in the SDK:
|
|
@pwannenmacher from my comment (#1786 (comment)),
Back then, I meticulously checked the code of all the Apache versions. However, nothing seemed wrong, and I couldn't explain how this could happen. I was very perplexed. Our customers (who don't use the Java SDK) hit this issue only with Amazon S3. |
|
Reproduced this bug with hadoop-* 3.3.5 and aws-java-sdk 1.12.310 for me this piece of code cause the problem I can see host name verifier is initialized with an X509HostnameVerifier the main difference between implementations (left one causes problem): |
|
seems to me there's a couple of problems being discussed here.
Fix there: identify the JAR with the class, remove or upgrade it. If that can't be done: cut the file from the jar. |
I have recently re-written a service to use the newer AWS SDK (v2), but I am struggling with an error I just can't seem to figure out.
Short snippet:
Description
This service communicates a lot with a few other AWS services, and everything there is fine, but when it is running in production, it seems to have issues writing to customer buckets with the error above.
I have gone ahead and changed this project to only use AWS SDK v1 for S3 with almost identical commands, and it works fine.
I am running in an EC2 instance, using JDK 8, and using the latest version of this library.
Full stack trace
The text was updated successfully, but these errors were encountered: