Receive: stop relying on grpc server config to set grpc client secure/skipVerify #7219
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Changes
I'm using a thanos receive HA cluster on ECS Fargate, with 3 endpoints in the hashring, with one dedicated listener + grpc target group per endpoint.
The TLS termination is done at the listener level (so out of thanos). Therefore the thanos receive containers are configured with no TLS.
When testing, I ended up with
error reading server preface: http2: frame too large
errors.After debugging, I found that the receive grpc client is configured in secure mode only if the receive grpc server has TLS enabled.
This breaks the inter-endpoint communication between our hashring members.
To fix this problem, I introduce 2 new flags
--remote-write.client-tls-secure
and--remote-write.client-tls-skip-verify
to mimic what is done in the grpc client configuration of thanos query..Verification
Everything is now working on our cluster.