-
Notifications
You must be signed in to change notification settings - Fork 2.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
CVE: Fix Receiver malicious tenant #5969
Conversation
pkg/receive/handler.go
Outdated
@@ -202,7 +203,7 @@ func NewHandler(logger log.Logger, o *Options) *Handler { | |||
ins := extpromhttp.NewNopInstrumentationMiddleware() | |||
if o.Registry != nil { | |||
ins = extpromhttp.NewTenantInstrumentationMiddleware( | |||
o.TenantHeader, | |||
path.Base(o.TenantHeader), |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
So there are a couple of things here. I don't think this is the right place/input to sanitize, as this is the HTTP header name (by default THANOS-TENANT
) and this is passed to the Prometheus registry for collecting metrics. We would need to sanitize the value of this HTTP header instead.
To highlight the full flow, this header value is fetched in the receiveHTTP
handler, and then goes through handleRequest
-> forward
-> fanoutForward
and passed to h.writer.Write()
which creates a multiTSDB TenantAppendable
to write data for that tenant. There is some logic (getOrLoadTenant
) around how a tenant TSDB is started, but the HTTP header value is eventually used to decide dataDir for the tenant using the defaultTenantDataDir
method.
So, we can sanitize this in receiveHTTP
itself, but I wonder if we should treat tenant id like a path at all. If tenant header value is set to something like ///
, path.Base
would return /
. Instead maybe we should use some regex that fails if tenant id has some backward/forward slash. WDYT? 🙂
Also, I think we'd need to sanitize --receive.default-tenant-id
in a similar way.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
What do you think about having an aliased type that would be used for this? For example:
type tenantID string
func (t *tenantID) String() string {
return string(*t)
}
func NewTenantID(s string) *tenantID {
s = path.Base(s)
return &tenantID{s}
}
(untested but shows my idea). We could put this in a separate package so that it would be impossible to use tenantID
directly. This way, we would have the cleaning-up logic only in one place.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yup, I think this would make it way cleaner!
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hi all I think that using path.Base
is only safe at configuration parse-time. Any time this is used at run-time we are liable to run into very sketchy situations where we think we are safe but we actually aren't. Consider for example nice tenant A
and malicious tenant B/A
. If we path.Base
at run-time and let tenant B/A
write data to disk, they will be writing into tenant A
's directory.
This means that we cannot just rely on doing path.Base(tenantHeaderValue)
to find the directory to write to. We actually have to test path.Base(tenantHeaderValue) == tenantHeaderValue
and return a 503 or something if not.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hi o/ thanks for the review! totally, I didn't think of that, but deff can be used for overriding a tenant :\
87e07c2
to
f52f689
Compare
@GiedriusS how about putting that for now in the |
pkg/receive/handler.go
Outdated
if tenant == "" { | ||
tenant = h.options.DefaultTenantID | ||
tenant = path.Base(h.options.DefaultTenantID) | ||
} | ||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think that the logic can be simplified so that there is less room for error if we keep the original code as-is and then add the following four lines right here. We should determine the tenant value like we did originally and then sanitize it. Otherwise, we are doing path.Base
too many times, which spreads out our sanitation logic and makes it more likely that we will miss one of the lines in future maintenance:
if tenant != path.Base(tenant) {
http.Error(w, err.Error(), http.StatusInternalServerError)
return
}
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yep, totally agree, still new to the codebase so thanks for pointing that out. I'll update the PR.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks, that's better! Thinking about it more, we should actually move this sanitization down below the next if
block where we get the tenant from the certificate.
Maybe we even want to extract tenant value extraction and sanitization into its own, isolated function to keep the logic very tight.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks! I think @GiedriusS recommended something similar too. 🙂
I wonder if we should use path.Base
at all here, though. In case the tenant header value is set to something like /
, we can still have path.Base("/") == "/"
. Maybe instead just checking if the string contains any /
would be easier here?
f52f689
to
92c8fc1
Compare
92c8fc1
to
4ab0ca9
Compare
pkg/receive/handler.go
Outdated
func (h *Handler) isTenantValid(tenant string, err error, w http.ResponseWriter) { | ||
if tenant != path.Base(tenant) { | ||
http.Error(w, err.Error(), http.StatusInternalServerError) | ||
return |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Maybe we can return an error here, and then log + respond 400 in receiveHTTP
instead? 🙂
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
+1, will make the changes, it'll make testing easier. Will push a draft so we can iterate
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It's shaping up nicely @danielmellado! 👍 Adding a test case for this would be nice.
pkg/receive/handler.go
Outdated
"github.com/mwitkow/go-conntrack" | ||
"github.com/opentracing/opentracing-go" | ||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Accidentally changed formatting?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
damn, blame the IDE, I'll put it back on my next commit xD
pkg/receive/handler.go
Outdated
@@ -403,6 +405,13 @@ func (h *Handler) handleRequest(ctx context.Context, rep uint64, tenant string, | |||
return h.forward(ctx, tenant, r, wreq) | |||
} | |||
|
|||
func (h *Handler) isTenantValid(tenant string, err error, w http.ResponseWriter) { | |||
if tenant != path.Base(tenant) { | |||
http.Error(w, err.Error(), http.StatusInternalServerError) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Should be server error? 🤔 Or bad request?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
hmm that's a good point. It could be either but in any case I'll go with Saswata's suggestion, stay tuned!
4ab0ca9
to
41d1c47
Compare
41d1c47
to
5275d09
Compare
c5391f8
to
e382d5a
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM!
Just a few small suggestions.
96bb424
to
bb87648
Compare
Failure on the tests seems totally unrealted |
If running as root or with enough privileges, receiver can create a directory outside of the configured TenantHeader. This commit fixes it up by sanitizing the user input and explicity not allowing such behavior. Signed-off-by: Daniel Mellado <dmellado@redhat.com>
bb87648
to
74d2560
Compare
* Update Thanos engine to latest version (thanos-io#6069) This commit updates the Thanos PromQL engine to the latest version. Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com> Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com> * Receive: Tenants' external labels proposal (thanos-io#5720) * Receive external labels proposal Signed-off-by: haanhvu <haanh6594@gmail.com> * Restructure and edit proposal's content Signed-off-by: haanhvu <haanh6594@gmail.com> * Update proposal Signed-off-by: haanhvu <haanh6594@gmail.com> * Fix doc error Signed-off-by: haanhvu <haanh6594@gmail.com> Signed-off-by: haanhvu <haanh6594@gmail.com> * fixing doc CI (thanos-io#6072) Signed-off-by: Ben Ye <benye@amazon.com> Signed-off-by: Ben Ye <benye@amazon.com> * Fix stores filtering resets on reload (thanos-io#6063) * Fix stores filtering resets on reload `g0.store_matches` parameter appears in the url but doesn't applies in the frontend. Looks like it has been done on purpose and by removing a small piece of code fixes this issue. variable named `debugMode` is used for the store filtering checkbox which is an unappropriate name. Using `enableStoreFiltering` variable to represent the state of checkbox. Signed-off-by: Pradyumna Krishna <git@onpy.in> * Regenerate bindata.go Signed-off-by: Pradyumna Krishna <git@onpy.in> Signed-off-by: Pradyumna Krishna <git@onpy.in> * Store: Make initial sync more robust Added re-try mechanism for store inital sync, where if the initial sync fails, it tries to do the initial sync again for given timeout duration. Signed-off-by: Kartik-Garg <kartik.garg@infracloud.io> * Recover from panics in Series calls (thanos-io#6077) * Recover from panics in Series calls This commit adds panic recovery for Series calls in all Store servers. Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com> * Apply error suggestion Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com> --------- Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com> * query: reuse our own gate (thanos-io#6079) Do not call promgate directly but rather use our own wrapper that does everything we want - duration histogram, current in-flight calls, total calls. Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> * Store: Support disable cache index header file. (thanos-io#5773) * Store: Support disable cache index header file. Signed-off-by: wanjunlei <wanjunlei@kubesphere.io> * Store: add a seprate flag to disable caching index header file Signed-off-by: wanjunlei <wanjunlei@kubesphere.io> * Tools: add cleanup API for bucket web Signed-off-by: wanjunlei <wanjunlei@kubesphere.io> * resolve conversation Signed-off-by: wanjunlei <wanjunlei@kubesphere.io> * resolve confilcts Signed-off-by: wanjunlei <wanjunlei@kubesphere.io> * change the flag to `--cache-index-header` Signed-off-by: wanjunlei <wanjunlei@kubesphere.io> * Wrap mem writer in file writer Signed-off-by: wanjunlei <wanjunlei@kubesphere.io> * update CHANGELOG Signed-off-by: wanjunlei <wanjunlei@kubesphere.io> * update CHANGELOG Signed-off-by: wanjunlei <wanjunlei@kubesphere.io> * fix bug Signed-off-by: wanjunlei <wanjunlei@kubesphere.io> --------- Signed-off-by: wanjunlei <wanjunlei@kubesphere.io> Co-authored-by: wanjunlei <wanjunlei@yujnify.com> * CVE: Fix Receiver malicious tenant (thanos-io#5969) If running as root or with enough privileges, receiver can create a directory outside of the configured TenantHeader. This commit fixes it up by sanitizing the user input and explicity not allowing such behavior. Signed-off-by: Daniel Mellado <dmellado@redhat.com> * Add adopter Grupo MasMovil (thanos-io#6084) Signed-off-by: Pablo Moncada Isla <pablo.moncada@masmovil.com> * fix typo (thanos-io#6087) Signed-off-by: cyip <cyip@jackhenry.com> Co-authored-by: cyip <cyip@jackhenry.com> * optimize selector to string (thanos-io#6076) Signed-off-by: Kama Huang <kamatogo13@gmail.com> * Fix: Failure to close BlockSeriesClient cause store-gateway deadlock (thanos-io#6086) * Fix: Failure to close BlockSeriesClient cause store-gateway deadlock Signed-off-by: Alan Protasio <alanprot@gmail.com> * Adding tests Signed-off-by: Alan Protasio <alanprot@gmail.com> * reverting the change on get series Signed-off-by: Alan Protasio <alanprot@gmail.com> * fix lint Signed-off-by: Alan Protasio <alanprot@gmail.com> --------- Signed-off-by: Alan Protasio <alanprot@gmail.com> * Cut 0.30.2 (thanos-io#6081) * tracing: fixed panic because of nil sampler (thanos-io#6066) * fixed panic because of nil sampler Signed-off-by: Vasiliy Rumyantsev <4119114+xBazilio@users.noreply.github.com> * added CHANGELOG entry Signed-off-by: Vasiliy Rumyantsev <4119114+xBazilio@users.noreply.github.com> Signed-off-by: Vasiliy Rumyantsev <4119114+xBazilio@users.noreply.github.com> * bump version to 0.30.2 Signed-off-by: Ben Ye <benye@amazon.com> * Updates busybox SHA (thanos-io#6046) Signed-off-by: GitHub <noreply@github.com> Signed-off-by: GitHub <noreply@github.com> Co-authored-by: yeya24 <yeya24@users.noreply.github.com> * Use `e2edb.NewMinio` to disable SSE-S3 in e2e tests (thanos-io#6055) * Use e2edb.NewMinio to disable SSE Signed-off-by: Saswata Mukherjee <saswataminsta@yahoo.com> * Use temp fork for TLS Signed-off-by: Saswata Mukherjee <saswataminsta@yahoo.com> * Fix broken rules api fanout test Signed-off-by: Saswata Mukherjee <saswataminsta@yahoo.com> * Fix broken query compatibility test Signed-off-by: Saswata Mukherjee <saswataminsta@yahoo.com> * Remove fork Signed-off-by: Saswata Mukherjee <saswataminsta@yahoo.com> Signed-off-by: Saswata Mukherjee <saswataminsta@yahoo.com> --------- Signed-off-by: Vasiliy Rumyantsev <4119114+xBazilio@users.noreply.github.com> Signed-off-by: Ben Ye <benye@amazon.com> Signed-off-by: GitHub <noreply@github.com> Signed-off-by: Saswata Mukherjee <saswataminsta@yahoo.com> Co-authored-by: Vasiliy Rumyantsev <4119114+xBazilio@users.noreply.github.com> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: yeya24 <yeya24@users.noreply.github.com> Co-authored-by: Saswata Mukherjee <saswataminsta@yahoo.com> * cherry pick store gateway fix to release 0.30 (thanos-io#6089) * Fix: Failure to close BlockSeriesClient cause store-gateway deadlock (thanos-io#6086) * Fix: Failure to close BlockSeriesClient cause store-gateway deadlock Signed-off-by: Alan Protasio <alanprot@gmail.com> * Adding tests Signed-off-by: Alan Protasio <alanprot@gmail.com> * reverting the change on get series Signed-off-by: Alan Protasio <alanprot@gmail.com> * fix lint Signed-off-by: Alan Protasio <alanprot@gmail.com> --------- Signed-off-by: Alan Protasio <alanprot@gmail.com> * update changelog Signed-off-by: Ben Ye <benye@amazon.com> --------- Signed-off-by: Alan Protasio <alanprot@gmail.com> Signed-off-by: Ben Ye <benye@amazon.com> Co-authored-by: Alan Protasio <alanprot@gmail.com> * fix changelog entries Signed-off-by: Ben Ye <benye@amazon.com> * docs: improving the description for tsdb.retention on the receiver Signed-off-by: Victor Fernandes <victorhbfernandes@gmail.com> * Receiver: Use `intern` package when reallocating label strings (thanos-io#5926) * Cleanup go mod Signed-off-by: Matej Gera <matejgera@gmail.com> * Use string interning for labels realloc method Signed-off-by: Matej Gera <matejgera@gmail.com> * Enhance label realloc benchmarks Signed-off-by: Matej Gera <matejgera@gmail.com> * Make interning optional; put behind hiddend flag Signed-off-by: Matej Gera <matej.gera@coralogix.com> * Update CHANGELOG Signed-off-by: Matej Gera <matej.gera@coralogix.com> * Address feedback - Fix wrong condition - Adjust benchmarks Signed-off-by: Matej Gera <matej.gera@coralogix.com> --------- Signed-off-by: Matej Gera <matejgera@gmail.com> Signed-off-by: Matej Gera <matej.gera@coralogix.com> Signed-off-by: Matej Gera <38492574+matej-g@users.noreply.github.com> * Updaing README with drawing fixes and minor wording clarification (thanos-io#6078) * New drawing and wording for Thanos other deployment models Signed-off-by: Jonah Kowall <jkowall@kowall.net> * New drawing and wording for Thanos other deployment models Signed-off-by: Jonah Kowall <jkowall@kowall.net> * Added comments to README.md and updated the quick-tutorial.md with the same diagram updates and text to match Signed-off-by: Jonah Kowall <jkowall@kowall.net> * Ran make docs Signed-off-by: Jonah Kowall <jkowall@kowall.net> --------- Signed-off-by: Jonah Kowall <jkowall@kowall.net> * Compact: Remove spam of replica label removed log (thanos-io#6088) * Remove spam of replica label removed log Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Reduce amount of removed replica label instead of removing it Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Reformat code Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> --------- Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Store: Don't error when no stores are matched (thanos-io#6082) It's normal and not an error if a query does not match due to no downstream stores. This is common when querying with external labels and tiered query servers. This bug was introduced in thanos-io#5296 Fixes: thanos-io#5862 Signed-off-by: SuperQ <superq@gmail.com> * docs: Fix must have Ruler alerts definition (thanos-io#6058) * Fix must have Ruler alerts definition ThanosRuler missing rule intervals metric used the wrong comparator sign, confusing users trying to create the rule. Signed-off-by: Maxim Muzafarov <m.muzafarov@gmail.com> * Update docs/components/rule.md Co-authored-by: Saswata Mukherjee <saswataminsta@yahoo.com> Signed-off-by: Maxim Muzafarov <m.muzafarov@gmail.com> --------- Signed-off-by: Maxim Muzafarov <m.muzafarov@gmail.com> Co-authored-by: Saswata Mukherjee <saswataminsta@yahoo.com> * Fix conflicts Signed-off-by: haanhvu <haanh6594@gmail.com> * Specify overwriting behavior in flag and add validation Signed-off-by: haanhvu <haanh6594@gmail.com> * Add log and doc Signed-off-by: haanhvu <haanh6594@gmail.com> * Mixins(Rule): Fix query for long rule evaluations (thanos-io#6103) * mixin(Rule): Fix query for long rule evaluations Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Update changelog Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> --------- Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> --------- Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com> Signed-off-by: haanhvu <haanh6594@gmail.com> Signed-off-by: Ben Ye <benye@amazon.com> Signed-off-by: Pradyumna Krishna <git@onpy.in> Signed-off-by: Kartik-Garg <kartik.garg@infracloud.io> Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> Signed-off-by: wanjunlei <wanjunlei@kubesphere.io> Signed-off-by: Daniel Mellado <dmellado@redhat.com> Signed-off-by: Pablo Moncada Isla <pablo.moncada@masmovil.com> Signed-off-by: cyip <cyip@jackhenry.com> Signed-off-by: Kama Huang <kamatogo13@gmail.com> Signed-off-by: Alan Protasio <alanprot@gmail.com> Signed-off-by: Vasiliy Rumyantsev <4119114+xBazilio@users.noreply.github.com> Signed-off-by: GitHub <noreply@github.com> Signed-off-by: Saswata Mukherjee <saswataminsta@yahoo.com> Signed-off-by: Victor Fernandes <victorhbfernandes@gmail.com> Signed-off-by: Matej Gera <matejgera@gmail.com> Signed-off-by: Matej Gera <matej.gera@coralogix.com> Signed-off-by: Matej Gera <38492574+matej-g@users.noreply.github.com> Signed-off-by: Jonah Kowall <jkowall@kowall.net> Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> Signed-off-by: SuperQ <superq@gmail.com> Signed-off-by: Maxim Muzafarov <m.muzafarov@gmail.com> Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> Co-authored-by: Filip Petkovski <filip.petkovsky@gmail.com> Co-authored-by: Ha Anh Vu <75315486+haanhvu@users.noreply.github.com> Co-authored-by: Ben Ye <benye@amazon.com> Co-authored-by: Pradyumna Krishna <git@onpy.in> Co-authored-by: Kartik-Garg <kartik.garg@infracloud.io> Co-authored-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> Co-authored-by: wanjunlei <53003665+wanjunlei@users.noreply.github.com> Co-authored-by: wanjunlei <wanjunlei@yujnify.com> Co-authored-by: Daniel Mellado <1313475+danielmellado@users.noreply.github.com> Co-authored-by: Pablo Moncada <pmoncadaisla@gmail.com> Co-authored-by: Chantel Yip <52993239+sshantel@users.noreply.github.com> Co-authored-by: cyip <cyip@jackhenry.com> Co-authored-by: Kama Huang <121007071+kama910@users.noreply.github.com> Co-authored-by: Alan Protasio <alanprot@gmail.com> Co-authored-by: Vasiliy Rumyantsev <4119114+xBazilio@users.noreply.github.com> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: yeya24 <yeya24@users.noreply.github.com> Co-authored-by: Saswata Mukherjee <saswataminsta@yahoo.com> Co-authored-by: Victor Fernandes <victorhbfernandes@gmail.com> Co-authored-by: Matej Gera <38492574+matej-g@users.noreply.github.com> Co-authored-by: Jonah Kowall <jkowall@kowall.net> Co-authored-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> Co-authored-by: Ben Kochie <superq@gmail.com> Co-authored-by: Maxim Muzafarov <m.muzafarov@gmail.com> Co-authored-by: haanhvu <haanh6594@gmail.com>
If running as root or with enough privileges, receiver can create a directory outside of the configured TenantHeader. This commit fixes it up by sanitizing the user input and explicity not allowing such behavior. Signed-off-by: Daniel Mellado <dmellado@redhat.com>
If running as root or with enough privileges, receiver can create a directory outside of the configured TenantHeader. This commit fixes it up by sanitizing the user input and explicity not allowing such behavior. Signed-off-by: Daniel Mellado <dmellado@redhat.com>
If running as root or with enough privileges, receiver can create a
directory outside of the configured TenantHeader.
This commit fixes it up by sanitizing the user input and explicity not
allowing such behavior.
Signed-off-by: Daniel Mellado dmellado@redhat.com