Thanos and multi-tenancy

[matej-g]

Off the back of my comment: #5415 (comment)

I was encouraged to start a discussion regarding how we want to handle tenancy, and, especially in the light of some recent experimental features (#5520, #5527), which will expand the tenancy features in Thanos.

With the upcoming aim of preparing Thanos for stable (v1) release, I think it would be good to discuss what is the direction and ideology that Thanos wants to take with regards to tenancy features.

Up until now, Thanos did not really have a tenancy features, if we omit some tenancy-relevant configuration for the receiver component (arguably, this is the minimum that the receiver needs to know in order for e.g. hashing to work properly etc.). My understanding was that until now, Thanos did not want to deal with tenancy and this is what prompted existence of other projects (e.g. https://github.com/observatorium/observatorium) that enable tenancy features on top of Thanos. Introducing such features as mentioned above, we are now letting tenancy into Thanos, changing the dynamic and expanding the feature surface (and with it the codebase complexity, maintainance burden etc.).

Whereas previously I would turn to projects outside of Thanos to cover my tenancy needs, I'm not left with a split solution - some needs will be covered by Thanos, yet other remain within projects outside of the Thanos project. It opens a question of whether

1. Thanos wants to change it's ideology and allow for multi-tenancy features (and this might have never been questioned, but perhaps was never discussed or codified?);
2. Does this open gates for other tenancy features (e.g. authZ/authN) and which features should be targeted if so?

I was wondering about other opinons in the community, thanks.

[douglascamata]

We have to consider the impact of not making multi-tenancy a big part of Thanos from the point of view of possible users/operators that might be looking at long-term storage solutions for Prometheus. Would people give up on Thanos if it doesn't clearly state that multi-tenancy is supported? Both Cortex and Mimir officially state they are multi-tenant. Does Thanos want to be seem as the system that you can configure multi-tenancy "at your own risk" and at the cost of higher resource usage due to having to run external components maintained by 3rd parties?

I think embracing tenancy in Thanos opens the door for many desirable features to be efficiently implemented. Not every tenant related feature should be included in Thanos though. There are two types of tenant-aware features that I can see but don't know exactly what criteria separates them, but here are my suggestions:

• The ones that require some "insider knowledge" in form of implementation details or state from Thanos' systems. They seem to be more tightly coupled with Thanos.

  For example, request (rate) limiting based on the amount of data sent requires unmarshalling the protobuffer contents of a remote write request (insider knowledge). This can be done inside Thanos in places where such unmarshalling has already happened (efficient), or it can be done in a separate proxy-like component/service, that will also have to do this unmarshalling work. The later incurs in increased resource usage and latency in the write path is relevant in a high load environment.

• The ones do not require this "insider knowledge"/state from Thanos' system. Not coupled with Thanos at all.

  For example, for authZ/authN no information from inside Thanos is required. In Observatorium, only knowledge of the tenant header is needed and successful requests are proxied into Thanos Receive with the tenant stamped in such header.

[matej-g]

Those are good points @douglascamata, thanks for bringing them!

I think it could still a reasonable solution to focus on one area and leave multi-tenancy to some external project, and it would still be acceptable. If you think about it, in some way this is also relationship between Prometheus and Thanos/Cortex/Mimir, i.e. Prometheus won't support HA / long term storage, but you can use these trusted solutions that build on top of it. Perhaps similar relationship could be built between Thanos and Observatorium (or any other project that emerges).

With regards to insider knowledge, as I mentioned previously in the other discussion, this could be a legitimate concern, but in that case we should have a good understanding when / why this is not possible. It might seem like adding more work if doing things without insider knowledge, but do we have the numbers? Do we know by how much? And at which point should it outweigh other aspects I mentioned previously (larger codebase, maintanance burden etc.)?

And another point that I'm more wary of is splitting the user experience, if we go with the insider / outsider knowledge approach - suddenly, I have two different components that I need to configure to achieve one aim (multi-tenancy). Perhaps this could be also solved by a third component that could bring them together and configure them, but I'm not fully sold on that.

[douglascamata]

It might seem like adding more work if doing things without insider knowledge, but do we have the numbers? Do we know by how much? And at which point should it outweigh other aspects I mentioned previously (larger codebase, maintenance burden etc.)?

I mean "work" in the computational sense (i.e. snappy decode request body, unmarshall protobuffer, process it, marshall, snappy encode again, proxy down to Thanos). Not sure if you mean the same. This isn't exactly measurable, just like the larger codebase or maintenance burden.

There is als other "work" of operating all these extra pieces/services too, which should be considered.

[fpetkovski]

I don't have a strong opinion on whether multitenancy should be baked in or not. However, I would prefer for it to be treated internally like a separate module that is loaded dynamically based on configuration parameters. I would like for multi-tenancy features and various limits to not interfere with the maintainability of the codebase, or with the performance for users that don't use multi-tenancy.

[bwplotka]

Thanks for this discussion!

Whereas previously I would turn to projects outside of Thanos to cover my tenancy needs, I'm not left with a split solution - some needs will be covered by Thanos, yet others remain within projects outside of the Thanos project. It opens the question of whether
Thanos wants to change its ideology and allow for multi-tenancy features (and this might have never been questioned, but perhaps was never discussed or codified?);
Does this open gates for other tenancy features (e.g. authZ/authN), and which features should be targeted if so?

I don't we ever had the ideology to either accept or reject the tenancy features. I think the ideology (if we can call it like that) was to have a system that:

• Solves the problem it defines to solve (distributed database metric signal of observability with Prometheus data model) - so implementing all APIs Prometheus do, plus a few debugging ones, integration with common tools. We have already grown a bit towards the analytics side (we probably could do it more).
• Simplest possible to understand, install and operate for any scale (obviously, with scale, it will always be more effort for users, but keeping it low) - so single binary, consistent, minimal configuration, least amount of different services, no magic gossip which requires non-trivial debugging skills or special order of deployment etc.
• Cheap to run.

I think multi-tenancy is a grey area. We want to enable others to run Thanos in a soft muli-tenancy fashion (we already do so). The question is rather:

• Do we need to have everything designed for a particular tenancy format, e.g. certain user ID that works well with, like AWS IAM?
• Or should we rather be "tenancy agnostic", i.e. Create extensions, API and integrations allowing multi-tenancy on top? e.g. billing, usage measurement and isolation per certain labels or requests. Writes isolation per certain label (external one, "tenant" one in receive).

To answer this, I would take a detailed look at how that might look from both propositions for both user and maintainer. Will try to prepare some visualisation of this in next days.

We discussed this on our Thanos Community Call as well and:

• The reason why we should NOT have tenant depending logic not are unclear.
• Making it tenant agnostic (e.g. write limits per label not tenant) makes all problems harder to implement as they are more generic. This is because tenant specific implementations can be much more opinionated and more efficient due to certain assumptions we can make (e.g tenant label is always external label). Plus, debugging and error logs will be more clear when we bake tenancy in Thanos.

So far I think we are more into making full multi-tenancy awareness for all Thanos components 🤔