[Feature][Manager] Support multi-tenancy

[vernedeng]

Description

Background

The current Inlong does not support multi-tenancy, and only has a simple permission model to implement data and resource isolation at the manager level. This means that for a single ordinary user, they can only see the resources they created or those where others have added them as the manager. This mode leads to the following problems:

1. Every time a new resource is created, many chargers need to be added, which is not user-friendly.
2. For adding or removing members from projects, a large number of resource managers need to be updated separately, which is not user-friendly and might cause a cascading effect (e.g. changing the manager of the tdmanager's bid leads to simultaneous updates to more than 5000 tid us tasks, causing update queue congestion and other issues).
3. Various IDs and names are prone to conflicts, and to avoid conflicts, users might set their names to be particularly long.
4. It is not possible to conduct cost accounting and other tasks based on manager information.
5. For managers, all retrieval operations require field matching on the inCharges field, which affects performance.
   find_in_set(#{currentUser, jdbcType=VARCHAR}, in_charges)

Solutions

In the industry, multi-tenant solutions are typically formulated from two perspectives: isolation levels and tenant hierarchy.

1. Isolation level of tenants:
   1. Independent deployment for each tenant (single-tenant mode): Each tenant has a completely independent set of services.
      • Advantages:
        • With each tenant having a completely independent set of services, this scheme has the highest level of user data isolation and the best security. The scope of impact when faults occur is small and easy to repair.
      • Disadvantages:
        • Higher deployment and operational costs.
   2. Fully shared resources (complete multi-tenant mode): All tenants completely share all components.
      • Advantages:
        • Maximum resource utilization and good scalability.
        • Information movement between tenants does not require migration between two independent services.
      • Disadvantages:
        • Risk of data leakage if resource isolation is not handled properly.
        • Modification operations must be very cautious.
        • Large tenants may squeeze resources for small tenants.
        • Wide-ranging impacts in case of anomalies.
   3. Tenants share some resources (horizontal partitioning deployment mode): Tenants share some components while some components are deployed independently, such as shared message queues, with each tenant maintaining its own database.
      • Advantages:
        • When most system loads are due to specific components that can be individually deployed for each tenant, horizontal partitioning deployment helps alleviate the issue of "noisy neighbor" interference. For example, the database may absorb most of the system load due to high query workload. If a single tenant sends a large number of requests to the solution, the performance of the database may be negatively affected, but the databases of other tenants (and shared components, such as the application layer) remain unaffected.
      • Disadvantages:
        • More demanding for the designers in terms of component scalability and ease of deployment.

2. Tenant hierarchy:
Different organizations have different requirements for data isolation, making it difficult to find a universally applicable multi-user model to fit all organizations.

1. Tenants in Kubernetes (k8s):

   1. Tenant based on Namespace: Kubernetes uses Namespaces to express the ownership of a specific resource, with most object resources in Kubernetes being assigned to a specific Namespace. If we use Namespace to represent the ownership of a resource, then all resource objects within a Namespace are described as belonging to the same user. Namespace creation and usage require authentication, authorization, and granting specific rights. Only a super administrator can create a Namespace, and other users need explicit permissions to use these Namespaces, including: creating, viewing, and modifying resource objects under a Namespace. By setting appropriate security policies, it is possible to prevent non-privileged users from operating on specific resource objects.

   However, the single-level tenant concept also brings some problems:

   • Different Namespaces do not have a concept of common ownership, i.e., the same team cannot use the same set of authentication to control multiple different Namespaces simultaneously.
   • Namespace creation requires a high-level permission from a super administrator account, so ordinary members of the development team do not have permission to create a new Namespace, and all new creation requests passed to the upper layer become very cumbersome.

   2. Tenant based on Hierarchical Namespaces: To address the problems brought by a single-level Namespace, Kubernetes has introduced the concept of hierarchical namespaces (HN). A HN is a regular Namespace indicating a single, optional parent Namespace.

      Hierarchical Namespace ownership can achieve two additional key capabilities based on Namespace ownership:

      • Inheritance of policies: If a Namespace is a subspace of another Namespace, permission policies (e.g., RBAC RoleBindings) will be directly inherited from the parent space to the subspace.
      • Inherited creation permissions: Typically, administrator permissions are required to create a Namespace. However, the hierarchical namespace provides a new scheme: only part of the permissions within the parent Namespace are needed to operate the child Namespace.

      With these two capabilities, super administrators can create a Root Namespace for a team and the necessary permission policies, then grant the team's members the right to create child Namespaces. This allows team members to create their child Namespaces without violating cluster policies.

2. Tenants in Pulsar:
   Pulsar has considered multi-tenant features since its inception and has continuously improved its implementation in the subsequent process. Multi-tenant features allow different departments to share the same set of data without having to deploy individual systems to operate the data, ensuring data consistency across departments and simplifying maintenance costs.

Pulsar uses a hierarchy of tenant, namespace, and topic to support multi-tenancy:

    1. Tenant: Tenants can span clusters and represent specific business units, product lines, core functions in organizations that are responsible for different departments or teams. Each tenant can have separate authentication and authorization mechanisms and can set storage quotas, message time-to-live (TTL), and isolation policies for tenants.

    2. Namespace: The namespace is the management unit of the tenant, and multiple namespaces can be created under each tenant. The topic under the namespace can be managed through the configuration policies set on the namespace, allowing access permissions to be set for all topics within the namespace, adjusting replication settings, managing cross-cluster, and cross-regional message replication, and controlling message expiration time.

3. Tenants in HBase:

    1. Namespace: HBase implements multi-tenant isolation through the Namespace. The Namespace is similar to the schema in relational databases, providing a logical grouping for tables and other objects. Each tenant has its own Namespace, which separates its data from that of other tenants.
   

   hbase> create_namespace 'tenant1'
   hbase> create 'tenant1:table1', 'cf1'
   

   2. Access control: HBase provides user- and role-based Access Control Lists (ACLs) for secure access. Administrators can assign specific permissions for users or user groups, such as read, write, create tables, and more. Using Namespace-level access control, tenants can be restricted to access tables within their Namespace.

   hbase> grant 'tenant1_user', 'RW', 'tenant1:table1'
   

   3. Resource isolation and quota management: HBase provides resource isolation based on Regions, allocating cluster resources to different tenants. Additionally, HBase offers quota management features, allowing administrators to set limits on read and write request rates, storage space, etc., for each tenant.

How should Inlong be implemented?

From the perspective of isolation levels, Inlong undoubtedly needs to be able to support a fully shared resource mode.

As for tenant hierarchy, it can be seen from the multi-tenant solutions of Kubernetes (k8s) and Pulsar that the introduction of multi-tier multi-tenant solutions is mainly to solve issues like multiple team authorization sets and partial configuration sharing among tenants.

For Inlong:

InlongGroup + InlongStream can already achieve partial resource isolation capability. StreamSource, StreamSink separate through the combination of InlongGroup + InlongStream, whereas, datanode and cluster can only act as shared resources. InlongGroup + InlongStream is equivalent to the sub-space in k8s or the namespace in Pulsar. Therefore, by encapsulating the concept of tenants in the current architecture, the isolation of shared resources between tenants can be achieved, reaching a two-tier tenant effect.

Furthermore, more hierarchical namespace options don't seem to be necessary. Firstly, the single-level multi-tenant model can solve most of the problems; secondly, a multi-level tenant solution would bring significant changes to the current architecture, and potentially make it challenging to smoothly upgrade existing businesses.

User Permissions

Inlong should consider two levels of user permissions.
Inlong developer/operator permissions. This type of permission has the ability to query all resources under all tenants, making it easier for Inlong to operate and troubleshoot issues as a whole.
User permissions under a tenant. For a single tenant, members under the tenant should be given simple permissions such as responsible person and follower, which are used to distinguish between resource creation/deletion and resource viewing.

The original incharges and follower fields are retained, but are no longer the smallest unit for resource operations. Instead, they indicate all owners and responsible persons for the resource under the tenant.

How to adapt to more complex multi-tenant requirements For more complex scenarios with more product hierarchy requirements, multiple levels of tenant products can be flattened into one level and mapped to Inlong. For example, tenant A , first-level product B, second-level product C, mapped to Inlong is a tenant named "A-B-C". Searches for resources under tenant A, Inlong matches the prefix "A" in the tenant field. If searching for resources under tenant A and first-level product B, the prefix matches "A-B". This way, no matter how the upper-level system plans the tenant model, it is reflected as a first-level tenant at the Inlong level.

Advantages:

Inlong's multi-tenant model is not limited by more complex upper-level systems.
The implementation is relatively simple and requires minimal changes to the current architecture.

Disadvantages:

Lack of the ability to share resources between tenants, such as the possibility of a public Hive cluster and a tenant's private Hive cluster, which is difficult to distinguish.
Prefix matching also affects search performance.

3. How to modify?

1. Modifying the tables

• Add a tenant table for maintaining basic tenant information

CREATE TABLE IF NOT EXISTS `inlong_tenant`
(
`id` int(11) NOT NULL AUTO_INCREMENT,
`tenant` varchar(256) NOT NULL COMMENT 'Namespace, not support modification',
`description` varchar(256) DEFAULT '' COMMENT 'Description of tenant',
`is_deleted` int(11) DEFAULT '0' COMMENT 'Whether to delete, 0 is not deleted, if greater than 0, delete',
`creator` varchar(256) NOT NULL COMMENT 'Creator name',
`modifier` varchar(256) DEFAULT NULL COMMENT 'Modifier name',
`create_time` datetime NOT NULL DEFAULT CURRENT_TIMESTAMP COMMENT 'Create time',
`modify_time` datetime NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP COMMENT 'Modify time',
`version` int(11) NOT NULL DEFAULT '1' COMMENT 'Version number, which will be incremented by 1 after modification',
PRIMARY KEY (`id`),
UNIQUE KEY `unique_user_role_key` (`tenant`, `is_deleted`)
) ENGINE = InnoDB

• Modify the user role table by adding the tenant field to distinguish the user's tenant. A user can have multiple records, indicating multiple tenants.

CREATE TABLE IF NOT EXISTS `tenant_user_role`
(
`id` int(11) NOT NULL AUTO_INCREMENT,
`user_name` varchar(256) NOT NULL COMMENT 'Username',
`role_code` varchar(256) NOT NULL COMMENT 'User role code',
`disabled` tinyint(1) NOT NULL DEFAULT '0' COMMENT 'Whether to disabled, 0: enabled, 1: disabled',
`is_deleted` int(11) DEFAULT '0' COMMENT 'Whether to delete, 0 is not deleted, if greater than 0, delete',
`creator` varchar(256) NOT NULL COMMENT 'Creator name',
`modifier` varchar(256) DEFAULT NULL COMMENT 'Modifier name',
`create_time` datetime NOT NULL DEFAULT CURRENT_TIMESTAMP COMMENT 'Create time',
`modify_time` datetime NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP COMMENT 'Modify time',
`tenant` varchar(256) NOT NULL COMMENT 'User tenant',
`version` int(11) NOT NULL DEFAULT '1' COMMENT 'Version number, which will be incremented by 1 after modification',
PRIMARY KEY (`id`),
UNIQUE KEY `unique_user_role_key` (`tenant`, `user_name`, `role_code`, `is_deleted`)
) ENGINE = InnoDB

• Add the inlong superuser table.

CREATE TABLE IF NOT EXISTS `inlong_user_role`
(
`id` int(11) NOT NULL AUTO_INCREMENT,
`user_name` varchar(256) NOT NULL COMMENT 'Username',
`role_code` varchar(256) NOT NULL COMMENT 'User role code',
`disabled` tinyint(1) NOT NULL DEFAULT '0' COMMENT 'Whether to disabled, 0: enabled, 1: disabled',
`is_deleted` int(11) DEFAULT '0' COMMENT 'Whether to delete, 0 is not deleted, if greater than 0, delete',
`creator` varchar(256) NOT NULL COMMENT 'Creator name',
`modifier` varchar(256) DEFAULT NULL COMMENT 'Modifier name',
`create_time` datetime NOT NULL DEFAULT CURRENT_TIMESTAMP COMMENT 'Create time',
`modify_time` datetime NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP COMMENT 'Modify time',
`version` int(11) NOT NULL DEFAULT '1' COMMENT 'Version number, which will be incremented by 1 after modification',
PRIMARY KEY (`id`),
UNIQUE KEY `unique_inlong_user_key` (`user_name`, `role_code`, `is_deleted`)
) ENGINE = InnoDB

• Add tenant field to all other tables, and add the tenant field to the unique key index.

2. Frontend modifications

• After entering the inlong main page, users should be prompted to select their tenant.
• After the user selects a tenant, all request headers need to carry the tenant information.

3. OpenApi/Managerclient modifications

• Add a tenant field to the request header.

4. Manager server-side modification

• All request operations no longer find the person in charge based on the resource's "inCharge" or "follower" fields, but instead operate based on the tenant field carried by the request. The incharges and follower fields are retained, but they no longer serve as the minimum operation unit for resources. Instead, they indicate the owner and responsible person of the resource under that tenant.
• In the authentication filter, add tenant information to userinfo for GET request subsequent db operations.
• Add a filter to determine whether the user has developer permissions for the tenant for all modification/addition

Task List

• Add RequestBodyFliter, support repeatable read buffer stream. [Feature][Manager] Support repeatable read for http request #8068
• Add Inlong_tenant table [Feature][Manager] Add definition of Inlong tenant table #8087
• Add inlong tenant related APIs [Feature][Manager] Add inlong tenant related APIs #8093
• Rename "tenant" in InlongPulsarInfo to "pulsarTenant" [Improve][Manager] Rename "tenant" in InlongPulsarInfo to "pulsarTenant" #8159
• Make dataproxy config interface compatible with old versions [Improve][Manager][DataProxy] Make DataProxy config interface compatible with old versions #8163
• Add default tenant "public" if the request does not specify one. [Feature][Manager] Add default tenant "public" if the request does not specify one #8169
• [Feature][Manager] Fliter out requests which user has no permission to the specific tenant #8200
• [Improve][Manager] TenantRole list interface support fuzzy match #8394
• [Feature][Manager] Support resource migrate to different tenant #8403
• Add tenant into the login user info. [Feature][Manager] Add tenant into the login user info #8278
• Remove the permission check logic in Services and DAOs [Feature][Manager] Remove the permission check logic in Services and DAOs #8446
• Move LoginUserUtils to pojo to avoid cycle dependency [Improve][Manager] Move LoginUserUtils to pojo to avoid cycle dependency #8296
• All resources support multi-tenancy [Umbrella][Manager] All resources support multi-tenancy #8275
• Support sql interceptor to add tenant into each query [Improve][Manager] Support SQL interceptor to add tenant into each query #8290
• Manager Client support multi-tenancy [Feature][Manager] Manager client support multi-tenancy #8281
• Support inlong user permission control [Feature][Manager] Support Inlong user permission control #8098
• Support tenant user permission control [Feature][Manager] Support tenant user permission control #8118
• Add tenant into param and body of each request [Feature][Manager] Add tenant into param and body of each request #8231

Use case

No response

Are you willing to submit PR?

• Yes, I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[dockerzhang]

@vernedeng please add a design document to show the detail.

[vernedeng]

@vernedeng please add a design document to show the detail.

done