Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

support for dividing table for internal storage #647

Open
wants to merge 2 commits into
base: main
Choose a base branch
from
Open

support for dividing table for internal storage #647

wants to merge 2 commits into from

Conversation

calvin0327
Copy link

What type of PR is this?
/kind feature

What this PR does / why we need it:

Which issue(s) this PR fixes:
Fixes #601

Special notes for your reviewer:

Does this PR introduce a user-facing change?:

support for dividing table for internal storage

@clusterpedia-bot
Copy link

Hi @calvin0327,
Thanks for your pull request!
If the PR is ready, use the /auto-cc command to assign Reviewer to Review.
We will review it shortly.

Details

Instructions for interacting with me using comments are available here.
If you have questions or suggestions related to my behavior, please file an issue against the gh-ci-bot repository.

@clusterpedia-bot clusterpedia-bot added the kind/feature New feature label Jan 29, 2024
@calvin0327
Copy link
Author

I'll test the results later.

@calvin0327 calvin0327 force-pushed the support-split-table branch 2 times, most recently from b03565f to 2fa7853 Compare January 31, 2024 08:27
@pull-request-size pull-request-size bot added size/L and removed size/M labels Jan 31, 2024
@calvin0327 calvin0327 changed the title support for dividing table for internal storage WIP: support for dividing table for internal storage Jan 31, 2024
@calvin0327 calvin0327 changed the title WIP: support for dividing table for internal storage support for dividing table for internal storage Jan 31, 2024
@calvin0327
Copy link
Author

The test results:

  1. Create pediacluster cr:
apiVersion: cluster.clusterpedia.io/v1alpha2
kind: PediaCluster
metadata:
  name: cluster-example
spec:
  apiserver: "https://10.30.43.43:6443"
  caData: 
  tokenData:
  certData: 
  keyData: 
  syncResources:
    - group: apps
      resources:
        - deployments
    - group: ""
      resources:
        - pods
  1. After runing clustersynchro-manager component and generate these tables in clusterpedia database:
image

@calvin0327
Copy link
Author

@Iceber The pr basic feature have been completed, can we have reveiw first?

theseaofstars
theseaofstars reviewed Feb 22, 2024
View reviewed changes
Copy link

@theseaofstars theseaofstars left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  1. we do not support DivisionPolicyCustom in this pr?
  2. what if the divisionpolicy is "custom" or other value?

pkg/storage/internalstorage/storage.go Outdated Show resolved Hide resolved
pkg/storage/internalstorage/storage.go Show resolved Hide resolved
add extra config for dividing table
29c0306 
Signed-off-by: calvin <wen.chen@daocloud.io>
@calvin0327 calvin0327 force-pushed the support-split-table branch 3 times, most recently from ffb80ad to 36132b6 Compare March 5, 2024 06:20
theseaofstars
theseaofstars reviewed Mar 5, 2024
View reviewed changes
Copy link

@theseaofstars theseaofstars left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if func CleanClusterResource is called when controller is not in dividing table mode. Calling generatetableFor may be incorrect.

@calvin0327 calvin0327 force-pushed the support-split-table branch 2 times, most recently from 2c74c42 to f3a0dda Compare March 12, 2024 09:19
@calvin0327
Copy link
Author

/retest

Iceber
Iceber reviewed Mar 14, 2024
View reviewed changes
Copy link
Member

@Iceber Iceber left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How to resolve table deletions is something we'll need to consider later.

examples/pediacluster.yaml Outdated Show resolved Hide resolved
pkg/storage/internalstorage/config.go Outdated Show resolved Hide resolved
pkg/storage/internalstorage/config.go Outdated Show resolved Hide resolved
pkg/storage/internalstorage/config.go Outdated Show resolved Hide resolved
pkg/storage/internalstorage/storage.go Outdated Show resolved Hide resolved
pkg/storage/internalstorage/storage.go Outdated Show resolved Hide resolved
pkg/storage/internalstorage/storage.go Outdated Show resolved Hide resolved
pkg/storage/internalstorage/register.go Show resolved Hide resolved
pkg/storage/internalstorage/resource_storage.go Show resolved Hide resolved
@calvin0327 calvin0327 force-pushed the support-split-table branch 2 times, most recently from d3eb132 to ead917e Compare March 14, 2024 13:49
@calvin0327 calvin0327 force-pushed the support-split-table branch 4 times, most recently from 2559005 to b6836a3 Compare March 21, 2024 09:57
@calvin0327
Copy link
Author

/retest

@calvin0327 calvin0327 force-pushed the support-split-table branch 2 times, most recently from 8cef7a3 to d34827e Compare March 22, 2024 06:41
@calvin0327
Copy link
Author

here are test results:

apiVersion: cluster.clusterpedia.io/v1alpha2
kind: PediaCluster
metadata:
  name: cluster-example
spec:
  apiserver: "https://10.30.43.43:6443"
  caData:
  tokenData:
  certData:
  keyData:
  syncResources:
    - group: apps
      resources:
        - deployments
    - group: ""
      resources:
        - pods

[root@master01 ~]# kubectl  get pediacluster
NAME              READY   VERSION   APISERVER
cluster-example   True    v1.29.2   https://10.6.212.13:6443
[root@master01 ~]#

# DivisionPolicy is "None"

mysql> show tables;
+------------------------+
| Tables_in_clusterpedia |
+------------------------+
| resources              |
+------------------------+
1 row in set (0.01 sec)
mysql>

[root@master01 ~]# kubectl --cluster clusterpedia get deploy -A
NAMESPACE             CLUSTER           NAME                                  READY   UP-TO-DATE   AVAILABLE   AGE
clusterpedia-system   cluster-example   clusterpedia-apiserver                1/1     1            1           3h17m
clusterpedia-system   cluster-example   clusterpedia-clustersynchro-manager   1/1     1            1           3h17m
clusterpedia-system   cluster-example   clusterpedia-controller-manager       1/1     1            1           3h17m
kube-system           cluster-example   cilium-operator                       2/2     2            2           77d
kube-system           cluster-example   coredns                               2/2     2            2           84d
kubeflow              cluster-example   training-operator                     1/1     1            1           24d
volcano-system        cluster-example   volcano-admission                     1/1     1            1           30d
volcano-system        cluster-example   volcano-controllers                   1/1     1            1           30d
volcano-system        cluster-example   volcano-scheduler                     1/1     1            1           30d

[root@master01 ~]# kubectl delete pediacluster cluster-example
pediacluster.cluster.clusterpedia.io "cluster-example" deleted

[root@master01 ~]# kubectl --cluster clusterpedia get deploy -A
Error from server (NotFound): Unable to list "apps/v1, Resource=deployments": the server could not find the requested resource (get deployments.apps)

# DivisionPolicy is "GVR"

mysql> show tables;
+------------------------+
| Tables_in_clusterpedia |
+------------------------+
| apps_v1_deployments    |
| v1_pods                |
+------------------------+
2 rows in set (0.00 sec)

mysql>

[root@master01 clusterpedia]# kubectl --cluster clusterpedia get deploy -A
NAMESPACE             CLUSTER           NAME                                  READY   UP-TO-DATE   AVAILABLE   AGE
clusterpedia-system   cluster-example   clusterpedia-apiserver                1/1     1            1           3h57m
clusterpedia-system   cluster-example   clusterpedia-clustersynchro-manager   1/1     1            1           3h57m
clusterpedia-system   cluster-example   clusterpedia-controller-manager       1/1     1            1           3h57m
kube-system           cluster-example   cilium-operator                       2/2     2            2           77d
kube-system           cluster-example   coredns                               2/2     2            2           84d
kubeflow              cluster-example   training-operator                     1/1     1            1           24d
volcano-system        cluster-example   volcano-admission                     1/1     1            1           30d
volcano-system        cluster-example   volcano-controllers                   1/1     1            1           30d
volcano-system        cluster-example   volcano-scheduler                     1/1     1            1           30d

[root@master01 clusterpedia]# kubectl delete pediacluster cluster-example
pediacluster.cluster.clusterpedia.io "cluster-example" deleted

[root@master01 clusterpedia]# kubectl --cluster clusterpedia get deploy -A
Error from server (NotFound): Unable to list "apps/v1, Resource=deployments": the server could not find the requested resource (get deployments.apps)

@calvin0327 calvin0327 force-pushed the support-split-table branch 3 times, most recently from d865443 to 599a403 Compare March 25, 2024 10:12
support split table for internal storage
723c0d2 
Signed-off-by: calvin <wen.chen@daocloud.io>
@calvin0327 calvin0327 force-pushed the support-split-table branch from 599a403 to 723c0d2 Compare March 25, 2024 10:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@Iceber Iceber Iceber left review comments

@theseaofstars theseaofstars theseaofstars left review comments

Assignees
No one assigned
Labels
kind/feature New feature size/L
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Support for storing event resources and other resources in separate tables
4 participants
@calvin0327 @clusterpedia-bot @theseaofstars @Iceber