Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[KIC-2.0] performance test cases and data collection #1488

Merged
merged 18 commits into from
Jul 9, 2021
Merged

Conversation

ccfishk
Copy link
Contributor

@ccfishk ccfishk commented Jul 5, 2021

What this PR does / why we need it:

Implement ingress performance test cases and collect test data
Which issue this PR fixes **: fixes #
part of #1197
The performance test cases calculate ingress time consuming after creation until ingress status is changed to ready.
Special notes for your reviewer:
Ingress performance test cases
PR Readiness Checklist:

Complete these before marking the PR as ready to review:
Performance test cases are able to execute and collecting data.

@ccfishk ccfishk requested a review from a team as a code owner July 5, 2021 06:26
@ccfishk ccfishk temporarily deployed to Configure ci July 5, 2021 06:26 Inactive
@ccfishk ccfishk changed the title add performance tests and collecting data based on tests for both 1.3.0 and 2.0.x [WIP] add performance tests and collecting data based on tests for both 1.3.0 and 2.0.x Jul 5, 2021
@codecov
Copy link

codecov bot commented Jul 5, 2021

Codecov Report

Merging #1488 (6f3d50c) into next (f42f1c9) will increase coverage by 0.12%.
The diff coverage is n/a.

Impacted file tree graph

@@            Coverage Diff             @@
##             next    #1488      +/-   ##
==========================================
+ Coverage   51.40%   51.53%   +0.12%     
==========================================
  Files          91       91              
  Lines        6316     6316              
==========================================
+ Hits         3247     3255       +8     
+ Misses       2774     2769       -5     
+ Partials      295      292       -3     
Flag Coverage Δ
integration-test 48.51% <ø> (+0.23%) ⬆️
unit-test 38.93% <ø> (+0.04%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
...trollers/configuration/zz_generated_controllers.go 48.41% <0.00%> (+0.57%) ⬆️
pkg/parser/parser.go 84.51% <0.00%> (+1.25%) ⬆️
railgun/internal/ctrlutils/ingress-status.go 62.93% <0.00%> (+1.29%) ⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update f42f1c9...6f3d50c. Read the comment docs.

@ccfishk ccfishk temporarily deployed to Configure ci July 6, 2021 15:49 Inactive
@ccfishk ccfishk temporarily deployed to Configure ci July 6, 2021 15:49 Inactive
@ccfishk ccfishk changed the title [WIP] add performance tests and collecting data based on tests for both 1.3.0 and 2.0.x [KIC] add performance tests and collecting data based on incluster deployment Jul 6, 2021
@ccfishk ccfishk changed the title [KIC] add performance tests and collecting data based on incluster deployment [KIC] add performance test cases and data collect based on within cluster deployment Jul 6, 2021
@ccfishk ccfishk temporarily deployed to Configure ci July 6, 2021 18:17 Inactive
@ccfishk ccfishk temporarily deployed to Configure ci July 6, 2021 18:17 Inactive
Copy link
Contributor

@shaneutt shaneutt left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ideally I was hoping any performance tooling we add would be something we could trigger with a CLI, or perhaps a docker container, e.t.c. This mechanism will lock performance testing tightly into our testing framework, which does not seem to align well with some of the stated goals in #1197.

Particularly, the main goal stated there:

We want to create a performance testing tool that can be applied generically with any particular Kubernetes cluster backend.

Building the performance testing into our KIC tests will specifically not be generic as it will mean:

  1. anyone wishing to run the performance tests needs to be educated on our git repo
  2. anyone wishing to run the performance tests needs to understand go test

And also we currently only support kind on the github actions infrastructure, which not only doesn't seem like a strong testing environment (I believe this environment is highly contentious for compute resources being on shared infrastructure) but also doesn't allow for any other cluster options as per the any particular Kubernetes cluster backend bit.

My recommendation would be for us to transplant the functionality that is currently in railgun/test/integration into a discreet library which could then be consumed by the integration tests for basic validation and regression testing, but also then can be consumed more generally by some CLI or other caller who can bring whatever Kubernetes cluster (via *rest.Config) to the table they want to.

railgun/test/integration/perf_ingress_test.go Outdated Show resolved Hide resolved
railgun/test/integration/perf_ingress_test.go Outdated Show resolved Hide resolved
railgun/test/integration/perf_ingress_test.go Outdated Show resolved Hide resolved
railgun/test/integration/perf_ingress_test.go Outdated Show resolved Hide resolved
railgun/test/integration/perf_ingress_test.go Outdated Show resolved Hide resolved
railgun/test/integration/perf_ingress_test.go Outdated Show resolved Hide resolved
railgun/test/integration/perf_ingress_test.go Outdated Show resolved Hide resolved
railgun/test/integration/perf_ingress_test.go Outdated Show resolved Hide resolved
railgun/test/integration/perf_ingress_test.go Outdated Show resolved Hide resolved
railgun/test/integration/perf_ingress_test.go Outdated Show resolved Hide resolved
@ccfishk ccfishk changed the title [KIC] add performance test cases and data collect based on within cluster deployment [kic-2.0] add performance test cases and data collect based on within cluster deployment Jul 7, 2021
railgun/test/performance/perf_ingress_test.go Outdated Show resolved Hide resolved
railgun/test/performance/perf_ingress_test.go Outdated Show resolved Hide resolved
@ccfishk ccfishk changed the title [kic-2.0] add performance test cases and data collect based on within cluster deployment WIP [kic-2.0] add performance test cases and data collect based on within cluster deployment Jul 8, 2021
@ccfishk ccfishk changed the title WIP [kic-2.0] add performance test cases and data collect based on within cluster deployment WIP [KIC-2.0] add performance test cases and data collect based on within cluster deployment Jul 8, 2021
Co-authored-by: Shane Utt <shaneutt@linux.com>
@ccfishk ccfishk temporarily deployed to Configure ci July 8, 2021 16:51 Inactive
@ccfishk ccfishk temporarily deployed to Configure ci July 8, 2021 16:51 Inactive
@github-actions
Copy link

github-actions bot commented Jul 8, 2021

Licenses differ between commit 78e5c43215900106af20cfc46de0de8311df3fa2 and base:

+++ pr_licenses.csv	2021-07-08 16:53:39.245364864 +0000
@@ -1,8 +1,6 @@
 github.com/Masterminds/goutils,https://github.com/Masterminds/goutils/blob/master/LICENSE.txt,Apache-2.0
 github.com/Masterminds/semver,https://github.com/Masterminds/semver/blob/master/LICENSE.txt,MIT
 github.com/Masterminds/sprig,https://github.com/Masterminds/sprig/blob/master/LICENSE.txt,MIT
-github.com/alecthomas/template,https://github.com/alecthomas/template/blob/master/LICENSE,BSD-3-Clause
-github.com/alecthomas/units,https://github.com/alecthomas/units/blob/master/COPYING,MIT
 github.com/beorn7/perks/quantile,https://github.com/beorn7/perks/blob/master/quantile/LICENSE,MIT
 github.com/blang/semver/v4,https://github.com/blang/semver/blob/master/v4/LICENSE,MIT
 github.com/bombsimon/logrusr,https://github.com/bombsimon/logrusr/blob/master/LICENCE,MIT
@@ -73,7 +71,6 @@
 go.uber.org/zap,Unknown,MIT
 gomodules.xyz/jsonpatch/v2,Unknown,Apache-2.0
 google.golang.org/protobuf,Unknown,BSD-3-Clause
-gopkg.in/alecthomas/kingpin.v2,Unknown,MIT
 gopkg.in/go-playground/pool.v3,Unknown,MIT
 gopkg.in/inf.v0,Unknown,BSD-3-Clause
 gopkg.in/ini.v1,Unknown,Apache-2.0```

@ccfishk ccfishk temporarily deployed to Configure ci July 9, 2021 02:19 Inactive
@ccfishk ccfishk temporarily deployed to Configure ci July 9, 2021 02:19 Inactive
@ccfishk ccfishk temporarily deployed to Configure ci July 9, 2021 02:20 Inactive
@ccfishk ccfishk temporarily deployed to Configure ci July 9, 2021 02:20 Inactive
@shaneutt shaneutt modified the milestones: Blockers for cutting KIC 2.0-beta.1, POST KIC 2.0 Jul 9, 2021
Co-authored-by: Shane Utt <shaneutt@linux.com>
@ccfishk ccfishk temporarily deployed to Configure ci July 9, 2021 14:16 Inactive
@ccfishk ccfishk temporarily deployed to Configure ci July 9, 2021 14:16 Inactive
Co-authored-by: Shane Utt <shaneutt@linux.com>
@ccfishk ccfishk temporarily deployed to Configure ci July 9, 2021 14:17 Inactive
@ccfishk ccfishk temporarily deployed to Configure ci July 9, 2021 14:17 Inactive
@ccfishk ccfishk temporarily deployed to Configure ci July 9, 2021 14:20 Inactive
@ccfishk ccfishk temporarily deployed to Configure ci July 9, 2021 14:20 Inactive
@ccfishk ccfishk requested a review from shaneutt July 9, 2021 14:25
@ccfishk ccfishk removed the area/debt label Jul 9, 2021
@ccfishk ccfishk temporarily deployed to Configure ci July 9, 2021 17:49 Inactive
@ccfishk ccfishk temporarily deployed to Configure ci July 9, 2021 17:49 Inactive
Copy link
Contributor

@shaneutt shaneutt left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think given the context that we're still in the experimenting stage with performance testing and have several follow up issues, plus the fact that this code is behind it's own build tag this is a good starting point: we can merge what we have now, and iterate on it as we further decide as a team what new features we want.

@ccfishk ccfishk temporarily deployed to Configure ci July 9, 2021 21:56 Inactive
@ccfishk ccfishk temporarily deployed to Configure ci July 9, 2021 21:56 Inactive
@ccfishk ccfishk merged commit d88ce02 into next Jul 9, 2021
@ccfishk ccfishk deleted the performance_test branch July 9, 2021 22:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants