Add SIG Testing charter #2777

spiffxp · 2018-10-09T18:45:32Z

/cc @fejta @stevekuznetsov @timothysc
sig chairs

FYI @kubernetes/sig-testing

krzyzacy · 2018-10-09T18:58:27Z

sig-testing/charter.md

+- We are not resonpsible for troubleshooting or writing tests or jobs for 
+  features owned by other SIGs
+- We are not responsible for ongoing maintenance of kubernetes' e2e test 
+  framework


just out of curiosity, is there any ongoing plan for the e2e framework?

not at present, we had a long chat about this during the most recent sig-testing meeting, and are opting to keep it in scope but not really actively staffed, so I'd like to see if it's worth pointing this out here

spiffxp · 2018-10-09T18:58:53Z

sig-testing/charter.md

+
+### Deviations from [sig-governance]
+
+- Chairs also fulfill the role of Tech Lead


For discussion: I've seen most SIGs do this but I think I'd rather see the roles explicitly split

spiffxp · 2018-10-09T18:59:27Z

sig-testing/charter.md

+
+- We are not resonpsible for troubleshooting or writing tests or jobs for 
+  features owned by other SIGs
+- We are not responsible for ongoing maintenance of kubernetes' e2e test 


For discussion: maybe this falls under the testing-commons subproject? But in terms of staffing I don't see ongoing maintenance

discussed during today's meeting, we'll call it out as owned by this sig, but unstaffed... to paraphrase the meeting, we know where at least some of the bodies are buried, but that doesn't mean there were no more bodies buried on top

spiffxp · 2018-10-09T19:00:26Z

sig-testing/charter.md

+  the maintenance of release jobs
+- When rolling out changes that may potentially impact the project as a whole
+  we consult with SIG Contributor Experience, and follow [lazy consensus] by 
+  notifying kubernetes-dev, providing a deadline,and a rationale for the 


For discussion: this is really squishy, how can we firm this up

Seems accurate to the process we use, though -- do you mean we should firm up the process or this doc?

spiffxp · 2018-10-09T19:01:11Z

sig-testing/charter.md

+- Tools that facilitate local testing of kubernetes such as [greenhouse] 
+  and [kind]
+- Jobs that automate away project toil via [@fejta-bot]
+- Ensuring all of the above is kept running on a best effort basis


For discussion: I eventually want this to go to the CNCF / infra WG once we have handed infra to them, but for now this is where it belongs IMO

krzyzacy · 2018-10-09T18:59:31Z

sig-testing/charter.md

+- Display and analysis of test artifacts via tools like [gubernator], 
+  [testgrid], [triage] and [velodrome]
+- Configuration management of jobs and ensuring they use a consistent
+  process via tools such as [job configs], [kubetest]


would boskos fall here or under project CI w/ prow?

boskos is not part of prow... it's built in with kubetest for gcp project provision, you can also write customer applications for other resource provision purposes, for example, cluster provisioning (mason)

I added boskos to "infrastructure to support CI at scale"

krzyzacy · 2018-10-09T19:01:17Z

sig-testing/charter.md

+- Configuration management of jobs and ensuring they use a consistent
+  process via tools such as [job configs], [kubetest]
+- Tools that facilitate local testing of kubernetes such as [greenhouse] 
+  and [kind]


ack, planter is what I meant, not greenhouse

greenhouse, github cache etc are probably also in scope under another area ~nebulously infrastructure / automation, not sure how many things we really need to call out though.

replaced with planter and put greenhouse, boskos, ghproxy under a different bullet to support CI at our scale

krzyzacy · 2018-10-09T19:01:52Z

sig-testing/charter.md

+
+#### Cross-cutting and Externally Facing Processes
+
+- The Release Team [test-infra role] is staffed by a memer of SIG Testing, as


s/memer/member

I dunno I kinda like:

The Release Team [test-infra role] is staffed by a memer of SIG Testing

🙃

cjwagner · 2018-10-09T19:00:49Z

sig-testing/charter.md

+
+## Roles and Organization Management
+
+This sig follows adheres to the Roles and Organization Management outlined in 


nit: "follows adheres" is redundant.

cjwagner · 2018-10-09T19:01:27Z

sig-testing/charter.md

+
+
+[sig-governance]: https://github.com/kubernetes/community/blob/master/committee-steering/governance/sig-governance.md
+[sig-subprojects]: https://github.com/kubernetes/community/blob/master/sig-YOURSIG/README.md#subprojects


This is unused.

ack, will delete

cjwagner · 2018-10-09T19:02:11Z

sig-testing/charter.md

+## Roles and Organization Management
+
+This sig follows adheres to the Roles and Organization Management outlined in 
+[sig-governance] and opts-in to updates and modifications to [sig-governance].


Should these both be sig-governance?

this is copy-pasta from the template, but I think it makes sense to be explicit here

stevekuznetsov · 2018-10-10T06:16:07Z

sig-testing/charter.md

+  the maintenance of release jobs
+- When rolling out changes that may potentially impact the project as a whole
+  we consult with SIG Contributor Experience, and follow [lazy consensus] by 
+  notifying kubernetes-dev, providing a deadline,and a rationale for the 


Seems accurate to the process we use, though -- do you mean we should firm up the process or this doc?

stevekuznetsov · 2018-10-10T06:17:44Z

sig-testing/charter.md

+[kubetest]: https://git.k8s.io/test-infra/kubetest
+[kind]: https://github.com/kubernetes-sigs/kind
+[@fejta-bot]: https://github.com/fejta-bot
+[greenhouse]: https://git.k8s.io/test-infra/greenhouseu


I think this should have been planter, but FYI the link is broken

timothysc · 2018-10-11T00:10:15Z

sig-testing/charter.md

+## Scope
+
+SIG Testing is interested in effective testing of Kubernetes.  We do not write or
+troubleshoot the project's tests, but instead focus on tooling that makes it


We often do troubleshoot though...

It should be presented as an act of goodwill, though, and best-effort.

I'm rewording to be a little more explicit that we should be the last ask, not the first ask, and our goal is remediation rather than owning a fix.

timothysc · 2018-10-11T00:56:14Z

sig-testing/charter.md

+- Project CI and merge automation via tools such as [prow] and [tide]
+- Extraction of test results from GCS and populating a public accessible
+  BigQuery dataset via [kettle]
+- Display and analysis of test artifacts via tools like [gubernator], 


Are all of the tools totally opensourced now? I thought parts of test-grid were still not open?

Open sourcing testgrid is still the plan, but for now I would file this under "services"

In terms of code, the open source pieces we own are yaml config, and the yaml->proto conversion code. I'm not sure whether we "own" all of the dashboards though. I would consider it analogous to how we own config management of jobs, but encourage individual owners to write their own jobs

We could consider testgrid a google-staffed subproject of sig-testing if there is concern that this places an undue burden on members who can't update all parts of its code.

timothysc · 2018-10-11T00:58:44Z

sig-testing/charter.md

+
+- We are not resonpsible for troubleshooting or writing tests or jobs for 
+  features owned by other SIGs
+- We are not responsible for ongoing maintenance of kubernetes' e2e test 


I would nix this statement, b/c it falls under the 1st part of the scope.

ixdy · 2018-10-11T05:08:14Z

sig-testing/charter.md

+- Extraction of test results from GCS and populating a public accessible
+  BigQuery dataset via [kettle]
+- Display and analysis of test artifacts via tools like [gubernator], 
+  [testgrid], [triage] and [velodrome]


link for velodrome is missing

ixdy · 2018-10-11T05:09:55Z

sig-testing/charter.md

+[Kubernetes Charter README]: https://github.com/kubernetes/community/blob/master/committee-steering/governance/README.md
+
+[gubernator]: http://k8s-gubernator.appspot.com
+[kettle]: https:/git.k8s.io/test-infra/kettle


you're missing a / after https:/

ixdy · 2018-10-11T05:10:49Z

sig-testing/charter.md

+
+#### Cross-cutting and Externally Facing Processes
+
+- The Release Team [test-infra role] is staffed by a memer of SIG Testing, as


this link is also missing

spiffxp · 2018-10-11T18:31:40Z

/committee steering

spiffxp · 2018-10-13T22:23:39Z

/hold
because I'm going to remove the wip

PTAL, I consider this ready to ask steering to review if there are no more objections

neolit123 · 2018-10-16T09:35:50Z

SGTM, nice to see a good charter that will merge fast.

spiffxp · 2018-10-16T18:43:23Z

/assign @dims @brendandburns
For steering committee review

dims · 2018-10-16T19:08:52Z

sig-testing/charter.md

+  features or subprojects owned by other SIGs
+- We are not responsible for ongoing maintenance of the project's CI Signal,
+  as this is driven by tests and jobs owned by other SIGs. We do however have
+  an interest in producing tools to help improve the signal.


Do you want to add stuff like on-call are out of scope too?

Is there something you want to add about turn around times for things when they break? (can't guarantee is fine!)

I guess I'd rather not get too specific about it if I don't need to. I stated as "in scope" that we support our infrastructure on a best effort basis. I aspire for us to have more than that in terms of rigor or metrics, but this is where we are today.

If I have to spell it out, it would be something like:

As stated above, we support and maintain the project's testing and automation infrastructure on a best effort basis. As such, we do not provide response time guarantees or off-hours support, though we do understand the importance of keeping a global high-traffic open source project moving.

let's see if anyone else feels like it should be.

fejta

/lgtm
/hold

dims · 2018-10-17T10:55:50Z

/lgtm

dims · 2018-10-17T15:12:58Z

@brendandburns This looks like its good to go

cblecker · 2018-10-17T19:16:41Z

Haven't had a chance to look through this until now. Fantastic work! I love it.

brendandburns

Small stuff. Approved, but would love the cleanups.

brendandburns · 2018-10-18T18:55:26Z

sig-testing/charter.md

+contribute, analyze and act upon test results.
+
+We are not responsible for writing, fixing, nor actively troubleshooting the
+project's tests, as this is the responsibility of the respective test, feature,


shorten to:

"is the responsibility of the owning SIG"

brendandburns · 2018-10-18T18:57:16Z

sig-testing/charter.md

+
+### Out of scope
+
+- We are not responsible for troubleshooting or writing tests or jobs for


This is redundant with the statement in 'Scope' I'd put it in one place or the other but not both.

I want at least some kind of call out up top, but linked down here for the details

brendandburns · 2018-10-18T18:59:41Z

sig-testing/charter.md

+[Kubernetes Charter README]: https://github.com/kubernetes/community/blob/master/committee-steering/governance/README.md
+[lazy consensus]: http://en.osswiki.info/concepts/lazy_consensus
+
+[@fejta-bot]: https://github.com/fejta-bot


This is a pointer to a robot user. Is it really a sub-project? Maybe it really ought to be a pointer to the code behind the bot?

redid to link to the periodic jobs

spiffxp · 2018-10-18T21:10:52Z

@brendandburns updated, PTAL

dims · 2018-10-19T00:17:46Z

/lgtm

reads well.

timothysc · 2018-10-19T15:57:52Z

sig-testing/charter.md

+- Tools, frameworks and libraries that make it possible to write tests against
+  kubernetes such as e2e\* or integration test frameworks.
+
+  \* Note that while we are the current de facto owners of the kubernetes e2e


This is gr8, thx for the edit.

timothysc · 2018-10-19T15:58:55Z

nice work, LGTM from me.

spiffxp · 2018-10-19T17:18:31Z

/approve
self-approving

brendanburns [5:03 PM]
I'll try to look tonight, but if I don't feel free to merge w/o my response

k8s-ci-robot · 2018-10-19T17:18:38Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: spiffxp

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~sig-testing/OWNERS~~ [spiffxp]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

spiffxp · 2018-10-19T17:57:19Z

/hold cancel
Forgot to remove the /hold as well

Initial stab at SIG Testing charter

72e4340

k8s-ci-robot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Oct 9, 2018

k8s-ci-robot requested review from fejta and stevekuznetsov October 9, 2018 18:45

k8s-ci-robot added the size/M Denotes a PR that changes 30-99 lines, ignoring generated files. label Oct 9, 2018

k8s-ci-robot requested a review from timothysc October 9, 2018 18:45

k8s-ci-robot added cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. sig/testing Categorizes an issue or PR as relevant to SIG Testing. labels Oct 9, 2018

krzyzacy reviewed Oct 9, 2018

View reviewed changes

spiffxp commented Oct 9, 2018

View reviewed changes

krzyzacy reviewed Oct 9, 2018

View reviewed changes

cjwagner reviewed Oct 9, 2018

View reviewed changes

stevekuznetsov reviewed Oct 10, 2018

View reviewed changes

timothysc reviewed Oct 11, 2018

View reviewed changes

ixdy reviewed Oct 11, 2018

View reviewed changes

k8s-ci-robot added the committee/steering Denotes an issue or PR intended to be handled by the steering committee. label Oct 11, 2018

jdumars mentioned this pull request Oct 11, 2018

Charters Meta-issue kubernetes/steering#31

Closed

Address review comments

20fe5a3

k8s-ci-robot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. and removed size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Oct 13, 2018

spiffxp force-pushed the sig-testing-charter branch from fbda4a5 to 553d5d1 Compare October 13, 2018 22:17

k8s-ci-robot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Oct 13, 2018

spiffxp changed the title ~~[wip] Initial stab at SIG Testing charter~~ Add SIG Testing charter Oct 13, 2018

neolit123 approved these changes Oct 16, 2018

View reviewed changes

k8s-ci-robot removed the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Oct 16, 2018

k8s-ci-robot assigned brendandburns and dims Oct 16, 2018

Added comments re: workflow and automation, linkable headers

0bac6c6

spiffxp force-pushed the sig-testing-charter branch from c4fb635 to 0bac6c6 Compare October 16, 2018 18:45

dims reviewed Oct 16, 2018

View reviewed changes

fejta approved these changes Oct 16, 2018

View reviewed changes

k8s-ci-robot assigned fejta Oct 16, 2018

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Oct 16, 2018

brendandburns approved these changes Oct 18, 2018

View reviewed changes

Address brendandburns review comments

1e83f00

k8s-ci-robot removed the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Oct 18, 2018

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Oct 19, 2018

timothysc reviewed Oct 19, 2018

View reviewed changes

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Oct 19, 2018

k8s-ci-robot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Oct 19, 2018

k8s-ci-robot merged commit f54169e into kubernetes:master Oct 19, 2018

spiffxp deleted the sig-testing-charter branch October 19, 2018 18:20


		### Deviations from [sig-governance]

		- Chairs also fulfill the role of Tech Lead


		#### Cross-cutting and Externally Facing Processes

		- The Release Team [test-infra role] is staffed by a memer of SIG Testing, as


		## Roles and Organization Management

		This sig follows adheres to the Roles and Organization Management outlined in



		[sig-governance]: https://github.com/kubernetes/community/blob/master/committee-steering/governance/sig-governance.md
		[sig-subprojects]: https://github.com/kubernetes/community/blob/master/sig-YOURSIG/README.md#subprojects


		### Out of scope

		- We are not responsible for troubleshooting or writing tests or jobs for

Add SIG Testing charter #2777

Add SIG Testing charter #2777

Conversation

spiffxp commented Oct 9, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

spiffxp commented Oct 11, 2018

spiffxp commented Oct 13, 2018

neolit123 commented Oct 16, 2018

spiffxp commented Oct 16, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

fejta left a comment

Choose a reason for hiding this comment

dims commented Oct 17, 2018

dims commented Oct 17, 2018

cblecker commented Oct 17, 2018 • edited Loading

brendandburns left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

spiffxp commented Oct 18, 2018

dims commented Oct 19, 2018

Choose a reason for hiding this comment

timothysc commented Oct 19, 2018

spiffxp commented Oct 19, 2018

k8s-ci-robot commented Oct 19, 2018

spiffxp commented Oct 19, 2018

cblecker commented Oct 17, 2018 •

edited

Loading