TestGrid Open-Sourcing Planning

[michelle192837]

ETA (7/25/19): We've got some updates! Check it below

tl;dr: If you're interested in TestGrid being open-sourced, add your comments, support, and use cases here!

ETA: While TestGrid open-source is in the works, you can add your results to testgrid.k8s.io by following the normal process. Your results will need to conform to expected results for now; I'll update when more possibilities are available.

I'm pulling together a roadmap for TestGrid development in 2019! (Since I'll be on vacation in the latter half of December, I'm drafting it this week, and the final one will get posted in January). There's been scattered support, comments, and bugs for TestGrid open-sourcing, but I want to collect all that in one bug for convenience's sake.

So! Comments! If you have details on what you need specifically (i.e. what parts are most important to prioritize, like being able to stand up a separate frontend, upload results, run your own updaters, etc.), or on what you'd be using TestGrid for, note that too. The more comments the better!

Related issues: #3324, #3323

/kind feature
/area testgrid

[BenTheElder]

cc @munnerz @stevekuznetsov

[stevekuznetsov]

/cc @smarterclayton

[stevekuznetsov]

Talked with @michelle192837 in person today - would be great to have a quick overview of the architecture of TestGrid and current progress.

From a naive opinion it seems like we could process our test results and upload our own proto for a third party front end to use, that would need minimal effort to get us to dashboards.

[donaldguy]

👋 I work on AKS; (likely @juan-lee and) I would be interested in using it very much like upstream for our staging edge

Btw, I am also happy to talk about getting our public face on "real" test-grid

I suspect @jackfrancis and co might be similarly interested vis-a-vis Azure/aks-engine

[rdrgmnzs]

I would be in using testgrid internally, currently to be able to visualize the current integration tests I have setup to run against my clusters. The alerting portion if tests fail consecutively is also something I'm looking forward to having.

[neolit123]

would testgrid go in a separate repo?

[michelle192837]

Thanks for the comments so far all! Keep adding if you've got them. ^^

@neolit123 I think ideally, since conceptually it's not k8s-specific? No idea on if/when it would be in a separate repo though.

[neolit123]

i think i see it as a kubernetes/testgrid repo.
my concern here is to avoid the potential bump in volume of test-infra, since it's already one of the higher volume repos under the k org, after k/k and k/website.

[michelle192837]

Thanks for comments so far all! (Feel free to continue adding to this, or encourage others to).

Just got back from vacation, so I'll be pulling together the plan for early 2019 and will publish that publicly when it's more finalized.

[ffledgling]

Adding my use case here after discussion with @michelle192837/@amerai I have a bunch of clusters that running internally and am starting to run the kubernetes e2e conformance tests against them periodically. There are also internal tests added into the same framework to test for custom stuff. I'd like very much to use testgrid as the overview dashboard for these periodic tests and as an indicator of overall cluster health and compliance. A self-hostable version of testgrid would be very very appreciated (even if it initially requires setting up a private GCP buckets etc).

I'd be happy to act as beta/alpha tester for on-perm/self-hosted testgrid in case that helps!

[jstrachan]

would love to use testgrid too (in our case to vsualise all the different system tests running on different kubernetes clusters & configs in Jenkins X).

Even before the code becomes OSS, if there are public usable docker images for the testgrid containers I’d be happy to try create helm charts to make them easy to use on k8s (though I may need a hand figuring out how to wire it all together)

[michelle192837]

Just a quick update; I don't have much externally for a 2019 roadmap atm (been finishing up some work from last year, and going on vacation for 3 weeks in Feb), but I promise I'm still working on this and do intend to do some work in GitHub when I'm back in March.

Thanks again! ^^

(In the meanwhile, if someone's particularly itching to enhance TestGrid, feel free to take a look at #10701 )

[sjanota]

We would like to enhance Prow instance for Kyma with TestGrid. The part we find most useful is a clear view of test stability: how often they fail, easy access to logs and alerting.

[rdrgmnzs]

Sorry to pester but do we have any new info on what the status of this is? I'd really love to have testgrid available for internal use.

[michelle192837]

Np! At the moment, quietly chugging along; I'll probably have a better update around late June/early July. I and others are doing some way-paving at the moment (internal and external), but still no real specifics at the moment.

…

On Mon, Jun 10, 2019, 10:11 AM Rodrigo Menezes ***@***.***> wrote: Sorry to pester but do we have any new info on what the status of this is? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#10409?email_source=notifications&email_token=AAE7MNN4HHV6Z2I6J6NZC2TPZ2DMVA5CNFSM4GJYWGDKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODXKQG3Q#issuecomment-500499310>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAE7MNLOPOH2R77IBPGSWO3PZ2DMVANCNFSM4GJYWGDA> .

[xiaoxubeii]

Sorry to pester but do we have any new info on what the status of this is? I'd really love to have testgrid available for internal use.

+1, there is little tools in e2e test results visualization, and i think testgrid is definitely helpful.

[michelle192837]

Hey everyone, and thanks for your patience! :D Here's our rough timeline for open-sourcing. This covers the next couple quarters (there will be more work to do next year, I'll update when we're further along). Plans subject to change, but feel free to poke if it looks like we're lagging on this. ^^

Q3 (July - September 2019):

• Move TestGrid project + code into its own repo
• Add all post-update jobs
• Entomologist (finds issues associated with tests)
• Summarizer (creates a summary for a dashboard tab)
• Alerter (sends email alerts based on summaries)
• Use open-source code in production services

Q4 (October - December 2019):

• Add updater (gathers test results and turns into TestGrid state)
• PubSub update requester (monitor GCS, request updates on changes)
• Results gatherer (gathers/processes Prow's JUnit results)
• State updater (translates processed results into state.proto)

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[cblecker]

/remove-lifecycle stale

[zcain117]

Has the open-sourcing progressed far enough that we can create a Testgrid tab or dashboard? We're running our tests as Kubernetes workloads that are kicked off by cronjob. We want to visualize the pass/fail history for each test. Alerting and issue creation would be nice to have but not necessary.

[michelle192837]

Theoretically, I think you could have something obfuscate your private results sufficiently for you to be confident in uploading them publicly. The public logs need enough information to show up in TestGrid (as outlined in https://github.com/kubernetes/test-infra/tree/master/gubernator#job-artifact-gcs-layout). But beyond that, you don't need to add logs or anything, you'll just have less useful information (and you won't have the automatic support of Prow's decorated jobs uploading all the related logs and files for you; you'd need some obfuscation + upload step(s)).

[devbharat]

Thanks a lot, really appreciate it. I guess my last question would be an expected timeline for an alpha version coming out. This link above seems to go part of the way there.

I note that while the TestGrid concept of hierarchically organizing test results and displaying historical trends of these results seems very general, most of the current users seem to be serving k8s community itself. I work with the PX4 opensource autopilot community (and their closed source adopters), and am developing tools to run end-to-end system tests involving various component repositories (the AutoPilot(itself composed of many different componets), flight simulator, ground control software, payload interface etc) and using Prow to execute these tests, and TestGrid to provide the overview & status makes a lot of sense to me. Unless I am missing something in the details (still haven't completely onboard myself), Prow and TestGrid could find a lot of traction in this community.

[michelle192837]

Heh, that link is definitely a bit out of date. We're continuing to work on open-sourcing, but we'll announce more details on the repo itself more likely (https://github.com/GoogleCloudPlatform/testgrid).

That sounds awesome, yeah! There is an updater that's being worked on in GoogleCloudPlatform/testgrid, if you want to take a look; that might give some more insight into how we parse test formats and whatnot. (Just to be clear, the updater in GoogleCloudPlatform/testgrid is not what does the updates in testgrid.k8s.io at the moment, but we're working on making it that, and it's still useful to look at in the meanwhile.)

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[michelle192837]

/remove-lifecycle stale

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[aojea]

/remove-lifecycle stale
/cc

[cpanato]

+1 to opensource

Some use case I would like is to have an option to open a dashboard (like https://testgrid.k8s.io/sig-release-master-blocking#gce-cos-master-default) and save that as a png/jpg image, maybe passing a URL param, eg https://testgrid.k8s.io/sig-release-master-blocking#gce-cos-master-default?image=png

thanks!

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle rotten

[midnightconman]

/remove-lifecycle rotten

…

On Sun, Mar 7, 2021, 02:41 fejta-bot ***@***.***> wrote: Stale issues rot after 30d of inactivity. Mark the issue as fresh with /remove-lifecycle rotten. Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with /close. Send feedback to sig-contributor-experience at kubernetes/community <https://github.com/kubernetes/community>. /lifecycle rotten — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#10409 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AANH4O5XFOCSCC2ZKXLI7ZDTCNJ5DANCNFSM4GJYWGDA> .

[rahulbharuka]

Do we have S3 support in the roadmap ? Currently, looks like Testgrid works only with GCS.
https://github.com/GoogleCloudPlatform/testgrid/blob/master/util/gcs/gcs.go#L94

[michelle192837]

Not at the moment; it's not something we have the bandwidth to implement, but contributions are welcome in that respect! I'd recommend opening an issue in https://github.com/GoogleCloudPlatform/testgrid itself to track, and tagging with help-wanted.

[rahulbharuka]

Thanks. Created GoogleCloudPlatform/testgrid#489

[aijamalnk]

Hi @michelle192837 - I have a couple questions out of curiosity, as I'm working with knative community.

What is the current status of testgrid's open sourcing? Can it be run fully with the code in https://github.com/GoogleCloudPlatform/testgrid? Or is it still partially closed-source? Is there an ETA / target date / estimate for when it may be fully out?

[spiffxp]

/sig testing

[chases2]

Hi @michelle192837 - I have a couple questions out of curiosity, as I'm working with knative community.

What is the current status of testgrid's open sourcing? Can it be run fully with the code in https://github.com/GoogleCloudPlatform/testgrid? Or is it still partially closed-source? Is there an ETA / target date / estimate for when it may be fully out?

The backend can be run completely in open-source on kubernetes. https://github.com/GoogleCloudPlatform/testgrid/blob/master/standalone.md#setting-up-testgrid has specific details on this, with some examples.

[mykaul]

Can the readme update to reflect the latest status of the open sourcing initiative? It still points to this PR.

[palnabarun]

Is there a plan to open-source the Testgrid frontend? I think that would be useful for other projects as well.