Add Framework for Conformance Tests #98

tssurya · 2023-05-17T12:48:36Z

NOTE: Most of the framework is blatantly adopted from https://github.com/kubernetes-sigs/gateway-api/tree/main/conformance gateway-api code without reinventing the wheel so that we can also adopt the new conformance profile GEP. Of course I have modified some of this to suite the needs of netpol conformance. Kudos/Credit to the gateway-api team.

NOTE2: This PR only adds the test framework. The actual tests will be added in a second PR.

NOTE3: I contemplated using gateway-api imports into this repo to refer to their conformance test utilities but that didn't seem like the right thing to do because moving forward we would still need to customize some of these utils according to how we modify our tests and so on. @shaneutt has brought up some great points around consolidating some of these pieces into a separate repo that both parties and more parties in future can use extensibly but that is something to ponder in the future.

Sample Output:

[surya@hidden-temple network-policy-api]$ make conformance                                                                                                                   
go test -v ./conformance/... -args                                                                                                                                           
=== RUN   TestConformance                                                                                                                                                    
    conformance_test.go:56: Running conformance tests with cleanup: true                                                                                                     
         debug: false                                                                                                                                                        
         enable all features: false                                                                                                                                          
         supported features: []                                                                                                                                              
         exempt features: []                                                                                                                                                 
    suite.go:116: Test Setup: Applying base manifests                                                                                                                        
    apply.go:124: Creating network-policy-conformance-gryffindor Namespace                                                                                                   
    apply.go:124: Creating network-policy-conformance-slytherin Namespace                                                                                                    
    apply.go:124: Creating network-policy-conformance-hufflepuff Namespace                                                                                                   
    apply.go:124: Creating network-policy-conformance-ravenclaw Namespace                                                                                                    
    apply.go:124: Creating harry-potter StatefulSet                                                                                                                          
    apply.go:124: Creating draco-malfoy StatefulSet                                                                                                                          
    apply.go:124: Creating cedric-diggory StatefulSet                                                                                                                        
    apply.go:124: Creating luna-lovegood StatefulSet                                                                                                                         
    suite.go:119: Test Setup: Ensuring Namespaces and Pods from base manifests are ready                                                                                     
    helper.go:32: Ready was not in conditions list                                                                                                                           
    helper.go:34: network-policy-conformance-gryffindor/harry-potter-0 Pod not ready yet                                                                                     
    helper.go:32: Ready condition set to False, expected True                                                                                                                
    helper.go:32: Ready was not in conditions list                                                                                                                           
    helper.go:34: network-policy-conformance-gryffindor/harry-potter-0 Pod not ready yet                                                                                     
    helper.go:32: Ready condition set to False, expected True                                                                                                                
    helper.go:32: Ready was not in conditions list                                                                                                                           
    helper.go:34: network-policy-conformance-gryffindor/harry-potter-1 Pod not ready yet                                                                                     
    helper.go:32: Ready condition set to False, expected True                                                                                                                
    helper.go:32: Ready was not in conditions list                                                                                                                           
    helper.go:34: network-policy-conformance-gryffindor/harry-potter-1 Pod not ready yet                                                                                     
    helper.go:32: Ready condition set to False, expected True                                                                                                                
    helper.go:32: Ready was not in conditions list                                                                                                                           
    helper.go:34: network-policy-conformance-hufflepuff/cedric-diggory-1 Pod not ready yet                                                                                   
    helper.go:39: Namespaces and Pods in network-policy-conformance-gryffindor, network-policy-conformance-slytherin, network-policy-conformance-hufflepuff, network-policy-conformance-ravenclaw namespaces ready
   apply.go:132: Deleting luna-lovegood StatefulSet
    apply.go:132: Deleting cedric-diggory StatefulSet
    apply.go:132: Deleting draco-malfoy StatefulSet
    apply.go:132: Deleting harry-potter StatefulSet
    apply.go:132: Deleting network-policy-conformance-ravenclaw Namespace
    apply.go:132: Deleting network-policy-conformance-hufflepuff Namespace
    apply.go:132: Deleting network-policy-conformance-slytherin Namespace
    apply.go:132: Deleting network-policy-conformance-gryffindor Namespace
--- PASS: TestConformance (5.19s)
PASS
ok      sigs.k8s.io/network-policy-api/conformance      5.217s
?       sigs.k8s.io/network-policy-api/conformance/tests        [no test files]
?       sigs.k8s.io/network-policy-api/conformance/utils/config [no test files]
?       sigs.k8s.io/network-policy-api/conformance/utils/flags  [no test files]
=== RUN   TestPrepareResources
=== RUN   TestPrepareResources/empty_namespace_labels
=== RUN   TestPrepareResources/simple_namespace_labels
=== RUN   TestPrepareResources/overwrite_namespace_labels
--- PASS: TestPrepareResources (0.00s)
    --- PASS: TestPrepareResources/empty_namespace_labels (0.00s)
    --- PASS: TestPrepareResources/simple_namespace_labels (0.00s)
    --- PASS: TestPrepareResources/overwrite_namespace_labels (0.00s)
PASS
ok      sigs.k8s.io/network-policy-api/conformance/utils/kubernetes     0.014s
?       sigs.k8s.io/network-policy-api/conformance/utils/suite  [no test files]

Closes #35

netlify · 2023-05-17T12:48:43Z

✅ Deploy Preview for kubernetes-sigs-network-policy-api ready!

Name	Link
🔨 Latest commit	`00bf1ff`
🔍 Latest deploy log	https://app.netlify.com/sites/kubernetes-sigs-network-policy-api/deploys/646baea4e462070009f08de2
😎 Deploy Preview	https://deploy-preview-98--kubernetes-sigs-network-policy-api.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site settings.

This commit tracks the go.mod and go.sum changes to import some of the dependencies we need to build and run conformance tests. Signed-off-by: Surya Seetharaman <suryaseetharaman.9@gmail.com>

conformance/base/manifests.yaml

conformance/utils/config/timeout.go

conformance/utils/suite/suite.go

astoycos · 2023-05-22T16:48:38Z

conformance/utils/suite/features.go

+
+// StandardExtendedFeatures are extra generic features that implementations may
+// choose to support as an opt-in.
+var StandardExtendedFeatures = sets.New(


We should probably document somewhere that these are the "optional" features

yes good point: #102 -> I have opened that issue to document everything on conformance but probably the docs PR will go last once I've finished adding all the tests - but certainly planning on having a separate space in our website to talk about conformance.

astoycos · 2023-05-22T16:56:25Z

Tested locally things look good! Just some small NITs

astoycos · 2023-05-22T17:49:14Z

conformance/conformance_test.go

+	t.Logf("Running conformance tests with cleanup: %t\n debug: %t\n enable all features: %t \n supported features: [%v]\n exempt features: [%v]",
+		*flags.CleanupBaseResources, *flags.ShowDebug, *flags.EnableAllSupportedFeatures, *flags.SupportedFeatures, *flags.ExemptFeatures)
+
+	cSuite := suite.New(suite.Options{


@tssurya Would it be difficult to have a "--dry-run" flag here?

Then in our upstream CI we could run the tests against a vanilla KIND cluster without actually testing an implementation

It could help ensure the future stability of the tests

I'm assuming it would just involve adding it to your "poke..." function

hmm great point! I think this should be doable.. let me open a card for this to enhance this further - #107
Also note that all implementations should be running these exact tests downstream, so if something breaks here we will know for sure :D because I will have trouble in OVN-K CI :D - let me take a stab at this issue once I have figured out the whole picture of how I will digest this downstream on the CI...

This commit adds the initial framework we need to run conformance tests. ALL CREDIT to sig-network-gatewaya-api community and folks who added conformance there. We have blatantly copied from them and adapted the suite to our needs. https://github.com/kubernetes-sigs/gateway-api/tree/main/conformance Signed-off-by: Surya Seetharaman <suryaseetharaman.9@gmail.com>

astoycos · 2023-05-22T18:45:29Z

/lgtm
/approve

k8s-ci-robot · 2023-05-22T18:45:55Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: astoycos, tssurya

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [astoycos]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

k8s-ci-robot added the cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. label May 17, 2023

k8s-ci-robot requested review from astoycos and Dyanngg May 17, 2023 12:48

k8s-ci-robot added the size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. label May 17, 2023

tssurya mentioned this pull request May 17, 2023

Add ANP conformance tests for .Spec.Ingress and .Spec.Egress fields #99

Merged

tssurya force-pushed the conformance-test-framework branch 4 times, most recently from 1c4ecdf to 0d18643 Compare May 18, 2023 10:02

k8s-ci-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label May 20, 2023

Add modules required to support conformace tests

2f547fe

This commit tracks the go.mod and go.sum changes to import some of the dependencies we need to build and run conformance tests. Signed-off-by: Surya Seetharaman <suryaseetharaman.9@gmail.com>

tssurya force-pushed the conformance-test-framework branch from 0d18643 to 9f48fd2 Compare May 20, 2023 10:20

k8s-ci-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label May 20, 2023

tssurya force-pushed the conformance-test-framework branch from 9f48fd2 to 1e988f9 Compare May 20, 2023 14:09

astoycos reviewed May 22, 2023

View reviewed changes

conformance/base/manifests.yaml Show resolved Hide resolved

astoycos reviewed May 22, 2023

View reviewed changes

conformance/utils/config/timeout.go Outdated Show resolved Hide resolved

astoycos reviewed May 22, 2023

View reviewed changes

conformance/utils/suite/suite.go Outdated Show resolved Hide resolved

astoycos reviewed May 22, 2023

View reviewed changes

tssurya mentioned this pull request May 22, 2023

Add an option for a --dry-run in our conformance test suite #107

Closed

tssurya force-pushed the conformance-test-framework branch from 1e988f9 to 00bf1ff Compare May 22, 2023 18:04

tssurya requested a review from astoycos May 22, 2023 18:12

k8s-ci-robot assigned astoycos May 22, 2023

k8s-ci-robot added lgtm "Looks good to me", indicates that a PR is ready to be merged. approved Indicates a PR has been approved by an approver from all required OWNERS files. labels May 22, 2023

astoycos approved these changes May 22, 2023

View reviewed changes

k8s-ci-robot merged commit 0a797fb into kubernetes-sigs:master May 22, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add Framework for Conformance Tests #98

Add Framework for Conformance Tests #98

tssurya commented May 17, 2023 •

edited

Loading

netlify bot commented May 17, 2023 •

edited

Loading

astoycos May 22, 2023

tssurya May 22, 2023 •

edited

Loading

astoycos commented May 22, 2023

astoycos May 22, 2023 •

edited

Loading

astoycos May 22, 2023

tssurya May 22, 2023 •

edited

Loading

astoycos commented May 22, 2023

k8s-ci-robot commented May 22, 2023

Add Framework for Conformance Tests #98

Add Framework for Conformance Tests #98

Conversation

tssurya commented May 17, 2023 • edited Loading

netlify bot commented May 17, 2023 • edited Loading

✅ Deploy Preview for kubernetes-sigs-network-policy-api ready!

astoycos May 22, 2023

Choose a reason for hiding this comment

tssurya May 22, 2023 • edited Loading

Choose a reason for hiding this comment

astoycos commented May 22, 2023

astoycos May 22, 2023 • edited Loading

Choose a reason for hiding this comment

astoycos May 22, 2023

Choose a reason for hiding this comment

tssurya May 22, 2023 • edited Loading

Choose a reason for hiding this comment

astoycos commented May 22, 2023

k8s-ci-robot commented May 22, 2023

tssurya commented May 17, 2023 •

edited

Loading

netlify bot commented May 17, 2023 •

edited

Loading

tssurya May 22, 2023 •

edited

Loading

astoycos May 22, 2023 •

edited

Loading

tssurya May 22, 2023 •

edited

Loading