Snat tests: [agent is already updated] #1513

cgchinmay · 2021-06-17T05:37:31Z

What type of PR is this?
feature

Which issue does this PR fix:
Added Ginkgo tests for validating SNAT related env vars on aws-node daemonset

What does this PR do / Why do we need it:
Test automation

Will this break upgrades or downgrades. Has updating a running cluster been tested?:
No

Test Output:

ginkgo -v -r -- --cluster-kubeconfig=/Users/cgadgil/.kube/config --cluster-name=private-subnet-test --aws-region=us-west-2 --aws-vpc-id=vpc-0a8f1151f7928f01d

Running Suite: Snat Suite
=========================
Random Seed: 1623893071
Will run 3 of 3 specs

STEP: creating test namespace
STEP: Getting existing nodes in the cluster
STEP: verifying more than 1 nodes are present for the test
STEP: Getting Public and Private subnets
STEP: Creating a keyPair with name: test-key-pair if it doesn't exist
KeyPair doesn't exist, will be created
STEP: Getting Cluster Security Group Id
STEP: Deploying a self managed nodegroup of size 1 in private subnet subnet-08eb3685be10501a8
STEP: Fetching existing Security Groups from the newly created node group instance
STEP: Adding ClusterSecurityGroup to the new nodegroup Instance
SNAT test 
  Pod in private subnet should have Internet access with External SNAT enabled
  /Users/cgadgil/Documents/amazon-vpc-cni-k8s/test/integration-new/snat/snat_test.go:22
STEP: Setting External SNAT to true
STEP: getting the aws-node daemon set in namesapce kube-system
STEP: setting the environment variables on the ds to map[AWS_VPC_K8S_CNI_EXTERNALSNAT:true]
STEP: updating the daemon set with new environment variable
STEP: Checking External Domain Connectivity
STEP: Deploying a test pod to check External domain access
2021/06/17 01:31:06 External Domain Connectivity test passed

STEP: deleting the test pod
STEP: Reverting aws-node env variables to default values
STEP: getting the aws-node daemon set in namesapce kube-system
STEP: setting the environment variables on the ds to map[AWS_VPC_K8S_CNI_EXTERNALSNAT:false AWS_VPC_K8S_CNI_RANDOMIZESNAT:prng]
STEP: updating the daemon set with new environment variable
STEP: getting the aws-node daemon set in namesapce kube-system
STEP: setting the environment variables on the ds to map[AWS_VPC_K8S_CNI_EXCLUDE_SNAT_CIDRS:{}]
STEP: updating the daemon set with new environment variable

• [SLOW TEST:155.031 seconds]
SNAT test
/Users/cgadgil/Documents/amazon-vpc-cni-k8s/test/integration-new/snat/snat_test.go:21
  Pod in private subnet should have Internet access with External SNAT enabled
  /Users/cgadgil/Documents/amazon-vpc-cni-k8s/test/integration-new/snat/snat_test.go:22
------------------------------
SNAT test 
  Verify SNAT IP table rule by changing AWS_VPC_K8S_CNI_RANDOMIZESNAT
  /Users/cgadgil/Documents/amazon-vpc-cni-k8s/test/integration-new/snat/snat_test.go:32
STEP: Check whether SNAT IP table has random-fully with AWS_VPC_K8S_CNI_RANDOMIZESNAT set to default value of prng
STEP: creating pod to check iptable SNAT rules on the host
2021/06/17 01:32:19 Randomized SNAT test passed for AWS_VPC_K8S_CNI_RANDOMIZESNAT: prng

STEP: deleting the host networking setup pod
STEP: Setting AWS_VPC_K8S_CNI_RANDOMIZESNAT to none
STEP: getting the aws-node daemon set in namesapce kube-system
STEP: setting the environment variables on the ds to map[AWS_VPC_K8S_CNI_RANDOMIZESNAT:none]
STEP: updating the daemon set with new environment variable
STEP: Check where SNAT IP table rule is updated and it doesn't contain random port allocation
STEP: creating pod to check iptable SNAT rules on the host
2021/06/17 01:33:27 Randomized SNAT test passed for AWS_VPC_K8S_CNI_RANDOMIZESNAT: none

STEP: deleting the host networking setup pod
STEP: Reverting aws-node env variables to default values
STEP: getting the aws-node daemon set in namesapce kube-system
STEP: setting the environment variables on the ds to map[AWS_VPC_K8S_CNI_EXTERNALSNAT:false AWS_VPC_K8S_CNI_RANDOMIZESNAT:prng]
STEP: updating the daemon set with new environment variable
STEP: getting the aws-node daemon set in namesapce kube-system
STEP: setting the environment variables on the ds to map[AWS_VPC_K8S_CNI_EXCLUDE_SNAT_CIDRS:{}]
STEP: updating the daemon set with new environment variable

• [SLOW TEST:146.748 seconds]
SNAT test
/Users/cgadgil/Documents/amazon-vpc-cni-k8s/test/integration-new/snat/snat_test.go:21
  Verify SNAT IP table rule by changing AWS_VPC_K8S_CNI_RANDOMIZESNAT
  /Users/cgadgil/Documents/amazon-vpc-cni-k8s/test/integration-new/snat/snat_test.go:32
------------------------------
SNAT test 
  Verify External Domain Connectivity by modifying AWS_VPC_K8S_CNI_EXCLUDE_SNAT_CIDRS
  /Users/cgadgil/Documents/amazon-vpc-cni-k8s/test/integration-new/snat/snat_test.go:45
STEP: Getting CIDR for primary node's private subnet
STEP: Updating AWS_VPC_K8S_CNI_EXCLUDE_SNAT_CIDRS with private subnet CIDR
STEP: getting the aws-node daemon set in namesapce kube-system
STEP: setting the environment variables on the ds to map[AWS_VPC_K8S_CNI_EXCLUDE_SNAT_CIDRS:192.168.96.0/19]
STEP: updating the daemon set with new environment variable
STEP: Check External domain connectivity from this private subnet CIDR block
STEP: Deploying a test pod to check External domain access
2021/06/17 01:36:01 External Domain Connectivity test passed

STEP: deleting the test pod
STEP: Reverting aws-node env variables to default values
STEP: getting the aws-node daemon set in namesapce kube-system
STEP: setting the environment variables on the ds to map[AWS_VPC_K8S_CNI_EXTERNALSNAT:false AWS_VPC_K8S_CNI_RANDOMIZESNAT:prng]
STEP: updating the daemon set with new environment variable
STEP: getting the aws-node daemon set in namesapce kube-system
STEP: setting the environment variables on the ds to map[AWS_VPC_K8S_CNI_EXCLUDE_SNAT_CIDRS:{}]
STEP: updating the daemon set with new environment variable

• [SLOW TEST:142.943 seconds]
SNAT test
/Users/cgadgil/Documents/amazon-vpc-cni-k8s/test/integration-new/snat/snat_test.go:21
  Verify External Domain Connectivity by modifying AWS_VPC_K8S_CNI_EXCLUDE_SNAT_CIDRS
  /Users/cgadgil/Documents/amazon-vpc-cni-k8s/test/integration-new/snat/snat_test.go:45
------------------------------
STEP: Deleting key-pair: test-key-pair
STEP: Deleting test namespace
STEP: Deleting Managed Nodegroup

Ran 3 of 3 Specs in 898.224 seconds
SUCCESS! -- 3 Passed | 0 Failed | 0 Pending | 0 Skipped
PASS

Ginkgo ran 1 suite in 15m3.989189616s
Test Suite Passed

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

jayanthvn · 2021-07-02T22:44:59Z

As discussed offline, let's add a test to validate the behavior prior to setting the env variable.

test/integration-new/snat/snat_test.go

test/framework/options.go

test/framework/resources/aws/services/ec2.go

test/integration-new/cni/env_vars_test.go

test/integration-new/snat/snat_suite_test.go

test/integration-new/snat/snat_test.go

test/agent/cmd/snat-utils/main.go

abhipth · 2021-07-09T20:20:38Z

We may have to split the PR into two,

Changes to the Agent.
Using the new Agent version in the Ginko Test with SNAT utility.

See

test/agent/cmd/snat-utils/main.go

jayanthvn

LGTM 👍

cgchinmay · 2021-08-30T20:57:24Z

We may have to split the PR into two,

Changes to the Agent.

Using the new Agent version in the Ginko Test with SNAT utility.

See

Completed

abhipth · 2021-09-21T20:54:02Z

I see two new binary files added, what are these used for?

test/agent/aws-node-config 
test/agent/cni

cgchinmay · 2021-09-21T20:58:22Z

I see two new binary files added, what are these used for?
test/agent/aws-node-config 
test/agent/cni 

removed them

abhipth

Looks like there is a merge conflict. Overall looks good, have put some comments. Let me know your thoughts on it.

go.sum

test/e2e/snat/snat_suite_test.go

test/integration-new/ipamd/warm_target_test.go

test/integration-new/ipamd/env_vars_test.go

test/e2e/snat/snat_suite_test.go

test/e2e/snat/snat_test.go

Fixed go.sum in root folder Changed DescribeInstanceWithFilter to DescribeInstances Moved GetPrimaryInstanceId from ec2 interface Added GinkgoWriter Updated Readme for Snat test Rearranged snat_test logic Updated Readme for test/e2e

cgchinmay · 2021-09-23T17:18:52Z

Looks like there is a merge conflict. Overall looks good, have put some comments. Let me know your thoughts on it.

resolved

# Conflicts: # go.mod # go.sum # test/framework/utils/const.go # test/go.mod # test/go.sum # test/integration-new/ipamd/env_vars_test.go

Addressed the requested changes

…ariables if available removed unnecessary logs Update failing test Updated ClusterRole permissions Rename mType to metricType Fetch Region only if not available Remove redundant logging helm chart changes to use the new AWS_CLUSTER_ID env variable Minor fixes to fetching region and cluster_id logic Simply logic to fetch cluster_id and region Updated cni-metrics-helper Readme with instructions for using IRSA Updated clusterRole template for cni-metrics-helper helm chart Manifests and Readme updates (aws#1732) * Manifests and Readme updates * update manifest.jsonnet Readme updates (aws#1735) Updates to troubleshooting doc (aws#1737) * Updates to troubleshooting doc * updates to troubleshooting doc imdsv2 changes (aws#1743) fix flaky canary test (aws#1742) add CODEOWNERS (aws#1747) Snat tests: [agent is already updated] (aws#1513) * resolved conflicts with go.sum * Updated test agent image * Removed redundant files * Addressed PR comments Fixed go.sum in root folder Changed DescribeInstanceWithFilter to DescribeInstances Moved GetPrimaryInstanceId from ec2 interface Added GinkgoWriter Updated Readme for Snat test Rearranged snat_test logic Updated Readme for test/e2e * Minor change to logging Updated Chart version for cni-metrics-helper

* resolved conflicts with go.sum * Updated test agent image * Removed redundant files * Addressed PR comments Fixed go.sum in root folder Changed DescribeInstanceWithFilter to DescribeInstances Moved GetPrimaryInstanceId from ec2 interface Added GinkgoWriter Updated Readme for Snat test Rearranged snat_test logic Updated Readme for test/e2e * Minor change to logging

This VlanId will appear in the prevResult during cmdDel request CleanUp Pod Network using vlanId from prevResult in CNI itself No need to call ipamd Log formatting changes Added hostNetworking Setup test for pods using security groups Updated cleanUpPodENI method Skip processing Delete request if prevResult is nil Add Logging vlanId to ipamd Add support to test with containerd nodegroup in pod-eni test Add check for empty Netns() in cni Manifests and Readme updates (aws#1732) * Manifests and Readme updates * update manifest.jsonnet Readme updates (aws#1735) Updates to troubleshooting doc (aws#1737) * Updates to troubleshooting doc * updates to troubleshooting doc imdsv2 changes (aws#1743) fix flaky canary test (aws#1742) add CODEOWNERS (aws#1747) Snat tests: [agent is already updated] (aws#1513) * resolved conflicts with go.sum * Updated test agent image * Removed redundant files * Addressed PR comments Fixed go.sum in root folder Changed DescribeInstanceWithFilter to DescribeInstances Moved GetPrimaryInstanceId from ec2 interface Added GinkgoWriter Updated Readme for Snat test Rearranged snat_test logic Updated Readme for test/e2e * Minor change to logging Fix compilation errors (aws#1751) add support for running canary script in different regions (aws#1752) Regenerate pod eni values for new instance types (aws#1754) * Regenerate pod eni values for new instance types Co-authored-by: Senthil Kumaran <senthilx@amazon.com> Minor change to container runtime argument Check for Empty NetNs() first Fallback to older method if prevResult is nil Closed issue message (aws#1761) * closed issue message * update message fix typo in upload script (aws#1763) Update calico file path Use an unique s3 bucket name (aws#1760) Update region Workflow to build arm and x86 images (aws#1764) DataStore.GetStats() refactoring to simplify adding new fields (aws#1704) * DataStore.GetStats() refactoring to simplify adding new fields * cleanup * cleanup * cleanup * goimports * rename test to TestGetStatsV4 * address comments * fix typo * update * update "IP pool is too low" logging * GetStats() -> GetIpStats() * GetStats() -> GetIpStats() in tests and comments * update test * cleanup test * add logPoolStats comment Fix KOPS_STATE_STORE (aws#1770) Automation script for running IT (aws#1759) Update issue template Update issue template with email address Update issue template Update go.mod for integration folder (aws#1741) * Update go.mod for integration folder - Update go.mod for integration folder * Change integration test to use new K8s test framework * Modify server pod image * Switch to Nginx port 80 for server pod * Switch server port in client test * Remove custom command directive for Nginx pod * Added ping command for host checks README: mention arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy (aws#1768) Co-authored-by: Shreya027 <shrenaik@amazon.com> Add dl1.24xlarge to ENILimits override list (aws#1777) Chart and Manifest updates (aws#1771) * Chart and Manifest updates * Update probe timeout values Change workflow to use git install (aws#1785) - Change workflow to use git install as the go get command was altering go.mod file without updating go.sum file

Add VlanId in the cmdAdd Result struct This VlanId will appear in the prevResult during cmdDel request CleanUp Pod Network using vlanId from prevResult in CNI itself No need to call ipamd Log formatting changes Added hostNetworking Setup test for pods using security groups Updated cleanUpPodENI method Skip processing Delete request if prevResult is nil Add Logging vlanId to ipamd Add support to test with containerd nodegroup in pod-eni test Add check for empty Netns() in cni Manifests and Readme updates (aws#1732) * Manifests and Readme updates * update manifest.jsonnet Readme updates (aws#1735) Updates to troubleshooting doc (aws#1737) * Updates to troubleshooting doc * updates to troubleshooting doc imdsv2 changes (aws#1743) fix flaky canary test (aws#1742) add CODEOWNERS (aws#1747) Snat tests: [agent is already updated] (aws#1513) * resolved conflicts with go.sum * Updated test agent image * Removed redundant files * Addressed PR comments Fixed go.sum in root folder Changed DescribeInstanceWithFilter to DescribeInstances Moved GetPrimaryInstanceId from ec2 interface Added GinkgoWriter Updated Readme for Snat test Rearranged snat_test logic Updated Readme for test/e2e * Minor change to logging Fix compilation errors (aws#1751) add support for running canary script in different regions (aws#1752) Regenerate pod eni values for new instance types (aws#1754) * Regenerate pod eni values for new instance types Co-authored-by: Senthil Kumaran <senthilx@amazon.com> Minor change to container runtime argument Check for Empty NetNs() first Fallback to older method if prevResult is nil Closed issue message (aws#1761) * closed issue message * update message fix typo in upload script (aws#1763) Update calico file path Use an unique s3 bucket name (aws#1760) Update region Workflow to build arm and x86 images (aws#1764) DataStore.GetStats() refactoring to simplify adding new fields (aws#1704) * DataStore.GetStats() refactoring to simplify adding new fields * cleanup * cleanup * cleanup * goimports * rename test to TestGetStatsV4 * address comments * fix typo * update * update "IP pool is too low" logging * GetStats() -> GetIpStats() * GetStats() -> GetIpStats() in tests and comments * update test * cleanup test * add logPoolStats comment Fix KOPS_STATE_STORE (aws#1770) Automation script for running IT (aws#1759) Update issue template Update issue template with email address Update issue template Update go.mod for integration folder (aws#1741) * Update go.mod for integration folder - Update go.mod for integration folder * Change integration test to use new K8s test framework * Modify server pod image * Switch to Nginx port 80 for server pod * Switch server port in client test * Remove custom command directive for Nginx pod * Added ping command for host checks README: mention arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy (aws#1768) Co-authored-by: Shreya027 <shrenaik@amazon.com> Add dl1.24xlarge to ENILimits override list (aws#1777) Chart and Manifest updates (aws#1771) * Chart and Manifest updates * Update probe timeout values Change workflow to use git install (aws#1785) - Change workflow to use git install as the go get command was altering go.mod file without updating go.sum file

* resolved conflicts with go.sum * Updated test agent image * Removed redundant files * Addressed PR comments Fixed go.sum in root folder Changed DescribeInstanceWithFilter to DescribeInstances Moved GetPrimaryInstanceId from ec2 interface Added GinkgoWriter Updated Readme for Snat test Rearranged snat_test logic Updated Readme for test/e2e * Minor change to logging

cgchinmay requested review from abhipth and jayanthvn June 17, 2021 05:37

cgchinmay force-pushed the snat_tests branch 3 times, most recently from 33ccdf0 to 00b0b8d Compare June 17, 2021 07:24

jayanthvn reviewed Jul 8, 2021

View reviewed changes

test/integration-new/snat/snat_test.go Outdated Show resolved Hide resolved

abhipth suggested changes Jul 8, 2021

View reviewed changes

achevuru reviewed Jul 8, 2021

View reviewed changes

test/agent/cmd/snat-utils/main.go Outdated Show resolved Hide resolved

achevuru reviewed Jul 8, 2021

View reviewed changes

test/agent/cmd/snat-utils/main.go Outdated Show resolved Hide resolved

cgchinmay force-pushed the snat_tests branch from 471c968 to bdc6c31 Compare July 9, 2021 20:08

cgchinmay force-pushed the snat_tests branch 5 times, most recently from 44a6975 to d4bfc4c Compare July 16, 2021 18:34

jayanthvn reviewed Jul 16, 2021

View reviewed changes

test/agent/cmd/snat-utils/main.go Outdated Show resolved Hide resolved

jayanthvn approved these changes Jul 16, 2021

View reviewed changes

cgchinmay force-pushed the snat_tests branch 2 times, most recently from 14aa58c to 3922fa1 Compare July 19, 2021 21:52

cgchinmay mentioned this pull request Jul 19, 2021

Updated test agent for snat test #1533

Merged

resolved conflicts with go.sum

30a2bc5

cgchinmay force-pushed the snat_tests branch from 3922fa1 to 30a2bc5 Compare August 30, 2021 19:12

Chinmay Gadgil added 2 commits August 30, 2021 13:10

Merge remote-tracking branch 'upstream/master' into snat_tests

a03867f

Updated test agent image

97b8f8e

cgchinmay requested a review from abhipth August 30, 2021 21:55

Removed redundant files

4d00060

abhipth previously requested changes Sep 21, 2021

View reviewed changes

Chinmay Gadgil added 2 commits September 22, 2021 00:47

Merge branch 'master' into snat_tests

236033b

Addressed PR comments

23c4d76

Fixed go.sum in root folder Changed DescribeInstanceWithFilter to DescribeInstances Moved GetPrimaryInstanceId from ec2 interface Added GinkgoWriter Updated Readme for Snat test Rearranged snat_test logic Updated Readme for test/e2e

cgchinmay force-pushed the snat_tests branch from 5b42dd7 to 23c4d76 Compare September 22, 2021 19:36

Chinmay Gadgil added 2 commits November 12, 2021 10:32

Merge remote-tracking branch 'upstream/master' into snat_tests

41ab693

# Conflicts: # go.mod # go.sum # test/framework/utils/const.go # test/go.mod # test/go.sum # test/integration-new/ipamd/env_vars_test.go

Minor change to logging

1ee1d7d

cgchinmay requested a review from a team as a code owner November 12, 2021 19:24

cgchinmay changed the title ~~Snat tests: Need to update agent version and image after these changes are approved~~ Snat tests: [agent is already updated] Nov 12, 2021

cgchinmay merged commit ffa0146 into aws:master Nov 12, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Snat tests: [agent is already updated] #1513

Snat tests: [agent is already updated] #1513

cgchinmay commented Jun 17, 2021

jayanthvn commented Jul 2, 2021

abhipth commented Jul 9, 2021

jayanthvn left a comment

cgchinmay commented Aug 30, 2021

abhipth commented Sep 21, 2021

cgchinmay commented Sep 21, 2021

abhipth left a comment

cgchinmay commented Sep 23, 2021

Snat tests: [agent is already updated] #1513

Snat tests: [agent is already updated] #1513

Conversation

cgchinmay commented Jun 17, 2021

jayanthvn commented Jul 2, 2021

abhipth commented Jul 9, 2021

jayanthvn left a comment

Choose a reason for hiding this comment

cgchinmay commented Aug 30, 2021

abhipth commented Sep 21, 2021

cgchinmay commented Sep 21, 2021

abhipth left a comment

Choose a reason for hiding this comment

cgchinmay commented Sep 23, 2021