Unable to egress packets from pod #1750
Comments
|
@johngmyers Can you share your Neighbor table is fine, because we assign a |
apiVersion: apps/v1
kind: DaemonSet
metadata:
annotations:
deprecated.daemonset.template.generation: "1"
creationTimestamp: "2021-11-15T07:17:55Z"
generation: 1
labels:
addon.kops.k8s.io/name: networking.amazon-vpc-routed-eni
app.kubernetes.io/instance: aws-vpc-cni
app.kubernetes.io/managed-by: kops
app.kubernetes.io/name: aws-node
app.kubernetes.io/version: v1.9.3
k8s-app: aws-node
role.kubernetes.io/networking: "1"
name: aws-node
namespace: kube-system
resourceVersion: "679"
uid: 90c03a1e-818e-4c90-ac9d-dbd215f1c17d
spec:
revisionHistoryLimit: 10
selector:
matchLabels:
k8s-app: aws-node
template:
metadata:
creationTimestamp: null
labels:
app.kubernetes.io/instance: aws-vpc-cni
app.kubernetes.io/name: aws-node
k8s-app: aws-node
spec:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: kubernetes.io/os
operator: In
values:
- linux
- key: kubernetes.io/arch
operator: In
values:
- amd64
- arm64
- key: eks.amazonaws.com/compute-type
operator: NotIn
values:
- fargate
containers:
- env:
- name: AWS_VPC_K8S_CNI_CONFIGURE_RPFILTER
value: "false"
- name: ENABLE_IPv4
value: "false"
- name: ENABLE_IPv6
value: "true"
- name: ENABLE_PREFIX_DELEGATION
value: "true"
- name: WARM_PREFIX_TARGET
value: "1"
- name: MY_NODE_NAME
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: spec.nodeName
- name: CLUSTER_NAME
value: k8s.jgmyers.proofpoint-pg.net
image: johngmyers/amazon-k8s-cni:v1.10.0-219-gf2dac813
imagePullPolicy: Always
livenessProbe:
exec:
command:
- /app/grpc-health-probe
- -addr=:50051
- -connect-timeout=2s
- -rpc-timeout=2s
failureThreshold: 3
initialDelaySeconds: 60
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 5
name: aws-node
ports:
- containerPort: 61678
hostPort: 61678
name: metrics
protocol: TCP
readinessProbe:
exec:
command:
- /app/grpc-health-probe
- -addr=:50051
- -connect-timeout=2s
- -rpc-timeout=2s
failureThreshold: 3
initialDelaySeconds: 1
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 5
resources:
requests:
cpu: 10m
securityContext:
capabilities:
add:
- NET_ADMIN
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /host/opt/cni/bin
name: cni-bin-dir
- mountPath: /host/etc/cni/net.d
name: cni-net-dir
- mountPath: /host/var/log/aws-routed-eni
name: log-dir
- mountPath: /var/run/aws-node
name: run-dir
- mountPath: /var/run/dockershim.sock
name: dockershim
- mountPath: /run/xtables.lock
name: xtables-lock
dnsPolicy: ClusterFirst
hostNetwork: true
initContainers:
- env:
- name: DISABLE_TCP_EARLY_DEMUX
value: "false"
- name: ENABLE_IPv6
value: "true"
image: johngmyers/amazon-k8s-cni-init:v1.10.0-219-gf2dac813
imagePullPolicy: Always
name: aws-vpc-cni-init
resources: {}
securityContext:
privileged: true
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /host/opt/cni/bin
name: cni-bin-dir
priorityClassName: system-node-critical
restartPolicy: Always
schedulerName: default-scheduler
securityContext: {}
serviceAccount: aws-node
serviceAccountName: aws-node
terminationGracePeriodSeconds: 10
tolerations:
- operator: Exists
volumes:
- hostPath:
path: /opt/cni/bin
type: ""
name: cni-bin-dir
- hostPath:
path: /etc/cni/net.d
type: ""
name: cni-net-dir
- hostPath:
path: /run/containerd/containerd.sock
type: ""
name: dockershim
- hostPath:
path: /run/xtables.lock
type: ""
name: xtables-lock
- hostPath:
path: /var/log/aws-routed-eni
type: DirectoryOrCreate
name: log-dir
- hostPath:
path: /var/run/aws-node
type: DirectoryOrCreate
name: run-dir
updateStrategy:
type: OnDelete
status:
currentNumberScheduled: 2
desiredNumberScheduled: 2
numberAvailable: 2
numberMisscheduled: 0
numberReady: 2
observedGeneration: 1
updatedNumberScheduled: 2 |
|
|
@johngmyers Was asking for |
|
I gave you |
|
Ah, from inside the non-host-network pod. On a new cluster, running v0.10.1: That pod's node has: |
|
@johngmyers Thanks for sharing the info. Will go through them and get back. We validated IPv6 on AL2 and Bottlerocket AMIs. I will check the behavior with Ubuntu AMI and will get back. On a quick glance, I notice that there is no route for Pod IP ( |
|
Going by the |
|
Not sure what happened there. Here's the full |
|
|
|
@johngmyers Thanks for sharing all the requested info. Couldn't get to it last week, will try to test with Ubuntu AMI this week and will provide an update. |
|
This issue is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 14 days |
|
/not stale |
|
This issue is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 14 days |
|
/not stale |
|
This issue is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 14 days |
|
Issue closed due to inactivity. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
What happened:
Brought up IPv6-only cluster. Pods are unable to egress network traffic, not to nodes, other pods, services, or the Internet.
Attach logs
What you expected to happen:
IPv6 Traffic to egress pod, to nodes, other pods, services, and the Internet.
How to reproduce it (as minimally and precisely as possible):
Create IPv6 cluster using johngmyers/kops@cb6a583 and AMI
099720109477/ubuntu/images/hvm-ssd/ubuntu-impish-21.10-amd64-server-2021101Add the following to each instancegroup spec, to work around systemd/systemd#200803 (handling of AWS's broken DHCP6 responses):
kops update cluster --yes --adminCreate a Deployment running
digitalocean/doks-debug:latestand try pinging various IPv6 addresses.Anything else we need to know?:
From the pod, the routing table is:
Neighbor discovery does not appear functional:
The node has a much more functional routing table:
Environment:
kubectl version): 1.22.2cat /etc/os-release):099720109477/ubuntu/images/hvm-ssd/ubuntu-impish-21.10-amd64-server-2021101uname -a):5.13.0-1005-aws #6-Ubuntu SMPThe text was updated successfully, but these errors were encountered: