updating policy event logs with metadata #296

emilyhuaa · 2024-08-01T17:58:50Z

Issue #, if available:
When users enable policy event logs, it only contains IP and Port information for Source and Destination, and users have to look up name and namespace for the pod of the given IP address. This creates inefficiency for debugging network policy issues.

Description of changes:

This PR improves the visibility by supporting the Network Policy Agent to fetch pod Kubernetes metadata through a gRPC call with a new deployment on the customer's cluster. This deployment maintains a local cache of pod and service IP address and name/namespace info from the Kubernetes API server, syncing every 10 seconds.

Manual Testing
I did some testing of each of the functions and now the policy event logs look like this:
{"level":"info","ts":"2024-07-23T23:03:50.718Z","logger":"ebpf-client","caller":"utils/utils.go:106","msg":"Flow Info: ","Src IP":"192.168.42.144","Src Name":"client-5467bc68f4-zcl6g","Src Namespace":"client-5467bc68f4-zcl6g","Src Port":34765,"Dest IP":"192.168.77.216","Dest Name":"coredns-787cb67946-blpcn","Dest Namespace":"kube-system","Dest Port":53,"Proto":"UDP","Verdict":"DENY"}

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

Dockerfile

Makefile

pkg/ebpf/events/events.go

pkg/rpc/rpc_handler.go

pkg/utils/utils.go

achevuru · 2024-08-13T05:20:03Z

pkg/ebpf/events/events.go

+func logPolicyEventsWithK8sMetadata(log logr.Logger, message *string, nodeName, srcIP string, srcPort int, destIP string, destPort int, protocol, verdict string) {
+	srcName, srcNS, err := utils.GetPodMetadata(srcIP)
+	if err != nil {
+		log.Info("Failed to get source name and namespace metadata", "source IP:", srcIP)


Let's do away with this. We're already making an assumption that an unknown IP is an external IP to the cluster...

achevuru · 2024-08-13T05:21:12Z

pkg/ebpf/events/events.go

+	}
+	destName, destNS, err := utils.GetPodMetadata(destIP)
+	if err != nil {
+		log.Info("Failed to get destination name and namespace metadata for", "dest IP:", destIP)


Same as above. Let's directly call utils.LogFlowInfo instead from capturePolicyEvents

achevuru · 2024-08-13T05:30:35Z

pkg/rpc/rpc_handler.go

-	rpc.RegisterNPBackendServer(grpcServer, &server{policyReconciler: policyReconciler, log: rpcLog})
+
+	// Connect to metadata cache service
+	serviceIP, err := utils.GetServiceIP(clientset, "metadata-cache-service", "default", rpcLog)


We're making an assumption that aws-k8s-metadata service is available and we exit if we fail to find such a service on the cluster. We can't exit out of the process. We should make this optional (i.e.,) we print an enhanced log message if we find the service and if not we print the 5 tuple info of the flow (SrcIP/SrcPort/DstIP/DstPort/Protocol) like we do today.

metadata-cache-service --> let's make this a const and maybe add an aws prefix to it.

I see that we're creating a K8S client obj just to find the service IP of metadata cache service and there is no need to do it. We can just resolve the service name (DNS resolution) and that should provide us with service IP.

achevuru · 2024-08-13T05:36:16Z

pkg/ebpf/events/events.go

-					"Proto", protocol, "Verdict", verdict)
-
-				message = "Node: " + nodeName + ";" + "SIP: " + utils.ConvByteArrayToIP(rb.SourceIP) + ";" + "SPORT: " + strconv.Itoa(int(rb.SourcePort)) + ";" + "DIP: " + utils.ConvByteArrayToIP(rb.DestIP) + ";" + "DPORT: " + strconv.Itoa(int(rb.DestPort)) + ";" + "PROTOCOL: " + protocol + ";" + "PolicyVerdict: " + verdict
+				logPolicyEventsWithK8sMetadata(log, &message, nodeName, srcIP, srcPort, destIP, destPort, protocol, verdict)


We should call this only if we find the aws-k8s-metadata service running on the cluster. If not, we print the log in the current format and move on. If not, we will see empty values for metadata fields in the log messages and that will look odd..

achevuru · 2024-08-13T05:54:03Z

main.go

@@ -83,6 +85,18 @@ func main() {
 		os.Exit(1)
 	}

+	k8sconfig, err := rest.InClusterConfig()


I don't see a need for this. I see that we rely on this for deriving the service IP of metadata cache service and instead we can just resolve the metadata service name to get the service IP..

achevuru · 2024-08-13T06:00:16Z

pkg/utils/utils.go

+
+func LogFlowInfo(log logr.Logger, message *string, nodeName, srcIP, srcN, srcNS string, srcPort int, destIP, destN, destNS string, destPort int, protocol, verdict string) {
+	switch {
+	case srcN == "" && srcNS == "": // if no metadata for source IP


We might've to rearrange this flow. Let's say destN and destNS are empty as well, we will end up printing "" strings for DN and DNS. Same applies to rest of the combinations below...

Also, why are we using SHOSTIP for Source IP field here?

If it is a host networking pod such as aws-node pods or kube-proxy pods, we agreed to just say state that the IP is a host IP with no name or namespace.

go.mod

pkg/rpc/rpc_handler.go

jayanthvn · 2024-08-19T23:20:34Z

pkg/utils/utils.go

+		log.Info("Flow Info: ", "Src IP", srcIP, "Src Name", srcName, "Src Namespace", srcName, "Src Port", srcPort,
+			"Dest Ext Srv IP", destIP, "Dest Port", destPort, "Proto", protocol, "Verdict", verdict)
+		*message = "Node: " + nodeName + ";" + "SIP: " + srcIP + ";" + "SN" + srcName + ";" + "SNS" + srcName + ";" + "SPORT: " + strconv.Itoa(srcPort) + ";" +
+			"DEXTSRVIP: " + destIP + ";" + "DPORT: " + strconv.Itoa(destPort) + ";" + "PROTOCOL: " + protocol + ";" + "PolicyVerdict: " + verdict


nit - "DEXTSRVIP" -> "DESTSRVIP"

emilyhuaa requested a review from a team as a code owner August 1, 2024 17:58

emilyhuaa changed the base branch from main to event-log-enhancement August 2, 2024 17:00