Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add EnableLogging and LogLabel supports for Node NetworkPolicy #6626

Merged
merged 1 commit into from
Oct 10, 2024
Merged

Add EnableLogging and LogLabel supports for Node NetworkPolicy #6626

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
59 changes: 58 additions & 1 deletion docs/antrea-node-network-policy.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@
- [Introduction](#introduction)
- [Prerequisites](#prerequisites)
- [Usage](#usage)
- [Logs](#logs)
- [Limitations](#limitations)
<!-- /toc -->

Expand Down Expand Up @@ -66,11 +67,15 @@ spec:
ports:
- protocol: TCP
port: 80
enableLogging: true
logLabel: allow-http
- name: drop-other
action: Drop
ports:
- protocol: TCP
port: 80
enableLogging: true
logLabel: default-drop-others
```

An example Node NetworkPolicy that blocks egress traffic from Nodes with label
Expand Down Expand Up @@ -105,6 +110,58 @@ spec:
port: 22
```

## Logs

The `enableLogging` and `logLabel` options provide limited support for Node NetworkPolicies. Since Node NetworkPolicies
are implemented using iptables, enabling `enableLogging` causes the Linux kernel to log information about all matching
packets via the kernel log. However, Antrea cannot process these logs directly. Instead, these logs can be accessed
through syslog, allowing you to filter and direct them to specific files using syslog syntax.

By default, `enableLogging` is unsupported in KinD clusters. To enable it, set the host’s
`/proc/sys/net/netfilter/nf_log_all_netns` to 1. Antrea uses the iptables `LOG` target to log packet information,
but by default, `/proc/sys/net/netfilter/nf_log_all_netns` is 0, preventing containers from logging to the kernel to
avoid clutter. If logging is required, you can enable it by setting the value to 1, but please be cautious to do so
unless you are clear about the impact.

For example, consider the Node NetworkPolicy `restrict-http-to-node` above. It could generate the following logs:

```text
Sep 2 10:31:07 k8s-node-control-plane kernel: [6657320.789675] Antrea:I:Allow:allow-http:IN=ens224 OUT= MAC=00:50:56:a7:fb:18:00:50:56:a7:23:47:08:00 SRC=10.10.0.10 DST=192.168.240.200 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=52813 DF PROTO=TCP SPT=57658 DPT=80 WINDOW=64240 RES=0x00 SYN URGP=0
Sep 2 10:31:11 k8s-node-control-plane kernel: [6657324.899219] Antrea:I:Drop:default-drop:IN=ens224 OUT= MAC=00:50:56:a7:fb:18:00:50:56:a7:23:47:08:00 SRC=192.168.240.201 DST=192.168.240.200 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=27486 DF PROTO=TCP SPT=33152 DPT=80 WINDOW=64240 RES=0x00 SYN URGP=0
```

In these logs, prefixes like `Antrea:I:Allow:allow-http:` and `Antrea:I:Drop:default-drop:` are added by iptables using
the `--log-prefix` parameter. The iptables log prefix is limited to 29 characters, as described in the
[iptables-extensions manual](https://ipset.netfilter.org/iptables-extensions.man.html).

The log prefix format includes essential information of a Node NetworkPolicy rule, and consists of four parts,
formatted as follows:

```text
|---1--| |2| |---3--| |----------4--------|
|Antrea|:|I|:|Reject|:|user-provided label|:|
|6 |1|1|1|4-6 |1|1-12 |1|
```

- Part 1: Fixed, "Antrea"
- Part 2: Direction, "I" (In) or "O" (Out)
- Part 3: Action, "Allow", "Drop", or "Reject"
- Part 4: User-provided `logLabel`, up to 12 characters

Due to iptables' 29-character prefix limitation, the user-provided `logLabel` is restricted to a maximum of 12 characters.
To manage these logs effectively, you can configure rsyslog on each Node as follows:

```text
# Example rsyslog configuration to filter Antrea logs
:msg, contains, "Antrea:I:Allow:allow-http" /var/log/antrea-node-netpol-allow.log
:msg, contains, "Antrea:I:Drop:default-drop" /var/log/antrea-node-netpol-drop.log
& stop
```

This configuration directs logs with the prefix `Antrea:I:Allow:allow-http` to `/var/log/antrea-node-netpol-allow.log`
and logs with the prefix `Antrea:I:Drop:default-drop` to `/var/log/antrea-node-netpol-drop.log`. The `& stop` command
ensures that these logs are not processed further.

## Limitations

- This feature is currently only supported for Linux Nodes.
Expand All @@ -118,4 +175,4 @@ spec:
- FQDN is not supported for ACNPs applied to Nodes.
- Layer 7 NetworkPolicy is not supported yet.
- For UDP or SCTP, when the `Reject` action is specified in an egress rule, it behaves identical to the `Drop` action.
- Traffic logging is not supported yet for ACNPs applied to Nodes.
- Limited support for traffic logging for ACNPs applied to Nodes.
107 changes: 84 additions & 23 deletions pkg/agent/controller/networkpolicy/node_reconciler_linux.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -43,6 +43,10 @@ const (
ipv6Any = "::/0"
)

// The logging of Node NetworkPolicy is implemented by iptables target LOG, which turns on kernel logging of matching
// packets. The label is useful for distinguishing Node NetworkPolicy logs from other kernel logs.
const logLabelPrefix = "Antrea"

var ipsetTypeHashIP = ipset.HashIP

/*
Expand Down Expand Up @@ -124,7 +128,7 @@ directly.
type coreIPTRule struct {
ruleID string
priority *types.Priority
ruleStr string
ruleStrs []string
}

type chainKey struct {
Expand Down Expand Up @@ -256,7 +260,7 @@ func (r *nodeReconciler) batchAdd(rules []*CompletedRule) error {
}

// Collect all core iptables rules.
coreIPTRule := &coreIPTRule{ruleID, iptRule.Priority, iptRule.CoreIPTRule}
coreIPTRule := &coreIPTRule{ruleID, iptRule.Priority, iptRule.CoreIPTRules}
if rule.Direction == v1beta2.DirectionIn {
ingressCoreIPTRules[ipProtocol] = append(ingressCoreIPTRules[ipProtocol], coreIPTRule)
} else {
Expand Down Expand Up @@ -322,6 +326,11 @@ func (r *nodeReconciler) GetRuleByFlowID(ruleFlowID uint32) (*types.PolicyRule,

func (r *nodeReconciler) computeIPTRules(rule *CompletedRule) (map[iptables.Protocol]*types.NodePolicyRule, *nodePolicyLastRealized) {
ruleID := rule.ID
enableLogging := rule.EnableLogging
var logLabel string
if enableLogging {
logLabel = generateLogLabel(rule)
}
lastRealized := newNodePolicyLastRealized()
priority := &types.Priority{
TierPriority: *rule.TierPriority,
Expand Down Expand Up @@ -362,7 +371,12 @@ func (r *nodeReconciler) computeIPTRules(rule *CompletedRule) (map[iptables.Prot

var serviceIPTRules []string
if serviceIPTChain != "" {
serviceIPTRules = buildServiceIPTRules(ipProtocol, rule.Services, serviceIPTChain, serviceIPTRuleTarget)
serviceIPTRules = buildServiceIPTRules(ipProtocol,
rule.Services,
serviceIPTChain,
serviceIPTRuleTarget,
enableLogging,
logLabel)
}

ipnets := getIPNetsFromRule(rule, isIPv6)
Expand All @@ -383,14 +397,19 @@ func (r *nodeReconciler) computeIPTRules(rule *CompletedRule) (map[iptables.Prot
lastRealized.ipnets[ipProtocol] = ipnet
}

coreIPTRule := buildCoreIPTRule(ipProtocol,
coreIPTRules := buildCoreIPTRules(ipProtocol,
coreIPTChain,
ipset,
ipnet,
coreIPTRuleTarget,
coreIPTRuleComment,
service,
rule.Direction == v1beta2.DirectionIn)
rule.Direction == v1beta2.DirectionIn,
// If the target of a core iptables rule is not a service chain, the iptables rule for logging should be
// generated along with the core iptables rule. Otherwise, the iptables rules for logging should be generated
// along with the service iptables rules.
enableLogging && serviceIPTChain == "",
logLabel)

nodePolicyRules[ipProtocol] = &types.NodePolicyRule{
IPSet: ipset,
Expand All @@ -399,7 +418,7 @@ func (r *nodeReconciler) computeIPTRules(rule *CompletedRule) (map[iptables.Prot
ServiceIPTChain: serviceIPTChain,
ServiceIPTRules: serviceIPTRules,
CoreIPTChain: coreIPTChain,
CoreIPTRule: coreIPTRule,
CoreIPTRules: coreIPTRules,
IsIPv6: isIPv6,
}
}
Expand All @@ -422,7 +441,7 @@ func (r *nodeReconciler) add(rule *CompletedRule) error {
return err
}
}
if err := r.addOrUpdateCoreIPTRules(iptRule.CoreIPTChain, iptRule.IsIPv6, false, &coreIPTRule{ruleID, iptRule.Priority, iptRule.CoreIPTRule}); err != nil {
if err := r.addOrUpdateCoreIPTRules(iptRule.CoreIPTChain, iptRule.IsIPv6, false, &coreIPTRule{ruleID, iptRule.Priority, iptRule.CoreIPTRules}); err != nil {
return err
}
}
Expand Down Expand Up @@ -456,15 +475,15 @@ func (r *nodeReconciler) update(lastRealized *nodePolicyLastRealized, newRule *C
return err
}
if shouldUpdateCoreIPTRules {
if err := r.addOrUpdateCoreIPTRules(iptRule.CoreIPTChain, iptRule.IsIPv6, true, &coreIPTRule{ruleID, iptRule.Priority, iptRule.CoreIPTRule}); err != nil {
if err := r.addOrUpdateCoreIPTRules(iptRule.CoreIPTChain, iptRule.IsIPv6, true, &coreIPTRule{ruleID, iptRule.Priority, iptRule.CoreIPTRules}); err != nil {
return err
}
}
} else if prevIPSet != "" {
// If the previous rule used an ipset, sync the new core iptables rule first to remove its reference, then
// delete the unused ipset.
if shouldUpdateCoreIPTRules {
if err := r.addOrUpdateCoreIPTRules(iptRule.CoreIPTChain, iptRule.IsIPv6, true, &coreIPTRule{ruleID, iptRule.Priority, iptRule.CoreIPTRule}); err != nil {
if err := r.addOrUpdateCoreIPTRules(iptRule.CoreIPTChain, iptRule.IsIPv6, true, &coreIPTRule{ruleID, iptRule.Priority, iptRule.CoreIPTRules}); err != nil {
return err
}
}
Expand All @@ -473,7 +492,7 @@ func (r *nodeReconciler) update(lastRealized *nodePolicyLastRealized, newRule *C
}
} else {
if shouldUpdateCoreIPTRules {
if err := r.addOrUpdateCoreIPTRules(iptRule.CoreIPTChain, iptRule.IsIPv6, true, &coreIPTRule{ruleID, iptRule.Priority, iptRule.CoreIPTRule}); err != nil {
if err := r.addOrUpdateCoreIPTRules(iptRule.CoreIPTChain, iptRule.IsIPv6, true, &coreIPTRule{ruleID, iptRule.Priority, iptRule.CoreIPTRules}); err != nil {
return err
}
}
Expand Down Expand Up @@ -517,9 +536,7 @@ func (r *nodeReconciler) addOrUpdateCoreIPTRules(chain string, isIPv6 bool, isUp
// Get all iptables rules and synchronize them.
var ruleStrs []string
for _, rule := range rules {
if rule.ruleStr != "" {
ruleStrs = append(ruleStrs, rule.ruleStr)
}
ruleStrs = append(ruleStrs, rule.ruleStrs...)
}
if err := r.routeClient.AddOrUpdateNodeNetworkPolicyIPTables([]string{chain}, [][]string{ruleStrs}, isIPv6); err != nil {
return err
Expand Down Expand Up @@ -554,7 +571,7 @@ func (r *nodeReconciler) deleteCoreIPTRule(ruleID string, iptChain string, isIPv
// Get all the iptables rules and synchronize them.
var ruleStrs []string
for _, r := range rules {
ruleStrs = append(ruleStrs, r.ruleStr)
ruleStrs = append(ruleStrs, r.ruleStrs...)
}
if err := r.routeClient.AddOrUpdateNodeNetworkPolicyIPTables([]string{iptChain}, [][]string{ruleStrs}, isIPv6); err != nil {
return err
Expand Down Expand Up @@ -636,32 +653,35 @@ func getIPNetsFromRule(rule *CompletedRule, isIPv6 bool) sets.Set[string] {
return set
}

func buildCoreIPTRule(ipProtocol iptables.Protocol,
func buildCoreIPTRules(ipProtocol iptables.Protocol,
iptChain string,
ipset string,
ipnet string,
iptRuleTarget string,
iptRuleComment string,
service *v1beta2.Service,
isIngress bool) string {
isIngress bool,
enableLogging bool,
logLabel string) []string {
builder := iptables.NewRuleBuilder(iptChain)
var rules []string
if isIngress {
if ipset != "" {
builder = builder.MatchIPSetSrc(ipset, ipsetTypeHashIP)
} else if ipnet != "" {
builder = builder.MatchCIDRSrc(ipnet)
} else {
// If no source IP address is matched, return an empty string since the core iptables will never be matched.
return ""
// If no source IP address is matched, return an empty slice since the core iptables will never be matched.
return rules
}
} else {
if ipset != "" {
builder = builder.MatchIPSetDst(ipset, ipsetTypeHashIP)
} else if ipnet != "" {
builder = builder.MatchCIDRDst(ipnet)
} else {
// If no destination IP address is matched, return an empty string since the core iptables will never be matched.
return ""
// If no destination IP address is matched, return an empty slice since the core iptables will never be matched.
return rules
}
}
if service != nil {
Expand All @@ -679,13 +699,26 @@ func buildCoreIPTRule(ipProtocol iptables.Protocol,
builder = builder.MatchICMP(service.ICMPType, service.ICMPCode, ipProtocol)
}
}
return builder.SetTarget(iptRuleTarget).
if enableLogging {
rules = append(rules, builder.CopyBuilder().
SetTarget(iptables.LOGTarget).
SetLogPrefix(logLabel).
Done().
GetRule())
}
rules = append(rules, builder.SetTarget(iptRuleTarget).
SetComment(iptRuleComment).
Done().
GetRule()
GetRule())
return rules
}

func buildServiceIPTRules(ipProtocol iptables.Protocol, services []v1beta2.Service, chain string, ruleTarget string) []string {
func buildServiceIPTRules(ipProtocol iptables.Protocol,
services []v1beta2.Service,
chain string,
ruleTarget string,
enableLogging bool,
logLabel string) []string {
var rules []string
builder := iptables.NewRuleBuilder(chain)
for _, svc := range services {
Expand All @@ -703,6 +736,13 @@ func buildServiceIPTRules(ipProtocol iptables.Protocol, services []v1beta2.Servi
case "icmp":
copiedBuilder = copiedBuilder.MatchICMP(svc.ICMPType, svc.ICMPCode, ipProtocol)
}
if enableLogging {
rules = append(rules, copiedBuilder.CopyBuilder().
SetTarget(iptables.LOGTarget).
SetLogPrefix(logLabel).
Done().
GetRule())
}
rules = append(rules, copiedBuilder.SetTarget(ruleTarget).
Done().
GetRule())
Expand All @@ -729,3 +769,24 @@ func getServiceTransProtocol(protocol *v1beta2.Protocol) string {
}
return strings.ToLower(string(*protocol))
}

func generateLogLabel(rule *CompletedRule) string {
// Construct the log label used as iptables log prefix. According to https://ipset.netfilter.org/iptables-extensions.man.html,
// the log prefix is up to 29 letters long. The log label should include essential information to help filter the
// generated iptables kernel log. As a result, the user-provided log label is limited to 12 characters.
// The log label format:
// |Antrea|:|I|:|Reject|:|user-provided label|:|
// |6 |1|1|1|4-6 |1|1-12 |1|
logLabel := fmt.Sprintf("%s:%s:%s", logLabelPrefix, rule.Direction[:1], *rule.Action)
if rule.LogLabel != "" {
ruleLogLabel := rule.LogLabel
// Truncate the user-provided log label if it exceeds 12 characters.
if len(ruleLogLabel) > 12 {
klog.InfoS("The rule log label that exceeds 12 characters will be truncated", "rule.LogLabel", rule.LogLabel)
ruleLogLabel = ruleLogLabel[:12]
}
logLabel = fmt.Sprintf("%s:%s", logLabel, ruleLogLabel)
}
logLabel += ":"
return logLabel
}
Loading
Loading