Skip to content

Commit

Permalink
Add EnableLogging and LogLabel supports for Node NetworkPolicy
Browse files Browse the repository at this point in the history 
This commit introduces limited support for traffic logging in Node
NetworkPolicy. The limitations are:

- Traffic logs are written only to the system log (not managed by Antrea).
  Users can filter logs using syslog filters.
- The `LogLabel` for Node NetworkPolicy is restricted to a maximum of
  12 characters.

Node NetworkPolicy's data path is implemented via iptables. An iptables
"non-terminating target" `LOG` is added before the final matching rule to
log packets to the system kernel log. The logs provide packet match details,
such as:

```
Sep  2 10:31:07 k8s-node-control-plane kernel: [6657320.789675] Antrea:I:Allow:allow-http:IN=ens224 OUT= MAC=00:50:56:a7:fb:18:00:50:56:a7:23:47:08:00 SRC=10.10.0.10 DST=192.168.240.200 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=52813 DF PROTO=TCP SPT=57658 DPT=80 WINDOW=64240 RES=0x00 SYN URGP=0
Sep  2 10:31:11 k8s-node-control-plane kernel: [6657324.899219] Antrea:I:Drop:default-drop:IN=ens224 OUT= MAC=00:50:56:a7:fb:18:00:50:56:a7:23:47:08:00 SRC=192.168.240.201 DST=192.168.240.200 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=27486 DF PROTO=TCP SPT=33152 DPT=80 WINDOW=64240 RES=0x00 SYN URGP=0
```

The log prefix (e.g., `Antrea:I:Allow:allow-http:`) is up to 29 characters long
and includes a user-provided log label (up to 12 characters). The log prefix format:

```
|---1--| |2| |---3--| |----------4--------|
|Antrea|:|I|:|Reject|:|user-provided label|:|
|6     |1|1|1|4-6   |1|1-12               |1|
```

- Part 1: Fixed, "Antrea"
- Part 2: Direction, "I" (In) or "O" (Out)
- Part 3: Action, "Allow", "Drop", or "Reject"
- Part 4: User-provided log label, up to 12 characters

Signed-off-by: Hongliang Liu <hongliang.liu@broadcom.com>
  • Loading branch information
@hongliangl
hongliangl committed Oct 9, 2024
1 parent 2722184 commit 4feb47c
Show file tree
Hide file tree
Showing 10 changed files with 517 additions and 151 deletions.
59 changes: 58 additions & 1 deletion docs/antrea-node-network-policy.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@
- [Introduction](#introduction)
- [Prerequisites](#prerequisites)
- [Usage](#usage)
- [Logs](#logs)
- [Limitations](#limitations)
<!-- /toc -->

Expand Down Expand Up @@ -66,11 +67,15 @@ spec:
ports:
- protocol: TCP
port: 80
enableLogging: true
logLabel: allow-http
- name: drop-other
action: Drop
ports:
- protocol: TCP
port: 80
enableLogging: true
logLabel: default-drop-others
```
An example Node NetworkPolicy that blocks egress traffic from Nodes with label
Expand Down Expand Up @@ -105,6 +110,58 @@ spec:
port: 22
```

## Logs

The `enableLogging` and `logLabel` options provide limited support for Node NetworkPolicies. Since Node NetworkPolicies
are implemented using iptables, enabling `enableLogging` causes the Linux kernel to log information about all matching
packets via the kernel log. However, Antrea cannot process these logs directly. Instead, these logs can be accessed
through syslog, allowing you to filter and direct them to specific files using syslog syntax.

By default, `enableLogging` is unsupported in KinD clusters. To enable it, set the host’s
`/proc/sys/net/netfilter/nf_log_all_netns` to 1. Antrea uses the iptables `LOG` target to log packet information,
but by default, `/proc/sys/net/netfilter/nf_log_all_netns` is 0, preventing containers from logging to the kernel to
avoid clutter. If logging is required, you can enable it by setting the value to 1, but please be cautious to do so
unless you are clear about the impact.

For example, consider the Node NetworkPolicy `restrict-http-to-node` above. It could generate the following logs:

```text
Sep 2 10:31:07 k8s-node-control-plane kernel: [6657320.789675] Antrea:I:Allow:allow-http:IN=ens224 OUT= MAC=00:50:56:a7:fb:18:00:50:56:a7:23:47:08:00 SRC=10.10.0.10 DST=192.168.240.200 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=52813 DF PROTO=TCP SPT=57658 DPT=80 WINDOW=64240 RES=0x00 SYN URGP=0
Sep 2 10:31:11 k8s-node-control-plane kernel: [6657324.899219] Antrea:I:Drop:default-drop:IN=ens224 OUT= MAC=00:50:56:a7:fb:18:00:50:56:a7:23:47:08:00 SRC=192.168.240.201 DST=192.168.240.200 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=27486 DF PROTO=TCP SPT=33152 DPT=80 WINDOW=64240 RES=0x00 SYN URGP=0
```

In these logs, prefixes like `Antrea:I:Allow:allow-http:` and `Antrea:I:Drop:default-drop:` are added by iptables using
the `--log-prefix` parameter. The iptables log prefix is limited to 29 characters, as described in the
[iptables-extensions manual](https://ipset.netfilter.org/iptables-extensions.man.html).

The log prefix format includes essential information of a Node NetworkPolicy rule, and consists of four parts,
formatted as follows:

```text
|---1--| |2| |---3--| |----------4--------|
|Antrea|:|I|:|Reject|:|user-provided label|:|
|6 |1|1|1|4-6 |1|1-12 |1|
```

- Part 1: Fixed, "Antrea"
- Part 2: Direction, "I" (In) or "O" (Out)
- Part 3: Action, "Allow", "Drop", or "Reject"
- Part 4: User-provided `logLabel`, up to 12 characters

Due to iptables' 29-character prefix limitation, the user-provided `logLabel` is restricted to a maximum of 12 characters.
To manage these logs effectively, you can configure rsyslog on each Node as follows:

```text
# Example rsyslog configuration to filter Antrea logs
:msg, contains, "Antrea:I:Allow:allow-http" /var/log/antrea-node-netpol-allow.log
:msg, contains, "Antrea:I:Drop:default-drop" /var/log/antrea-node-netpol-drop.log
& stop
```

This configuration directs logs with the prefix `Antrea:I:Allow:allow-http` to `/var/log/antrea-node-netpol-allow.log`
and logs with the prefix `Antrea:I:Drop:default-drop` to `/var/log/antrea-node-netpol-drop.log`. The `& stop` command
ensures that these logs are not processed further.

## Limitations

- This feature is currently only supported for Linux Nodes.
Expand All @@ -118,4 +175,4 @@ spec:
- FQDN is not supported for ACNPs applied to Nodes.
- Layer 7 NetworkPolicy is not supported yet.
- For UDP or SCTP, when the `Reject` action is specified in an egress rule, it behaves identical to the `Drop` action.
- Traffic logging is not supported yet for ACNPs applied to Nodes.
- Limited support for traffic logging for ACNPs applied to Nodes.
107 changes: 84 additions & 23 deletions pkg/agent/controller/networkpolicy/node_reconciler_linux.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -43,6 +43,10 @@ const (
ipv6Any = "::/0"
)

// The logging of Node NetworkPolicy is implemented by iptables target LOG, which turns on kernel logging of matching
// packets. The label is useful for distinguishing Node NetworkPolicy logs from other kernel logs.
const logLabelPrefix = "Antrea"

var ipsetTypeHashIP = ipset.HashIP

/*
Expand Down Expand Up @@ -124,7 +128,7 @@ directly.
type coreIPTRule struct {
ruleID string
priority *types.Priority
ruleStr string
ruleStrs []string
}

type chainKey struct {
Expand Down Expand Up @@ -256,7 +260,7 @@ func (r *nodeReconciler) batchAdd(rules []*CompletedRule) error {
}

// Collect all core iptables rules.
coreIPTRule := &coreIPTRule{ruleID, iptRule.Priority, iptRule.CoreIPTRule}
coreIPTRule := &coreIPTRule{ruleID, iptRule.Priority, iptRule.CoreIPTRules}
if rule.Direction == v1beta2.DirectionIn {
ingressCoreIPTRules[ipProtocol] = append(ingressCoreIPTRules[ipProtocol], coreIPTRule)
} else {
Expand Down Expand Up @@ -322,6 +326,11 @@ func (r *nodeReconciler) GetRuleByFlowID(ruleFlowID uint32) (*types.PolicyRule,

func (r *nodeReconciler) computeIPTRules(rule *CompletedRule) (map[iptables.Protocol]*types.NodePolicyRule, *nodePolicyLastRealized) {
ruleID := rule.ID
enableLogging := rule.EnableLogging
var logLabel string
if enableLogging {
logLabel = generateLogLabel(rule)
}
lastRealized := newNodePolicyLastRealized()
priority := &types.Priority{
TierPriority: *rule.TierPriority,
Expand Down Expand Up @@ -362,7 +371,12 @@ func (r *nodeReconciler) computeIPTRules(rule *CompletedRule) (map[iptables.Prot

var serviceIPTRules []string
if serviceIPTChain != "" {
serviceIPTRules = buildServiceIPTRules(ipProtocol, rule.Services, serviceIPTChain, serviceIPTRuleTarget)
serviceIPTRules = buildServiceIPTRules(ipProtocol,
rule.Services,
serviceIPTChain,
serviceIPTRuleTarget,
enableLogging,
logLabel)
}

ipnets := getIPNetsFromRule(rule, isIPv6)
Expand All @@ -383,14 +397,19 @@ func (r *nodeReconciler) computeIPTRules(rule *CompletedRule) (map[iptables.Prot
lastRealized.ipnets[ipProtocol] = ipnet
}

coreIPTRule := buildCoreIPTRule(ipProtocol,
coreIPTRules := buildCoreIPTRules(ipProtocol,
coreIPTChain,
ipset,
ipnet,
coreIPTRuleTarget,
coreIPTRuleComment,
service,
rule.Direction == v1beta2.DirectionIn)
rule.Direction == v1beta2.DirectionIn,
// If the target of a core iptables rule is not a service chain, the iptables rule for logging should be
// generated along with the core iptables rule. Otherwise, the iptables rules for logging should be generated
// along with the service iptables rules.
enableLogging && serviceIPTChain == "",
logLabel)

nodePolicyRules[ipProtocol] = &types.NodePolicyRule{
IPSet: ipset,
Expand All @@ -399,7 +418,7 @@ func (r *nodeReconciler) computeIPTRules(rule *CompletedRule) (map[iptables.Prot
ServiceIPTChain: serviceIPTChain,
ServiceIPTRules: serviceIPTRules,
CoreIPTChain: coreIPTChain,
CoreIPTRule: coreIPTRule,
CoreIPTRules: coreIPTRules,
IsIPv6: isIPv6,
}
}
Expand All @@ -422,7 +441,7 @@ func (r *nodeReconciler) add(rule *CompletedRule) error {
return err
}
}
if err := r.addOrUpdateCoreIPTRules(iptRule.CoreIPTChain, iptRule.IsIPv6, false, &coreIPTRule{ruleID, iptRule.Priority, iptRule.CoreIPTRule}); err != nil {
if err := r.addOrUpdateCoreIPTRules(iptRule.CoreIPTChain, iptRule.IsIPv6, false, &coreIPTRule{ruleID, iptRule.Priority, iptRule.CoreIPTRules}); err != nil {
return err
}
}
Expand Down Expand Up @@ -456,15 +475,15 @@ func (r *nodeReconciler) update(lastRealized *nodePolicyLastRealized, newRule *C
return err
}
if shouldUpdateCoreIPTRules {
if err := r.addOrUpdateCoreIPTRules(iptRule.CoreIPTChain, iptRule.IsIPv6, true, &coreIPTRule{ruleID, iptRule.Priority, iptRule.CoreIPTRule}); err != nil {
if err := r.addOrUpdateCoreIPTRules(iptRule.CoreIPTChain, iptRule.IsIPv6, true, &coreIPTRule{ruleID, iptRule.Priority, iptRule.CoreIPTRules}); err != nil {
return err
}
}
} else if prevIPSet != "" {
// If the previous rule used an ipset, sync the new core iptables rule first to remove its reference, then
// delete the unused ipset.
if shouldUpdateCoreIPTRules {
if err := r.addOrUpdateCoreIPTRules(iptRule.CoreIPTChain, iptRule.IsIPv6, true, &coreIPTRule{ruleID, iptRule.Priority, iptRule.CoreIPTRule}); err != nil {
if err := r.addOrUpdateCoreIPTRules(iptRule.CoreIPTChain, iptRule.IsIPv6, true, &coreIPTRule{ruleID, iptRule.Priority, iptRule.CoreIPTRules}); err != nil {
return err
}
}
Expand All @@ -473,7 +492,7 @@ func (r *nodeReconciler) update(lastRealized *nodePolicyLastRealized, newRule *C
}
} else {
if shouldUpdateCoreIPTRules {
if err := r.addOrUpdateCoreIPTRules(iptRule.CoreIPTChain, iptRule.IsIPv6, true, &coreIPTRule{ruleID, iptRule.Priority, iptRule.CoreIPTRule}); err != nil {
if err := r.addOrUpdateCoreIPTRules(iptRule.CoreIPTChain, iptRule.IsIPv6, true, &coreIPTRule{ruleID, iptRule.Priority, iptRule.CoreIPTRules}); err != nil {
return err
}
}
Expand Down Expand Up @@ -517,9 +536,7 @@ func (r *nodeReconciler) addOrUpdateCoreIPTRules(chain string, isIPv6 bool, isUp
// Get all iptables rules and synchronize them.
var ruleStrs []string
for _, rule := range rules {
if rule.ruleStr != "" {
ruleStrs = append(ruleStrs, rule.ruleStr)
}
ruleStrs = append(ruleStrs, rule.ruleStrs...)
}
if err := r.routeClient.AddOrUpdateNodeNetworkPolicyIPTables([]string{chain}, [][]string{ruleStrs}, isIPv6); err != nil {
return err
Expand Down Expand Up @@ -554,7 +571,7 @@ func (r *nodeReconciler) deleteCoreIPTRule(ruleID string, iptChain string, isIPv
// Get all the iptables rules and synchronize them.
var ruleStrs []string
for _, r := range rules {
ruleStrs = append(ruleStrs, r.ruleStr)
ruleStrs = append(ruleStrs, r.ruleStrs...)
}
if err := r.routeClient.AddOrUpdateNodeNetworkPolicyIPTables([]string{iptChain}, [][]string{ruleStrs}, isIPv6); err != nil {
return err
Expand Down Expand Up @@ -636,32 +653,35 @@ func getIPNetsFromRule(rule *CompletedRule, isIPv6 bool) sets.Set[string] {
return set
}

func buildCoreIPTRule(ipProtocol iptables.Protocol,
func buildCoreIPTRules(ipProtocol iptables.Protocol,
iptChain string,
ipset string,
ipnet string,
iptRuleTarget string,
iptRuleComment string,
service *v1beta2.Service,
isIngress bool) string {
isIngress bool,
enableLogging bool,
logLabel string) []string {
builder := iptables.NewRuleBuilder(iptChain)
var rules []string
if isIngress {
if ipset != "" {
builder = builder.MatchIPSetSrc(ipset, ipsetTypeHashIP)
} else if ipnet != "" {
builder = builder.MatchCIDRSrc(ipnet)
} else {
// If no source IP address is matched, return an empty string since the core iptables will never be matched.
return ""
// If no source IP address is matched, return an empty slice since the core iptables will never be matched.
return rules
}
} else {
if ipset != "" {
builder = builder.MatchIPSetDst(ipset, ipsetTypeHashIP)
} else if ipnet != "" {
builder = builder.MatchCIDRDst(ipnet)
} else {
// If no destination IP address is matched, return an empty string since the core iptables will never be matched.
return ""
// If no destination IP address is matched, return an empty slice since the core iptables will never be matched.
return rules
}
}
if service != nil {
Expand All @@ -679,13 +699,26 @@ func buildCoreIPTRule(ipProtocol iptables.Protocol,
builder = builder.MatchICMP(service.ICMPType, service.ICMPCode, ipProtocol)
}
}
return builder.SetTarget(iptRuleTarget).
if enableLogging {
rules = append(rules, builder.CopyBuilder().
SetTarget(iptables.LOGTarget).
SetLogPrefix(logLabel).
Done().
GetRule())
}
rules = append(rules, builder.SetTarget(iptRuleTarget).
SetComment(iptRuleComment).
Done().
GetRule()
GetRule())
return rules
}

func buildServiceIPTRules(ipProtocol iptables.Protocol, services []v1beta2.Service, chain string, ruleTarget string) []string {
func buildServiceIPTRules(ipProtocol iptables.Protocol,
services []v1beta2.Service,
chain string,
ruleTarget string,
enableLogging bool,
logLabel string) []string {
var rules []string
builder := iptables.NewRuleBuilder(chain)
for _, svc := range services {
Expand All @@ -703,6 +736,13 @@ func buildServiceIPTRules(ipProtocol iptables.Protocol, services []v1beta2.Servi
case "icmp":
copiedBuilder = copiedBuilder.MatchICMP(svc.ICMPType, svc.ICMPCode, ipProtocol)
}
if enableLogging {
rules = append(rules, copiedBuilder.CopyBuilder().
SetTarget(iptables.LOGTarget).
SetLogPrefix(logLabel).
Done().
GetRule())
}
rules = append(rules, copiedBuilder.SetTarget(ruleTarget).
Done().
GetRule())
Expand All @@ -729,3 +769,24 @@ func getServiceTransProtocol(protocol *v1beta2.Protocol) string {
}
return strings.ToLower(string(*protocol))
}

func generateLogLabel(rule *CompletedRule) string {
// Construct the log label used as iptables log prefix. According to https://ipset.netfilter.org/iptables-extensions.man.html,
// the log prefix is up to 29 letters long. The log label should include essential information to help filter the
// generated iptables kernel log. As a result, the user-provided log label is limited to 12 characters.
// The log label format:
// |Antrea|:|I|:|Reject|:|user-provided label|:|
// |6 |1|1|1|4-6 |1|1-12 |1|
logLabel := fmt.Sprintf("%s:%s:%s", logLabelPrefix, rule.Direction[:1], *rule.Action)
if rule.LogLabel != "" {
ruleLogLabel := rule.LogLabel
// Truncate the user-provided log label if it exceeds 12 characters.
if len(ruleLogLabel) > 12 {
klog.InfoS("The rule log label that exceeds 12 characters will be truncated", "rule.LogLabel", rule.LogLabel)
ruleLogLabel = ruleLogLabel[:12]
}
logLabel = fmt.Sprintf("%s:%s", logLabel, ruleLogLabel)
}
logLabel += ":"
return logLabel
}
Loading

0 comments on commit 4feb47c

Please sign in to comment.