Deleting all IngressLink CR resources resetting default RD of CIS-managed Partition to RD 0

[adityoari]

Setup Details

CIS Version : 2.16.1
Build: f5networks/k8s-bigip-ctlr:latest
BIGIP Version: Big IP 17.1.1.3
AS3 Version: 3.46.2
Agent Mode: AS3
Orchestration: OCP
Orchestration Version: 4.14.16
Pool Mode: Cluster
Additional Setup details: Static Routes, CRD, IngressLink

Description

After deleting all IngressLink CRs from the cluster, CIS would push bare AS3 declaration with no defaultRouteDomain parameter, resetting the Partition default RD to 0 and breaking subsequent Static Route push.

Steps To Reproduce

1. Create BIG-IP Partition with non-0 RD (e.g. RD1) to be managed by CIS
2. Deploy CIS in ClusterIP + Static Route + CRD + IngressLink + RD1
3. Create IngressLink CR(s)
4. Delete all IngressLink CRs
5. Observe CIS-managed Partition in BIG-IP (tmsh list auth partition) --> Partition will have RD0 as default RD

Expected Result

Default RD for CIS-managed Partition is consistent with both CIS deployment args (i.e. --default-route-domain) and original Partition RD in BIG-IP

Actual Result

CIS-Managed Partition has RD reset to 0 with no IngressLink CR present, due to CIS pushed bare AS3 declaration without <tenant>/defaultRouteDomain parameter. This makes every Static Route push fails until another CR is created.
In the worst timing scenario, 30 seconds (or more, depends on the verify-interval parameter) would elapse after next CR is created with no connectivity, until the next Static Route push is accepted by BIG-IP.

Diagnostic Information

CIS Pod creation log

2024/05/05 12:25:46 [INFO] [INIT] Starting: Container Ingress Services - Version: 2.16.1, BuildInfo: azure-5932-c0934efcc07227fdc64bd9c8e17e8cc21a4bc3b2
..
2024/05/05 12:25:47 [DEBUG] [AS3] BIGIP is serving with AS3 version: 3.46.2
2024/05/05 12:25:47 [DEBUG] Controller Created
2024/05/05 12:25:47 [DEBUG] Client Created
..
2024/05/05 12:25:47 [DEBUG] Starting Custom Resource Worker
2024/05/05 12:25:47 [DEBUG] Processing Node Updates for static routes
2024/05/05 12:25:47 [DEBUG] [CCCL] ConfigWriter (0xc000194ed0) writing section name static-routes
2024/05/05 12:25:47 [DEBUG] [CCCL] ConfigWriter (0xc000194ed0) successfully wrote section (static-routes)
2024/05/05 12:25:47 [DEBUG] Wrote static route config section: {[]}

AS3 declaration after deleting CR

{
  "CIS-Inglink": {
    "Shared": {
      "class": "Application",
      "template": "shared"
    },
    "class": "Tenant",
    "label": "CIS-Inglink"
  },
  "class": "ADC",
  "controls": {
    "class": "Controls",
    "userAgent": "CIS/v2.16.1 OCP/v4.14.16",
    "archiveTimestamp": "2024-05-05T12:32:13.239Z"
  },
  "id": "urn:uuid:85626792-9ee7-46bb-8fc8-4ba708cfdc1d",
  "label": "CIS Declaration",
  "remark": "Auto-generated by CIS",
  "schemaVersion": "3.46.0",
  "updateMode": "selective"
}

Observations (if any)

Similar issue might also happen in other CR

[adityoari]

Can now confirm that the same symptom also happens in VirtualServer and TransportServer CRs

2024/05/06 12:58:08 [DEBUG] [AS3] Response from BIG-IP: code: 200 --- tenant:CIS-CRD --- message: success
2024/05/06 12:58:08 [DEBUG] [AS3] Raw response from Big-IP: map[declaration:map[CIS-CRD:map[Shared:map[class:Application template:shared] class:Tenant label:CIS-CRD] class:ADC controls:map[archiveTimestamp:2024-05-06T12:57:59.069Z class:Controls userAgent:CIS/v2.16.1 OCP/v4.14.16] id:urn:uuid:85626792-9ee7-46bb-8fc8-4ba708cfdc1d label:CIS Declaration remark:Auto-generated by CIS schemaVersion:3.46.0 updateMode:selective] results:[map[code:200 host:localhost lineCount:26 message:success runTime:2552 tenant:CIS-CRD]]]
...
2024/05/06 12:58:11 [ERROR] [2024-05-06 12:58:11,032 f5_cccl.service.manager ERROR] F5CcclResourceRequestError - 400 Unexpected Error: Bad Request for uri: https://10.0.0.9:443/mgmt/tm/net/route/ Text: '{"code":400,"message":"01070330:3: Static route gateway 10.0.0.133 is not directly connected via an interface.","errorStack":[],"apiError":3}'

[trinaths]

Created [CONTCNTR-4718] for internal tracking.