MongoDBCommunity in Pending State

[pedronieto1206]

MongoDB Community Agent Readiness Issue Report

Issue Summary

MongoDB Community instances created via Crossplane composition are failing to reach ready state due to MongoDB agents not reaching their goal configuration version. The agents remain stuck at version 35 while the target goal is version 36, with one agent reporting version -1.

Environment Information

• Date: October 20, 2025
• Kubernetes Cluster: AWS EKS (eu-west-1)
• MongoDB Community Operator Version: Based on logs from mongodb-kubernetes-operator-55c5fddf4d-2j5tg
• MongoDB Agent Version: 108.0.2.8729-1
• MongoDB Community Server Version: 6.0.5-ubi8
• Namespace: sample-service
• Instance Name: mongodb-sample-service-instance

Problem Description

Current Status

The MongoDB Community instance has been in Pending state for over 30 minutes with the following symptoms:

Command: kubectl describe mongodbcommunity mongodb-sample-service-instance -n sample-service

status:
  currentMongoDBMembers: 0
  currentStatefulSetReplicas: 0
  message: "ReplicaSet is not yet ready, retrying in 10 seconds"
  mongoUri: ""
  phase: Pending

Root Cause Analysis

Primary Issue: MongoDB agents are not synchronizing to the target configuration version:

• Pod 0: Agent at version 35, goal 36 (1 version behind)
• Pod 1: Agent at version 35, goal 36 (1 version behind)
• Pod 2: Agent at version -1, goal 36 (completely failed state)

Technical Details

Pod Status

All 3 StatefulSet pods are running (3/3 Ready) but readiness probes are failing:

Command: kubectl get pods -n sample-service -l app=mongodb-sample-service-instance-svc -o wide

NAME                                READY   STATUS    RESTARTS      AGE
mongodb-sample-service-instance-0   3/3     Running   0             30m
mongodb-sample-service-instance-1   3/3     Running   2 (27m ago)   29m
mongodb-sample-service-instance-2   3/3     Running   0             28m

MongoDB Operator Logs

The operator continuously logs the following pattern every 10 seconds:

Command: kubectl logs -n mongodb-operator mongodb-kubernetes-operator-55c5fddf4d-2j5tg --tail=20

2025-10-20T08:22:21.589Z DEBUG agent/agent_readiness.go:113 The Agent in the Pod 'mongodb-sample-service-instance-0' hasn't reached the goal state yet (goal: 36, agent: 35)
2025-10-20T08:22:21.589Z DEBUG agent/agent_readiness.go:113 The Agent in the Pod 'mongodb-sample-service-instance-1' hasn't reached the goal state yet (goal: 36, agent: 35)
2025-10-20T08:22:21.590Z DEBUG agent/agent_readiness.go:113 The Agent in the Pod 'mongodb-sample-service-instance-2' hasn't reached the goal state yet (goal: 36, agent: -1)
2025-10-20T08:22:21.618Z INFO controllers/mongodb_status_options.go:110 ReplicaSet is not yet ready, retrying in 10 seconds

Readiness Probe Failures

Pod readiness probes consistently fail with:

Command: kubectl describe pod mongodb-sample-service-instance-0 -n sample-service

{
  "level":"debug",
  "ts":"2025-10-20T08:14:19.570Z",
  "msg":"The Agent hasn't reported working on the new config yet, the last plan finished at 2025-10-20T07:56:06Z"
}
{
  "level":"info",
  "ts":"2025-10-20T08:14:19.570Z",
  "msg":"Reached the end of the check. Returning not ready."
}

Configuration Details

The MongoDB Community resource includes an agent.systemLog configuration that may be contributing to the issue:

Command: kubectl get mongodbcommunity mongodb-sample-service-instance -n sample-service -o yaml

spec:
  agent:
    systemLog:
      destination: file
      logAppend: false
      path: /var/log/mongodb-mms-automation/mongod.log
      verbosity: 1
  additionalMongodConfig:
    replication.replSetName: mongodb-sample-service-instance
    storage.wiredTiger.engineConfig.journalCompressor: zlib
    systemLog.verbosity: 5

Impact Assessment

Immediate Impact

• MongoDB URI unavailable: The mongoUri field remains empty, blocking dependent applications

Resource State

• Kubernetes resources: All underlying pods, services, and PVCs are healthy
• StatefulSet: Shows 3/3 ready replicas
• Storage: PVCs are successfully attached and mounted
• Network: Service discovery and networking appear functional