Creating an invalid agent (YAML) causes all agents to not load in the UI

[christian-posta]

try creating an invalid agent, the UI doesn't handle that gracefully

[inFocus7]

No resolution branch on my end [yet], but anyone who wants to reproduce this issue:

1. install kagent with the default Agents installed
2. Edit an Agent CR to delete the modelConfig (or update string to non-existent modelConfig)
3. Failure:

• UI Agents screen does not load.
• The Agent itself does have an appropiate status: Accepted -> False: AgentReconcileFailure

UI logs OK, Controller logs:

2025-08-22T18:47:15Z    DEBUG    http    Request started    {"method": "GET", "path": "/api/agents", "remote_addr": "10.244.0.15:35114", "user_id": "admin@kagent.dev"}
2025-08-22T18:47:15Z    DEBUG    http.agents-handler    Processing Agent    {"method": "GET", "path": "/api/agents", "remote_addr": "10.244.0.15:35114", "user_id": "admin@kagent.dev", "operation": "list-db", "agentRef": "kagent/argo-rollouts-conversion-agent"}
2025-08-22T18:47:15Z    DEBUG    http.agents-handler    Processing Agent    {"method": "GET", "path": "/api/agents", "remote_addr": "10.244.0.15:35114", "user_id": "admin@kagent.dev", "operation": "list-db", "agentRef": "kagent/argo-rollouts-conversion-agent"}
2025-08-22T18:47:15Z    DEBUG    http.agents-handler    Processing Agent    {"method": "GET", "path": "/api/agents", "remote_addr": "10.244.0.15:35114", "user_id": "admin@kagent.dev", "operation": "list-db", "agentRef": "kagent/cilium-debug-agent"}
2025-08-22T18:47:15Z    DEBUG    http.agents-handler    Processing Agent    {"method": "GET", "path": "/api/agents", "remote_addr": "10.244.0.15:35114", "user_id": "admin@kagent.dev", "operation": "list-db", "agentRef": "kagent/cilium-debug-agent"}
2025-08-22T18:47:15Z    DEBUG    http.agents-handler    Processing Agent    {"method": "GET", "path": "/api/agents", "remote_addr": "10.244.0.15:35114", "user_id": "admin@kagent.dev", "operation": "list-db", "agentRef": "kagent/cilium-policy-agent"}
2025-08-22T18:47:15Z    DEBUG    http.agents-handler    Processing Agent    {"method": "GET", "path": "/api/agents", "remote_addr": "10.244.0.15:35114", "user_id": "admin@kagent.dev", "operation": "list-db", "agentRef": "kagent/cilium-policy-agent"}
2025-08-22T18:47:15Z    DEBUG    http.agents-handler    Processing Agent    {"method": "GET", "path": "/api/agents", "remote_addr": "10.244.0.15:35114", "user_id": "admin@kagent.dev", "operation": "list-db", "agentRef": "kagent/helm-agent"}
2025-08-22T18:47:15Z    DEBUG    http.agents-handler    Processing Agent    {"method": "GET", "path": "/api/agents", "remote_addr": "10.244.0.15:35114", "user_id": "admin@kagent.dev", "operation": "list-db", "agentRef": "kagent/helm-agent"}
2025-08-22T18:47:15Z    DEBUG    http.agents-handler    Processing Agent    {"method": "GET", "path": "/api/agents", "remote_addr": "10.244.0.15:35114", "user_id": "admin@kagent.dev", "operation": "list-db", "agentRef": "kagent/k8s-agent"}
2025-08-22T18:47:15Z    DEBUG    http.agents-handler    Processing Agent    {"method": "GET", "path": "/api/agents", "remote_addr": "10.244.0.15:35114", "user_id": "admin@kagent.dev", "operation": "list-db", "agentRef": "kagent/k8s-agent"}
2025-08-22T18:47:15Z    DEBUG    http.agents-handler    ModelConfig not found    {"method": "GET", "path": "/api/agents", "remote_addr": "10.244.0.15:35114", "user_id": "admin@kagent.dev", "operation": "list-db", "modelConfigRef": {"name":"default-model-config-2","namespace":"kagent"}}
2025-08-22T18:47:15Z    ERROR    http    Unhandled error    {"method": "GET", "path": "/api/agents", "remote_addr": "10.244.0.15:35114", "user_id": "admin@kagent.dev", "error": "ModelConfig.kagent.dev \"default-model-config-2\" not found"}
github.com/kagent-dev/kagent/go/internal/httpserver.(*errorResponseWriter).RespondWithError
    /workspace/internal/httpserver/middleware_error.go:56
github.com/kagent-dev/kagent/go/internal/httpserver/handlers.(*AgentsHandler).HandleListAgents
    /workspace/internal/httpserver/handlers/agents.go:53
github.com/kagent-dev/kagent/go/internal/httpserver.(*HTTPServer).setupRoutes.adaptHandler.func23
    /workspace/internal/httpserver/server.go:217
net/http.HandlerFunc.ServeHTTP
    /usr/lib/go/src/net/http/server.go:2322
github.com/kagent-dev/kagent/go/internal/httpserver.errorHandlerMiddleware.func1
    /workspace/internal/httpserver/middleware_error.go:19
net/http.HandlerFunc.ServeHTTP
    /usr/lib/go/src/net/http/server.go:2322
github.com/kagent-dev/kagent/go/internal/httpserver.loggingMiddleware.func1
    /workspace/internal/httpserver/middleware.go:28
net/http.HandlerFunc.ServeHTTP
    /usr/lib/go/src/net/http/server.go:2322
github.com/kagent-dev/kagent/go/internal/httpserver.contentTypeMiddleware.func1
    /workspace/internal/httpserver/middleware.go:74
net/http.HandlerFunc.ServeHTTP
    /usr/lib/go/src/net/http/server.go:2322
github.com/kagent-dev/kagent/go/internal/httpserver.(*HTTPServer).setupRoutes.AuthnMiddleware.func39.1
    /workspace/internal/httpserver/auth/authn.go:67
net/http.HandlerFunc.ServeHTTP
    /usr/lib/go/src/net/http/server.go:2322
github.com/gorilla/mux.(*Router).ServeHTTP
    /root/go/pkg/mod/github.com/gorilla/mux@v1.8.1/mux.go:212
net/http.serverHandler.ServeHTTP
    /usr/lib/go/src/net/http/server.go:3340
net/http.(*conn).serve
    /usr/lib/go/src/net/http/server.go:2109
2025-08-22T18:47:15Z    INFO    http    Request completed    {"method": "GET", "path": "/api/agents", "remote_addr": "10.244.0.15:35114", "user_id": "admin@kagent.dev", "status": 500, "duration": "273.5µs"}
2025-08-22T18:47:15Z    DEBUG    http    Request started    {"method": "GET", "path": "/api/tools", "remote_addr": "10.244.0.15:35114", "user_id": "admin@kagent.dev"}
2025-08-22T18:47:15Z    DEBUG    http.tools-handler    Listing tools from database    {"method": "GET", "path": "/api/tools", "remote_addr": "10.244.0.15:35114", "user_id": "admin@kagent.dev", "operation": "list-db", "userID": "admin@kagent.dev"}
2025-08-22T18:47:15Z    INFO    http.tools-handler    Successfully listed tools    {"method": "GET", "path": "/api/tools", "remote_addr": "10.244.0.15:35114", "user_id": "admin@kagent.dev", "operation": "list-db", "userID": "admin@kagent.dev", "count": 157}
2025-08-22T18:47:15Z    INFO    http    Request completed    {"method": "GET", "path": "/api/tools", "remote_addr": "10.244.0.15:35114", "user_id": "admin@kagent.dev", "status": 200, "duration": "1.022584ms"}
2025-08-22T18:47:15Z    DEBUG    http    Request started    {"method": "GET", "path": "/api/modelconfigs", "remote_addr": "10.244.0.15:35114", "user_id": "admin@kagent.dev"}
2025-08-22T18:47:15Z    INFO    http.modelconfig-handler    Listing ModelConfigs    {"method": "GET", "path": "/api/modelconfigs", "remote_addr": "10.244.0.15:35114", "user_id": "admin@kagent.dev", "operation": "list"}
2025-08-22T18:47:15Z    INFO    http.modelconfig-handler    Successfully listed ModelConfigs    {"method": "GET", "path": "/api/modelconfigs", "remote_addr": "10.244.0.15:35114", "user_id": "admin@kagent.dev", "operation": "list", "count": 1}
2025-08-22T18:47:15Z    INFO    http    Request completed    {"method": "GET", "path": "/api/modelconfigs", "remote_addr": "10.244.0.15:35114", "user_id": "admin@kagent.dev", "status": 200, "duration": "99.417µs"}

[inFocus7]

In our httphander for agents (and maybe others), we exit early with an error if there's an issue -- for agents it can be modelConfigs. (this err propagates to the caller (e.g. list agents), which returns an error and 0 agents).

This isn't ideal, so we'll either want to:

1. Not return failed agents (but do still log warnings/errors in the controller)
2. Return the agents with an extra response field like Reconciled and display them in the UI with a Failed toast/indicator -- similar to the Ready/Not Ready one.

I have no strong opinions. I'm slightly leaning towards the first for simplicity (not needing to add more response fields and UI). I'm not 100% certain who the end user(s) of the UI would be, or if they'd care about Failed statuses vs. just having it in the CR's condition and logs for the agent deployers.

[EItanya]

Closed with #803