Troubleshooting blocked evaluation

[suikast42]

I have had played with the cpu settings of my vm. After that some allocations can't be placed.

With that Evaluation Response.

{
  "priority": 50,
  "type": "service",
  "triggeredBy": "queued-allocs",
  "status": "canceled",
  "statusDescription": "existing blocked evaluation exists for job \"observability\"",
  "failedTGAllocs": [
    {
      "Name": "mimir",
      "CoalescedFailures": 0,
      "NodesEvaluated": 1,
      "NodesExhausted": 0,
      "NodesAvailable": {
        "nomadder1": 1
      },
      "ClassFiltered": null,
      "ConstraintFiltered": null,
      "ClassExhausted": null,
      "DimensionExhausted": null,
      "QuotaExhausted": null,
      "Scores": null
    }
  ],
  "previousEval": "13dc8866-7b53-a94a-4d35-868913816f80",
  "nextEval": null,
  "blockedEval": null,
  "modifyIndex": 2518,
  "modifyTime": "2023-12-12T10:09:42.439Z",
  "createIndex": 2512,
  "createTime": "2023-12-12T10:09:35.334Z",
  "waitUntil": null,
  "namespace": "default",
  "plainJobId": "observability",
  "relatedEvals": [
    "13dc8866-7b53-a94a-4d35-868913816f80"
  ],
  "job": "[\"observability\",\"default\"]",
  "node": null
}

Nomad logs

Explore-logs-2023-12-12 11_11_32.txt

But how can I find the exact error for the placement error.

An information like not enough RAM or CPU could be very helpful.

[tgross]

Hi @suikast42! Nomad tries to keep only 1 evaluation per job in each stage of the eval broker, so that the scheduler doesn't have to run lots of no-op evaluations. (See Load Shedding in the Nomad Eval Broker if you want to learn more about this.)

So the evaluation you've got here was canceled, not blocked, because there was another blocked eval already:

"statusDescription": "existing blocked evaluation exists for job "observability"",

What you'll want to do is nomad eval list -job $jobid to find the blocked eval. Then you can nomad eval status on that evaluation to find out what it's blocked on.

[jrasell]

Hi @suikast42, I am going to close this issue as we have not heard back and the comment Tim seems accurate in describing the steps to take to understand the blocked eval. Thanks.

[suikast42]

Thanks. But I can't reproduce this anymore. Will look at back if that occurs again.

[github-actions]

I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.