-
Notifications
You must be signed in to change notification settings - Fork 113
Nomad Docs: Enhance client intro release note & add usage guide #1384
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change | ||||||||
|---|---|---|---|---|---|---|---|---|---|---|
|
|
@@ -11,13 +11,16 @@ description: |- | |||||||||
| In order to create a Nomad cluster out of individual nodes, you need to | ||||||||||
| introduce them to one another. There are several ways to perform this: | ||||||||||
|
|
||||||||||
| - Manually | ||||||||||
| - Manual bootstrap | ||||||||||
| - Cloud Auto-Join | ||||||||||
| - Consul | ||||||||||
|
|
||||||||||
| This tutorial describes each method and provides configuration snippets, which | ||||||||||
| This guide describes each method and provides configuration snippets, which | ||||||||||
| you can use as starting points for your own configuration. | ||||||||||
|
|
||||||||||
| You may also use client node introduction tokens to restrict which clients join | ||||||||||
| your cluster. | ||||||||||
|
|
||||||||||
| ## Manual clustering | ||||||||||
|
|
||||||||||
| Manually bootstrapping a Nomad cluster does not rely on additional tooling, but | ||||||||||
|
|
@@ -82,9 +85,9 @@ the client. This means only one server must be specified because, after initial | |||||||||
| contact, the full set of servers in the client's region are shared with the | ||||||||||
| client. | ||||||||||
|
|
||||||||||
| ## Join nodes using cloud auto-join | ||||||||||
| ## Use cloud auto-join | ||||||||||
|
|
||||||||||
| As of Nomad 0.8.4, [`retry_join`] accepts a unified interface using the | ||||||||||
| The [`retry_join`] parameter accepts a unified interface using the | ||||||||||
| [go-discover] library for doing automatic cluster joining using cloud metadata. | ||||||||||
| To use retry-join with a supported cloud provider, specify the configuration on | ||||||||||
| the command line or configuration file as a `key=value key=value ...` string. | ||||||||||
|
|
@@ -214,6 +217,221 @@ consul { | |||||||||
| Refer to the [`consul` stanza] documentation for the complete set of configuration | ||||||||||
| options. | ||||||||||
|
|
||||||||||
|
|
||||||||||
| ## Use client node introduction tokens | ||||||||||
|
|
||||||||||
| You may restrict which clients join your cluster by configuring client | ||||||||||
| introduction tokens. The client node introduction feature is like multi-factor | ||||||||||
| authentication for your Nomad clusters. It does not replace mTLS but adds a | ||||||||||
| second layer of security to prevent an unauthorized or misconfigured client from | ||||||||||
| joining a Nomad cluster. | ||||||||||
|
|
||||||||||
| Each layer answers a distinct question: | ||||||||||
|
|
||||||||||
| - mTLS: Does the client have valid certificates for the cluster? | ||||||||||
| - Client introduction token: Does the client have a valid token to join the | ||||||||||
| cluster? | ||||||||||
|
|
||||||||||
| You do not need to configure mTLS to use client node introduction tokens, but we | ||||||||||
| do recommend securing your cluster with mTLS. | ||||||||||
|
|
||||||||||
| You may optionally specify node names, node pools, and TTLs when you generate | ||||||||||
| client introduction tokens. | ||||||||||
|
|
||||||||||
| Follow these steps to use client node introduction tokens: | ||||||||||
|
|
||||||||||
| 1. [Create an ACL policy in which the node has write permissions](#create-an-acl-policy). | ||||||||||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
Simplifying this |
||||||||||
| 1. [Create an ACL role from the policy](#create-an-acl-role). | ||||||||||
| 1. [Generate a Nomad token](#generate-a-nomad-token). | ||||||||||
| 1. [Generate a client introduction token](#generate-a-client-introduction-token). | ||||||||||
| 1. [Configure the Nomad agent to use client introduction](#configure-the-nomad-agent). | ||||||||||
| 1. [Start the Nomad agent](#start-the-nomad-agent). | ||||||||||
| 1. [Monitor client join failures](#monitor-client-join-failures). | ||||||||||
|
|
||||||||||
| ### Create an ACL policy | ||||||||||
|
|
||||||||||
| This example creates the `node:write` policy required to generate tokens. | ||||||||||
|
|
||||||||||
| 1. Create a policy file called `client-introduction.hcl`. | ||||||||||
|
|
||||||||||
| ```hcl | ||||||||||
| node { | ||||||||||
| policy = "write" | ||||||||||
| } | ||||||||||
| ``` | ||||||||||
|
|
||||||||||
| 1. Create the policy. | ||||||||||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
To avoid repetition with the step before this one and clarify what is happening. |
||||||||||
|
|
||||||||||
| ```shell-session | ||||||||||
| nomad acl policy apply client-introduction client-introduction.hcl | ||||||||||
| ``` | ||||||||||
|
|
||||||||||
| ### Create an ACL role | ||||||||||
|
|
||||||||||
| This example creates an ACL role called `client-introduction`. | ||||||||||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
|
||||||||||
|
|
||||||||||
| ```shell-session | ||||||||||
| nomad acl role create --tls-skip-verify -name="client-introduction" \ | ||||||||||
| -policy="client-introduction" | ||||||||||
| ``` | ||||||||||
|
|
||||||||||
| Output is similar to the this: | ||||||||||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
|
||||||||||
|
|
||||||||||
| ```shell-session | ||||||||||
| ID = cf0b4a43-b00f-cc30-b656-b34d66151b04 | ||||||||||
| Name = client-introduction | ||||||||||
| Description = <none> | ||||||||||
| Policies = client-introduction | ||||||||||
| Create Index = 117 | ||||||||||
| Modify Index = 117 | ||||||||||
| ``` | ||||||||||
|
|
||||||||||
| ### Generate a Nomad token | ||||||||||
|
|
||||||||||
| Generate an ACL token from the `client-introduction` role. Nomad uses this | ||||||||||
| token to request client introduction tokens. In a production environment, we | ||||||||||
| recommend that you use Vault to generate and manage these tokens. | ||||||||||
|
|
||||||||||
| ```shell-session | ||||||||||
| nomad acl token create -name="client-intro-token-1" \ | ||||||||||
| -role-name="client-introduction" -type=client -ttl=8h | ||||||||||
| ``` | ||||||||||
|
|
||||||||||
| Output is similar to this: | ||||||||||
|
|
||||||||||
| ```shell-session | ||||||||||
| Accessor ID = 8c22a7c0-44f5-044d-ef84-bfa06118faf4 | ||||||||||
| Secret ID = d99d678d-426c-330e-74f3-de53a868e2f9 | ||||||||||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Highlighting the value of the |
||||||||||
| Name = client-intro-token-1 | ||||||||||
| Type = client | ||||||||||
| Global = false | ||||||||||
| Create Time = 2025-11-18 21:01:44.62075624 +0000 UTC | ||||||||||
| Expiry Time = 2025-11-19 05:01:44.62075624 +0000 UTC | ||||||||||
| Create Index = 128 | ||||||||||
| Modify Index = 128 | ||||||||||
| Policies = [] | ||||||||||
|
|
||||||||||
| Roles | ||||||||||
| ID Name | ||||||||||
| cf0b4a43-b00f-cc30-b656-b34d66151b04 client-introduction | ||||||||||
| ``` | ||||||||||
|
|
||||||||||
| ### Generate a client introduction token | ||||||||||
|
|
||||||||||
| The following snippets generate a basic token. | ||||||||||
|
|
||||||||||
| <Tabs> | ||||||||||
| <Tab heading="CLI"> | ||||||||||
|
|
||||||||||
| Use the [`nomad node intro create` command](/nomad/commands/node/intro/create) to generate the client introduction token. | ||||||||||
|
|
||||||||||
| ```shell-session | ||||||||||
| nomad node intro create > intro_token.jwt | ||||||||||
| ``` | ||||||||||
|
|
||||||||||
| The `intro-token.jwt` contents are similar to this. | ||||||||||
|
|
||||||||||
| ```shell-session | ||||||||||
| "eyJhbGciOiJSUzI1NiIsImtpZCI6IjljZDgy..." | ||||||||||
| ``` | ||||||||||
|
|
||||||||||
| </Tab> | ||||||||||
| <Tab heading="API"> | ||||||||||
|
|
||||||||||
| Use the [client introduction identity | ||||||||||
| endpoint](/nomad/api-docs/acl/identities#create-client-introduction-identity) to | ||||||||||
| generate the client introduction token. This example sends an empty JSON object, | ||||||||||
| but you may use a JSON payload file to specify parameters such as `NodeName`, | ||||||||||
| `NodePool`, and `TTL`. | ||||||||||
|
|
||||||||||
| ```shell-session | ||||||||||
| curl \ | ||||||||||
| --request POST \ | ||||||||||
| --header "X-Nomad-Token: d99d678d-426c-330e-74f3-de53a868e2f9" \ | ||||||||||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Here's the
But now I'm confused about the workflow. You create an ACL token so that a client node has permission to contact a server and change policies (which covers creating client introduction tokens). Then the node uses the client introduction token it creates to join the cluster it already has a valid ACL token from? Why the dedicated client token, then, when there's already an ACL token being used? |
||||||||||
| --data {} \ | ||||||||||
| https://localhost:4646/v1/acl/identity/client-introduction-token \ | ||||||||||
| >> intro_token.jwt | ||||||||||
| ``` | ||||||||||
|
|
||||||||||
| The response includes the JWT. | ||||||||||
|
|
||||||||||
| ```shell-session | ||||||||||
| { | ||||||||||
| "JWT": "eyJhbGciOiJSUzI1NiIsImtpZCI6IjljZDgy..." | ||||||||||
| } | ||||||||||
| ``` | ||||||||||
|
|
||||||||||
| </Tab> | ||||||||||
| </Tabs> | ||||||||||
|
|
||||||||||
| ### Configure the Nomad agent | ||||||||||
|
|
||||||||||
| Configure the Nomad agent's `server.client_introduction` block. This example | ||||||||||
| sets the `enforcement` parameter to `strict`, which means the agent rejects any | ||||||||||
| client without without a valid token. Refer to the [`server.client_introduction` | ||||||||||
| block | ||||||||||
| documentation](/nomad/docs/configuration/server#client_introduction-parameters) | ||||||||||
| for additional enforcement options. | ||||||||||
|
|
||||||||||
| <CodeBlockConfig highlight="8-12"> | ||||||||||
|
|
||||||||||
| ```hcl | ||||||||||
| data_dir = "/opt/nomad/" | ||||||||||
| acl { | ||||||||||
| enabled = true | ||||||||||
| } | ||||||||||
| server { | ||||||||||
| enabled = true | ||||||||||
| bootstrap_expect = 1 | ||||||||||
| client_introduction { | ||||||||||
| enforcement = "strict" # Default = "warn" | ||||||||||
| default_identity_ttl = "5m" # Default = "5m" | ||||||||||
| max_identity_ttl = "30m" # Default = "30m" | ||||||||||
| } | ||||||||||
| } | ||||||||||
| tls { | ||||||||||
| http = true | ||||||||||
| rpc = true | ||||||||||
| ca_file = "/opt/nomad/tls/nomad-agent-ca.pem" | ||||||||||
| cert_file = "/opt/nomad/tls/global-server-nomad.pem" | ||||||||||
| key_file = "/opt/nomad/tls/global-server-nomad-key.pem" | ||||||||||
| } | ||||||||||
| ``` | ||||||||||
|
|
||||||||||
| </CodeBlockConfig> | ||||||||||
|
|
||||||||||
| There is no additional client configuration. | ||||||||||
|
|
||||||||||
| ### Start the Nomad agent | ||||||||||
|
|
||||||||||
| We recommend setting the JWT token in an environment variable called | ||||||||||
| `NOMAD_CLIENT_INTRO_TOKEN`. The next option is to set it in the | ||||||||||
| [`-client-intro-token` parameter](/nomad/commands/agent#client-intro-token) of | ||||||||||
| the `nomad agent` command. Alternately, you may place the `intro_token.jwt` file | ||||||||||
| in the client's state directory, which is by default [the | ||||||||||
| `<data_dir>/client_state_dir>` | ||||||||||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Missing and/or extra bracket? |
||||||||||
| directory](/nomad/docs/configuration/client#state_dir). | ||||||||||
|
|
||||||||||
| This example starts the agent with the JWT token set in the | ||||||||||
| `-client-intro-token` parameter. | ||||||||||
|
|
||||||||||
| ```shell-session | ||||||||||
| nomad agent -config /etc/nomad.d/nomad.hcl \ | ||||||||||
| -client-intro-token "eyJhbGciOiJSUzI1NiIsImtpZCI6IjljZDgy..." | ||||||||||
| ``` | ||||||||||
|
|
||||||||||
| ### Monitor client join failures | ||||||||||
|
|
||||||||||
| You have the following options to determine when client registration fails: | ||||||||||
|
|
||||||||||
| - Check the agent logs for `[ERROR] nomad.client: node registration without | ||||||||||
| introduction token` messages. | ||||||||||
| - Monitor the [`nomad.client.introduction.enforcement` | ||||||||||
| counter](/nomad/docs/monitor#client-introduction), which increments when a | ||||||||||
| client tries to join without a valid client introduction token. | ||||||||||
|
|
||||||||||
|
|
||||||||||
|
|
||||||||||
| [`consul` stanza]: /nomad/docs/configuration/consul | ||||||||||
| [`node config` command]: /nomad/commands/node/config | ||||||||||
| [`retry_join`]: /nomad/docs/configuration/server_join#retry_join | ||||||||||
|
|
||||||||||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm trying to summarize the overall workflow, but finding it a bit convoluted.