Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -8,8 +8,7 @@ description: Nomad's node identity feature uniquely identities each Nomad client

This page provides conceptual information about Nomad's node identity feature,
which uniquely identities each Nomad client node and provides an authentication
mechanism for nodes to make RPC calls to the Nomad servers. This feature does
not replace mTLS.
mechanism for nodes to make RPC calls to the Nomad servers.

The Nomad cluster gives every node a default identity once the cluster is able
to fully support the feature with a defined lifetime. This node identity is a
Expand Down Expand Up @@ -48,7 +47,7 @@ lifetime check is bypassed.

A node's identity is stored within its local database, which is persisted within the
configured [`data_dir`][client_cfg_state_dir]. To view the stored identity,
along with other state resouces, run the
along with other state resources, run the
[`operator client-state` command][cli_operattor_client_state].

[JSON Web Token (JWT)]: https://datatracker.ietf.org/doc/html/rfc7519
Expand Down
226 changes: 222 additions & 4 deletions content/nomad/v1.11.x/content/docs/deploy/clusters/connect-nodes.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -11,13 +11,16 @@ description: |-
In order to create a Nomad cluster out of individual nodes, you need to
introduce them to one another. There are several ways to perform this:

- Manually
- Manual bootstrap
- Cloud Auto-Join
- Consul

This tutorial describes each method and provides configuration snippets, which
This guide describes each method and provides configuration snippets, which
you can use as starting points for your own configuration.

You may also use client node introduction tokens to restrict which clients join
your cluster.

## Manual clustering

Manually bootstrapping a Nomad cluster does not rely on additional tooling, but
Expand Down Expand Up @@ -82,9 +85,9 @@ the client. This means only one server must be specified because, after initial
contact, the full set of servers in the client's region are shared with the
client.

## Join nodes using cloud auto-join
## Use cloud auto-join

As of Nomad 0.8.4, [`retry_join`] accepts a unified interface using the
The [`retry_join`] parameter accepts a unified interface using the
[go-discover] library for doing automatic cluster joining using cloud metadata.
To use retry-join with a supported cloud provider, specify the configuration on
the command line or configuration file as a `key=value key=value ...` string.
Expand Down Expand Up @@ -214,6 +217,221 @@ consul {
Refer to the [`consul` stanza] documentation for the complete set of configuration
options.


## Use client node introduction tokens

You may restrict which clients join your cluster by configuring client
introduction tokens. The client node introduction feature is like multi-factor
authentication for your Nomad clusters. It does not replace mTLS but adds a
second layer of security to prevent an unauthorized or misconfigured client from
joining a Nomad cluster.

Each layer answers a distinct question:

- mTLS: Does the client have valid certificates for the cluster?
- Client introduction token: Does the client have a valid token to join the
cluster?

You do not need to configure mTLS to use client node introduction tokens, but we
do recommend securing your cluster with mTLS.

You may optionally specify node names, node pools, and TTLs when you generate
client introduction tokens.

Follow these steps to use client node introduction tokens:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
Follow these steps to use client node introduction tokens:
The workflow for client introduction tokens uses two tokens: an _ACL token_ and a _client introduction token_. The ACL token allows the node to create a client introduction token, which you apply to the Nomad client when you start it.

I'm trying to summarize the overall workflow, but finding it a bit convoluted.


1. [Create an ACL policy in which the node has write permissions](#create-an-acl-policy).
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
1. [Create an ACL policy in which the node has write permissions](#create-an-acl-policy).
1. [Create an ACL policy for a node with write permissions](#create-an-acl-policy).

Simplifying this

1. [Create an ACL role from the policy](#create-an-acl-role).
1. [Generate a Nomad token](#generate-a-nomad-token).
1. [Generate a client introduction token](#generate-a-client-introduction-token).
1. [Configure the Nomad agent to use client introduction](#configure-the-nomad-agent).
1. [Start the Nomad agent](#start-the-nomad-agent).
1. [Monitor client join failures](#monitor-client-join-failures).

### Create an ACL policy

This example creates the `node:write` policy required to generate tokens.

1. Create a policy file called `client-introduction.hcl`.

```hcl
node {
policy = "write"
}
```

1. Create the policy.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
1. Create the policy.
1. Add the policy to the Nomad cluster.

To avoid repetition with the step before this one and clarify what is happening.


```shell-session
nomad acl policy apply client-introduction client-introduction.hcl
```

### Create an ACL role

This example creates an ACL role called `client-introduction`.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
This example creates an ACL role called `client-introduction`.
Create an ACL role, and attach the policy you created to the role.
This example creates an ACL role with the name `client-introduction` that uses the policy `client-introduction`.
  1. Instructions to begin the section, then the example.
  2. Trying to point out both CLI flags because the example uses the same name but that isn't a requirement.


```shell-session
nomad acl role create --tls-skip-verify -name="client-introduction" \
-policy="client-introduction"
```

Output is similar to the this:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
Output is similar to the this:
The output describes the ACL role, including its attached policies and the Raft index at its creation.


```shell-session
ID = cf0b4a43-b00f-cc30-b656-b34d66151b04
Name = client-introduction
Description = <none>
Policies = client-introduction
Create Index = 117
Modify Index = 117
```

### Generate a Nomad token

Generate an ACL token from the `client-introduction` role. Nomad uses this
token to request client introduction tokens. In a production environment, we
recommend that you use Vault to generate and manage these tokens.

```shell-session
nomad acl token create -name="client-intro-token-1" \
-role-name="client-introduction" -type=client -ttl=8h
```

Output is similar to this:

```shell-session
Accessor ID = 8c22a7c0-44f5-044d-ef84-bfa06118faf4
Secret ID = d99d678d-426c-330e-74f3-de53a868e2f9
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Highlighting the value of the client-intro-token-1 ACL token, which all instructions up to this point have led to creating.

Name = client-intro-token-1
Type = client
Global = false
Create Time = 2025-11-18 21:01:44.62075624 +0000 UTC
Expiry Time = 2025-11-19 05:01:44.62075624 +0000 UTC
Create Index = 128
Modify Index = 128
Policies = []

Roles
ID Name
cf0b4a43-b00f-cc30-b656-b34d66151b04 client-introduction
```

### Generate a client introduction token

The following snippets generate a basic token.

<Tabs>
<Tab heading="CLI">

Use the [`nomad node intro create` command](/nomad/commands/node/intro/create) to generate the client introduction token.

```shell-session
nomad node intro create > intro_token.jwt
```

The `intro-token.jwt` contents are similar to this.

```shell-session
"eyJhbGciOiJSUzI1NiIsImtpZCI6IjljZDgy..."
```

</Tab>
<Tab heading="API">

Use the [client introduction identity
endpoint](/nomad/api-docs/acl/identities#create-client-introduction-identity) to
generate the client introduction token. This example sends an empty JSON object,
but you may use a JSON payload file to specify parameters such as `NodeName`,
`NodePool`, and `TTL`.

```shell-session
curl \
--request POST \
--header "X-Nomad-Token: d99d678d-426c-330e-74f3-de53a868e2f9" \
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here's the client-intro-token-1 ACL token value. So now I have a few questions.

  1. Should the CLI instructions have included a step to use the ACL token as well?
  2. Are you supposed to create a separate ACL token for each node, and that's why it's token-1?
  3. Or should you be creating a separate intro token for each client, using the one shared ACL token?

But now I'm confused about the workflow. You create an ACL token so that a client node has permission to contact a server and change policies (which covers creating client introduction tokens). Then the node uses the client introduction token it creates to join the cluster it already has a valid ACL token from? Why the dedicated client token, then, when there's already an ACL token being used?

--data {} \
https://localhost:4646/v1/acl/identity/client-introduction-token \
>> intro_token.jwt
```

The response includes the JWT.

```shell-session
{
"JWT": "eyJhbGciOiJSUzI1NiIsImtpZCI6IjljZDgy..."
}
```

</Tab>
</Tabs>

### Configure the Nomad agent

Configure the Nomad agent's `server.client_introduction` block. This example
sets the `enforcement` parameter to `strict`, which means the agent rejects any
client without without a valid token. Refer to the [`server.client_introduction`
block
documentation](/nomad/docs/configuration/server#client_introduction-parameters)
for additional enforcement options.

<CodeBlockConfig highlight="8-12">

```hcl
data_dir = "/opt/nomad/"
acl {
enabled = true
}
server {
enabled = true
bootstrap_expect = 1
client_introduction {
enforcement = "strict" # Default = "warn"
default_identity_ttl = "5m" # Default = "5m"
max_identity_ttl = "30m" # Default = "30m"
}
}
tls {
http = true
rpc = true
ca_file = "/opt/nomad/tls/nomad-agent-ca.pem"
cert_file = "/opt/nomad/tls/global-server-nomad.pem"
key_file = "/opt/nomad/tls/global-server-nomad-key.pem"
}
```

</CodeBlockConfig>

There is no additional client configuration.

### Start the Nomad agent

We recommend setting the JWT token in an environment variable called
`NOMAD_CLIENT_INTRO_TOKEN`. The next option is to set it in the
[`-client-intro-token` parameter](/nomad/commands/agent#client-intro-token) of
the `nomad agent` command. Alternately, you may place the `intro_token.jwt` file
in the client's state directory, which is by default [the
`<data_dir>/client_state_dir>`
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Missing and/or extra bracket?

directory](/nomad/docs/configuration/client#state_dir).

This example starts the agent with the JWT token set in the
`-client-intro-token` parameter.

```shell-session
nomad agent -config /etc/nomad.d/nomad.hcl \
-client-intro-token "eyJhbGciOiJSUzI1NiIsImtpZCI6IjljZDgy..."
```

### Monitor client join failures

You have the following options to determine when client registration fails:

- Check the agent logs for `[ERROR] nomad.client: node registration without
introduction token` messages.
- Monitor the [`nomad.client.introduction.enforcement`
counter](/nomad/docs/monitor#client-introduction), which increments when a
client tries to join without a valid client introduction token.



[`consul` stanza]: /nomad/docs/configuration/consul
[`node config` command]: /nomad/commands/node/config
[`retry_join`]: /nomad/docs/configuration/server_join#retry_join
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -15,19 +15,28 @@ We are pleased to announce the following Nomad updates.

Nomad's client node identity feature uniquely identities each Nomad client node
and provides an authentication mechanism for nodes to make RPC calls to the
Nomad servers. This feature does not replace mTLS.
Nomad servers.

Introduce Nomad clients to the cluster with JWT tokens. Configure Nomad servers
with introduction enforcement levels that dictate how clients join the cluster.
This approach results in logs and metrics to detail introduction violations.
Once registered, Nomad clients are now provided with an identity token, used for
RPC communication which is periodically renewed.

The client node introduction and identity feature functions as multi-factor
authentication for your Nomad clusters. It does not replace mTLS. Instead, it
adds a second layer of security to prevent an unauthorized client from joining a
Nomad cluster.

Using a client introduction token gives you additional control over misconfigured clients trying to join the Nomad cluster because you can specify node names, node pools, and TTLs for the tokens you generate.

#### Relevant documentation

- [Client node identity concepts](/nomad/docs/architecture/cluster/node-identity)
- [Client identity node pool TTL configuration option](/nomad/docs/other-specifications/node-pool#node_identity_ttl)
- [Client Introduction server configuration options](/nomad/docs/configuration/server#client_introduction-parameters)
- [Client identity node pool TTL configuration
option](/nomad/docs/other-specifications/node-pool#node_identity_ttl)
- [Use client node introduction tokens to connect clients to your Nomad agent](/nomad/docs/deploy/clusters/connect-nodes#use-client-node-introduction-tokens)
- [Client introduction server configuration options](/nomad/docs/configuration/server#client_introduction-parameters)
- [Client introduction monitoring detail](/nomad/docs/monitor#client-introduction)
- [Client introduction agent CLI token flag](/nomad/commands/agent#client-intro-token)

Expand Down
Loading