Templating to shared alloc/ directory

[valenvb]

Nomad version

Nomad v1.0.0 (cfca6405ad9b5f66dffc8843e3d16f92f3bedb43)

Issue

This seems to be a bug specifically with the docker driver.

I need to template a file in the group shared alloc/data/ directory, such that all tasks in the group can read it.
The docs for template indicate that destination is relative to the task working directory, and links to this page. This implies that I should be able to set the destination to alloc/data/file. However this results in an alloc/ directory created next to the tasks local/ directory, inside the task-only working directory.

ie: /var/nomad/alloc/<alloc_id> looks like

alloc/
<task1>/
 - alloc/data/file
- local/
- secrets/
<task2>/
- local/
- secrets/

when what I would expect to see is

alloc/
- data/file
<task1>/
- local/
- secrets/
<task2>/
- local/
- secrets/

in Nomad 0.12.4 I was able to circumvent this by using a destination of ../alloc/data/file, however this is considered an alloc directory escape now, resulting in the error: destination path escapes the alloc directory

Reproduction steps

Run a job file as below

Job file (if appropriate)

job "repro" {
  datacenters = ["dc1"]
  type = "batch"
  namespace = "eng-infra"

  group "repro" {

    task "a" {
      driver = "docker"
      config {
        image = "alpine:3"
        command = "cat"
        args = ["${NOMAD_ALLOC_DIR}/shared.txt"]
      }

      template {
        destination = "${NOMAD_ALLOC_DIR}/shared.txt"
        data = <<EOH
Hello, world!
EOH
      }
    }

    task "b" {
      driver = "docker"
      config {
        image = "alpine:3"
        command = "cat"
        args = ["${NOMAD_ALLOC_DIR}/shared.txt"]
      }
    }
  }
}

[cgbaker]

There's almost certainly opportunity for better docs here; this often results in a lot of trial and error to figure out these relative paths. However, I did have success with the following approach, which relies strictly on interpolation:

job "repro" {
  datacenters = ["dc1"]
  type = "batch"

  group "repro" {

    task "a" {
      driver = "exec"
      config {
        command = "cat"
        args = ["${NOMAD_ALLOC_DIR}/shared.txt"]
      }

      template {
        destination = "${NOMAD_ALLOC_DIR}/shared.txt"
        data = <<EOH
Hello, world!
EOH
      }
    }

    task "b" {
      driver = "exec"
      config {
        command = "cat"
        args = ["${NOMAD_ALLOC_DIR}/shared.txt"]
      }
    }
  }
}

[valenvb]

Hah, interesting... it does work fine with the exec driver, however it fails with the docker driver.
I'll update my sample failing job to be clearer on this

job "repro" {
  datacenters = ["dc1"]
  type = "batch"
  namespace = "eng-infra"

  group "repro" {

    task "a" {
      driver = "docker"
      config {
        image = "alpine:3"
        command = "cat"
        args = ["${NOMAD_ALLOC_DIR}/shared.txt"]
      }

      template {
        destination = "${NOMAD_ALLOC_DIR}/shared.txt"
        data = <<EOH
Hello, world!
EOH
      }
    }

    task "b" {
      driver = "docker"
      config {
        image = "alpine:3"
        command = "cat"
        args = ["${NOMAD_ALLOC_DIR}/shared.txt"]
      }
    }
  }
}

[cgbaker]

@valenvb , I've marked this as a bug, because it really seems like this behavior is subverting the intended workflow.

I haven't finished investigating, but it looks like this works for exec-based drivers because the client links the task alloc dir and the shared alloc dir before rendering the template. Therefore, when the template is rendered relative to the task working dir, files are created the shared dir. Unfortunately, for docker, this linking isn't done, so the template is rendered into the the task working dir; in the job spec above, this location is then overridden by the mounted /alloc dir, so none of the tasks have access to the rendered template!!!

We'll need to come up with some change so that we can honor this use case. In the short term, a workaround may be to render the file to the alloc in a prehook task and copy it to the shared alloc dir. This has the benefit of making explicit that the file is created and available before the "main" tasks start. Here is an example:

job "workaround" {
  datacenters = ["dc1"]
  type = "batch"

  group "repro" {
    task "init" {
      lifecycle {
        hook = "prestart"
      }

      template {
        destination = "local/shared.txt"
        data = <<EOH
Hello, world!
EOH
      }

      driver = "docker"
      config {
        image = "alpine:3"
        command = "cp"
        args = ["${NOMAD_TASK_DIR}/shared.txt", "${NOMAD_ALLOC_DIR}/shared.txt"]
      }
    }

    task "a" {
      driver = "docker"
      config {
        image = "alpine:3"
        command = "cat"
        args = ["${NOMAD_ALLOC_DIR}/shared.txt"]
      }

    }

    task "b" {
      driver = "docker"
      config {
        image = "alpine:3"
        command = "cat"
        args = ["${NOMAD_ALLOC_DIR}/shared.txt"]
      }
    }
  }
}

[valenvb]

Thanks for looking into this @cgbaker. I'll see if I can make the prestart job work in my use case - unfortunately the first job that needs the templated file is a prestart job already.. So far it seems like the tasks are run in the order defined by the jobfile, but I haven't seen that made explicit anywhere

[github-actions]

I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.