new feature: add nomad provider/resource (to run jobs thru nomad)

[ketzacoatl]

I have spent some time thinking about and playing with Nomad for task scheduling, and while I have had a lot of successes here, the one place I am still turning over is how to automate initial scheduling of jobs when creating a new deployment of my environment with Terraform. My first thoughts were attempting to string consul and consul-template together in some fashion, then I started to think this initial scheduling was more fit to run from Terraform with the local_exec provisioner.. eventually I realized: a Nomad resource for Terraform would be the most elegant way to do so. Maybe you would agree, or have even considered this.

I am not familiar enough with Terraform internals, but I would imagine the user provides a Nomad job definition as either inline or a file reference, points to the nomad server address, and whatever TLS/secrets/etc are needed to connect to Nomad's APIs.

[maguec]

+1

[grahamgreen]

+1

[sepulworld]

+1

[barryw]

+1

[apparentlymart]

@ketzacoatl please forgive me that I'm not super-familiar with Nomad yet, but I wonder what your rationale is for this being a provisioner rather than a resource.

For example,

resource "nomad_job" "foo" {
  region = "global"
  id = "example"
  name = "example"
  datacenters = ["dc1"]
  # etc, etc
}

Jobs seem like a natural fit for Terraform resources because they have POST, GET, PUT and DELETE operations that map well onto Terraform's resource lifecycle. Provisioners are more "one-shot", with Terraform immediately losing track of anything created by them.

[ketzacoatl]

@apparentlymart, you are so very right, I have updated the issue title and description.

[nicolai86]

I started playing around with this idea. It looks like mapping the nomad HCL configuration 1:1 to terraform DSL might be tricky, but something along these lines would work just fine:

provider "nomad" {
  address = "http://127.0.0.1:4646"
  region = "global"
}

resource "template_file" "redis-job" {
  # the redis.nomad from the nomad repo
  template = "${file("./redis.nomad")}"
}

resource "nomad_job" "redis" {
  job_definition = "${template_file.redis-job.rendered}"
}

# use `${nomad_job.redis.allocations}`

do you think this approach would be accepted upstream? WIP prototype located here

[ketzacoatl]

A common use case will be "deploy nomad cluster, setup nomad provider pointed at that new cluster but going over SSH, run some jobs" - is this feasible with local_exec and your implementation?

[nicolai86]

@ketzacoatl the SSH part won't work atm because I assumed the http api will be exposed & accessible. I'll do some digging to see if terraform allows to add connection settings as provisioners have on providers.

If you only plan to run some jobs on a newly created nomad cluster the null_resource should fit your needs perfectly, assuming you don't want to track the created jobs in any way: https://www.terraform.io/docs/provisioners/null_resource.html

[nicolai86]

That being said I expose the http auth settings already, so the nomad job definition can be managed on a secured cluster

[apparentlymart]

@nicolai86 Terraform doesn't currently have a solution for automatically creating tunnels to expose HTTP API's like Nomad's via an SSH bastion connection, assuming that this is what @ketzacoatl and you are discussing here.

I would suggest not trying to solve this locally within this resource, since this is a problem common to basically any provider that interacts with internal services (Consul, MySQL, PostgreSQL, Rundeck, Grafana, ...) and so in the short term there are general workarounds for this (e.g. running Terraform on a machine within the private network rather than from outside it, using an SSH-tunnelling SOCKS proxy set up outside of Terraform, or using a VPN rather than SSH) and in the long run I expect we'll try to solve this properly for all of these providers at once.

[ketzacoatl]

Thanks for clarifying @apparentlymart.

[nicolai86]

@apparentlymart thanks for the heads up. I think adding this on a provider level makes sense.

Are you actively working on something like this or are PRs for this welcome?

[apparentlymart]

@nicolai86 Assuming you're talking about the SSH tunnel thing, I think I remember @phinze saying he had some thoughts/ideas for ways to get Terraform to open SSH tunnels and route connections via them, but I don't remember the details. Perhaps he can remember what I'm talking about here. 😀

If not, and you'd like to work on this, my suggestion would be to open a proposal issue describing your idea for how to solve the problem, so the Terraform team can weigh in on the idea and how well it fits in with other stuff on the roadmap, rather than heading straight to a PR.

---

I don't think any of these should block starting a Nomad provider, though. Just build it under the assumption that Terraform is able to directly connect to the Nomad HTTP API, and then we can solve separately for helpers within Terraform to enable arbitrary TCP connections over SSH tunnels.

[ketzacoatl]

I am super excited to see @nicolai86's WIP PR! I completely understand the need to shelve the WIP for other things in life, but I am also curious what is needed to push that PR over the hill so it can be included in the next TF release. @nicolai86, or @apparentlymart (or anyone else for that matter), is it possible to get a list of what is needed in that PR to bring it up to par?

[nicolai86]

@ketzacoatl there's already a new PR from @mitchellh : #9538. So no worries :)

[mitchellh]

Done, will be in 0.8, the beta is starting tmrw.

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.