Add support for dynamic Datadog timeboards

[assiotis]

I would like to create a few modules so backend engineers can easily create dashboards for services they own. As discussed in #47, graphs are inline objects, so they don't support things like count or conditionals. One solution is to provide graphs as a "companion resource", similar to how aws_security_group and aws_security_group_rule work together. Is that possible? Has it been explored before?

I don't know golang and never written a TF plugin, but I think I want this bad enough to learn if someone from the team can give some guidance on whether it is even possible here and what needs to change.

Terraform Version

Terraform v0.10.8
Provider v1.0.3

Affected Resource(s)

Please list the resources as a list, for example:

• datadog_timeboard

Terraform Configuration Files

My goal is to end up being able to do something like this:

module "myservice" {
  source                 = "modules/service-timeboard"
  name                   = "myservice"
  environment            = "${var.environment}"
  include_system_metrics = true
  include_ebs_metrics    = true
  egress_services        = ["some_other_service1", "some_other_service_2"]
  mongo_clusters         = ["some_mongo_cluster_name"]
  redis_clusters         = []
  rds_instances          = ["${aws_db_instance.myservicerds.id}"]
}

To do that, the provider needs to support something like this

resource "datadog_timeboard" "this" {
  title       = "${title(var.name)} Timeboard [${upper(var.environment)}]"
  description = "A timeboard generated automatically by Terraform for the ${title(var.name)} service"
  read_only   = true
}

resource "datadog_timeboard_graph" "cpu" {
  count     = "${var.include_system_metrics ? 1 : 0}"
  timeboard = "${datadog_timeboard.this.id}"
  title     = "Hostmap (CPU)"
  viz       = "hostmap"

  request {
    q    = "avg:system.load.norm.5{...stuff...} by {host}"
    type = "fill"
  }

  scope = ["environment:${var.environment}", "service:${var.name}"]
  group = ["availability-zone"]

  style {
    palette = "green_to_orange"
  }

  include_no_metric_hosts = true
}

resource "datadog_timeboard_graph" "other_stuff" {
  timeboard = "${datadog_timeboard.this.id}"
  ...
}

[wendorf]

I spent some time investigating this, and the big difference between Datadog timeboards and AWS security groups is that there is an API for adding a single security group rule to a security group (authorize-security-group-egress and authorize-security-group-ingress), but the only API for timeboards requires setting all graphs.

It's still possible to do, but would probably require locking the dashboard with a mutex for each graph, pulling the existing config, merging in the graph config, and setting the entire dashboard. I think having a datadog_timeboard_graph resource would be very useful, and I wouldn't mind taking on the work to add it, but I'd like to get some validation from a codeowner here that this approach is reasonable.

[snemetz]

Why make the graph a resource (which it is not)? Just make it a data item. Then a timeboard resource can be composed from a number them.

Then in Terraform it would be more like:

data "datadog_timeboard_graph" "cpu" {
  count = ""
  ...
}

resource "datadog_timeboard" "this" {
  title       = "${title(var.name)} Timeboard [${upper(var.environment)}]"
  description = "A timeboard generated automatically by Terraform for the ${title(var.name)} service"
  read_only   = true
  graphs = ["${data.datadog_timeboard_graph.cpu}"]
}

Or it you want to treat it like a resource in Terraform, then you need to be able to build the full timeboard data structure before calling the API. Not sure is dependency control will be enough to accomplish this

[wendorf]

I like the data approach, because you're right, it's not a resource. The missing piece for me is that I'd like to be able to have a dynamic number of graphs without needing to specify them all in the timeboard. Something like this:

simple_app module:

data "datadog_timeboard_graph" "cpu" {
  title = "${var.name} CPU"
  viz   = "timeseries"
  timeboard = "${var.timeboard_id}"

  request {
    q       = "avg:docker.cpu.user{${var.label}:${var.value}}"
  }
}

data "datadog_timeboard_graph" "memory" {
  title = "${var.name} memory"
  viz   = "timeseries"
  timeboard = "${var.timeboard_id}"

  request {
    q       = "avg:docker.mem.rss{${var.label}:${var.value}}"
  }
}

base config:

resource "datadog_timeboard" "my-apps" {
  title       = "My Apps"
  description = "Metrics for all of my apps"
}

module "simple_app" {
  name = "Frontend"
  label = "app_name"
  value = "frontend_xyz"
  timeboard_id = "${datadog_timeboard.my-apps.id}"
}

module "simple_app" {
  name = "Backend"
  label = "app_name"
  value = "backend_xyz"
  timeboard_id = "${datadog_timeboard.my-apps.id}"
}

Then I could abstract common patterns more easily, and do things like add another graph for each app without needing to enumerate the list of new graphs.

[snemetz]

Well if they are data, they can't add themselves to a resource. The resource needs to reference them. But with a little work (and not pretty code) you could specify all of them in the timeboard resource and then use count to enable/disable each one in the data

[chi-yang]

Ran into the same thing. Could someone from team give some guidance or will this on future release?

[maartenvanderhoef]

I think I fixed this using locals, feel free to contribute to this module, I'm not a heavy Datadog user myself.

https://registry.terraform.io/modules/maartenvanderhoef/timeboard/datadog/0.0.4

https://github.com/maartenvanderhoef/terraform-datadog-timeboard/blob/master/examples/main.tf

[mtrienis]

@maartenvanderhoef I don't think locals works since you can't loop using the count option.

[iancward]

@assiotis have you looked at template variables? They should save you from having to create multiple identical dashboards and instead let you switch the data set (e.g. environment, cluster, etc) by using a drop-down.
https://docs.datadoghq.com/graphing/dashboards/template_variables/

[iksaif]

I do not think templates variables are a good alternative for that. The ability to dynamically create graphs would be really great to have.

Maybe dynamic nested blocks would be enough here ?

[assiotis]

I ended concocting an embarrassing hack, that so far seems to work OK. Using Terraform 0.10.x.

First setup the various components you want to dynamically switch in and out

locals {
  graphs_basics = [
  {
    title = "Hosts"
    viz   = "hostmap"
     // bla bla
   }, ... { 
      // more graphs 
   }]
}
locals {
  graphs_nginx = [
  { // more graphs
  }]
}
locals {
  graphs_business = [
  { // more graphs
  }]
}

Then in the main code for the module we construct the list of graphs step by step. At each step, we either append the next set of graphs or copy the previous step as-is, all depending if the component is enabled or disabled:

locals {
  step0 = "${local.graphs_basics}"

  step1 = "${slice(concat(local.step0, local.graphs_nginx),
          0,
          var.enable_nginx_graphs ? length(local.step0) + length(local.graphs_nginx) : length(local.step0))}"

  step2 = "${slice(concat(local.step1, local.graphs_business),
          0,
          var.enable_business_metric_graphs ? length(local.step1) + length(local.graphs_business) : length(local.step1))}"
   
  // more of the same
  step20 = // concat with the previous variable and slice differently depending if the component is enabled or not
  stepN = "${slice(concat(local.step20, local.graphs_blablabla),
           0,
           var.enable_blabla_component ? length(local.step20) + length(local.graphs_blabla) : length(local.step20))}"

  final = "${concat(local.stepN, var.extra_graphs)}"
}

// now construct timeboard
resource "datadog_timeboard" "this" {
  title       = "${upper(replace(var.name, "-_", " "))} Timeboard [TF]"
  description = "A timeboard generated automatically by Terraform for the ${title(var.name)} service"
  read_only   = false
  graph       = "${local.final}"

  // can setup template variables as well or whatever else is needed
  template_variable { }  
}

[bkabrda]

Seems like this could be solved using the dynamic blocks feature that's been newly introduced in TF 0.12 [1]. I'll create a PR for our docs to present example usage.

[1] https://www.terraform.io/docs/configuration/expressions.html#dynamic-blocks

[bkabrda]

PR #232 was merged, closing this issue. Thanks everyone for your participation!