[FeatureRequest] docker container status tag

[kamalvpm777]

Feature Request

Opening a feature request kicks off a discussion.

Proposal:

How to apply Tag or Filter within telegraf.conf to retrieve the metrics for only running containers ignoring all the dead containers. In this case we have total 8 active running containers and 15 dead containers in the docker host. we have the telegraf.conf directly running on the docker host. So when getting the docker metrics through the telegraf instead of 8 as number of containers we are getting total of 23 containers which includes even dead containers too. Please guide on how to apply the filter/tag on telegraf.conf to pull only 8 running dockers. All our running containers have the STATUS of "UP 3 weeks" and dead containers are with STATUS Of "Exited". Not sure if we can use this a filter or tag to retrieve the running containers

######Sudo docker info######
[centos@ip-10-204-71-175 etc]$ sudo docker info
Containers: 23
Running: 8
Paused: 0
Stopped: 15

######Telegraf conf######
[[inputs.docker]]
'# Docker Endpoint
'#To use TCP, set endpoint = "tcp://[ip]:[port]"
'# To use environment variables (ie, docker-machine), set endpoint = "ENV"
endpoint = "unix:///var/run/docker.sock"
'# Only collect metrics for these containers, collect all if empty
container_names = []

Current behavior:

Desired behavior:

Number of Docker containers as "8" which is active running containers count

Use case: [Why is this important (helps with prioritizing requests)]

Currently the number of 23 docker containers are misleading though we have only 8 running containers when looking at grafana dashboard. This is very critical to setup to telegraf integrating with Grafana as part of production monitoring

[sparrc]

if I understand you correctly, you just want a tag that gives the docker container status? or a flag to only collect running containers?

[kamalvpm777]

Thanks for your reply, we are looking on how to apply the flag to retrieve only collect the running docker containers.

FYI, here is our workflow to monitor docker containers
We are integrating telegraf with cloud watch to send the docker metrics, finally pulling metrics from cloudwatch to grafana to plot the graph within grafana.

[ssorathia]

I actually didn't realize that it was doing that, but seeing as this would probably affect us as well, I could probably write up a quick PR for this since I'm already familiar with the code base in question.

Here is what I'm thinking:

1. Add a status tag based on the status of the container. Should this status only be running or stopped(exited), what about the other statuses? How should they come back? How would you want the following to be mapped:
   created
   restarting
   running
   paused
   exited
   dead

2. Add a flag to only include running containers.

Does this make sense?

[kamalvpm777]

yup this makes ton of sense.

Please keep posted once it is ready to use. It would be great if someone can share some sample grafana query with the flag status for the datasource as Cloud watch

[sparrc]

@ssorathia you might want to wait until #2432 is merged before working on this

[cjpit]

Hi there, any plans on get this included? The docker plugin is missing this important feature to really monitor the state of any of the containers, which is one of the main reasons for using it in my instance.
I noticed #3493 was just added, which seems like it will add those features? Assuming @ssorathia there wasn't anything done about this request specifically?

[impala454]

This is old but I'm in the same boat (or just missing how to use the feature possibly). I would like the docker_container_status metric to include a run_status string, just as the docker_health_status includes a health_status string. I don't see any metrics in the docker plugin that provide the docker run status of a given container. E.g. the statues @ssorathia mentions above. Seems like this would be a very basic necessity? It's the only thing keeping me from using this over cadvisor.

[danielnelson]

It's been a while since I looked at this one, but it seems like this issue was resolved some time ago with the container_status tag:

docker_container_status,container_image=telegraf,container_name=hungry_keldysh,container_status=running,container_version=unknown,engine_host=debian-stretch-docker.virt,host=loaner,server_version=19.03.6 container_id="3802ffedc22a8b9c8cd279ac7fef3d1f49657893f9c4f0166fdb79161da4f97e",exitcode=0i,oomkilled=false,pid=32236i,started_at=1588181197746303441i,uptime_ns=8399991391i 1588181206000000000

It may also be useful to control which containers are collected with the container_state_include and container_state_exclude options:

  ## Container states to include and exclude. Globs accepted.
  ## When empty only containers in the "running" state will be captured.
  ## example: container_state_include = ["created", "restarting", "running", "removing", "paused", "exited", "dead"]
  ## example: container_state_exclude = ["created", "restarting", "running", "removing", "paused", "exited", "dead"]
  # container_state_include = []
  # container_state_exclude = []

[danielnelson]

Closed in #4259