Start Telegraf container with latest config from Influx Web UI [workaround]

[default-student]

I am trying to implement a TIG stack.
Telegraf should be configurable from the influx UI,
but the Telegraf container expects a config file, while the Influx UI instructs one to start telegraf with a call to the api that serves the config created by influx.

This makes it quite tedious to have both configs actually be the same.

There is no command i could start the telegraf container with which always gets the latest config created in the influx ui.

Below i will share my attempts to create a compose file for this.

[default-student]

My first attempt was to upload the config to git.
I used a setup container that downloads it and puts it into a shared volume.
The telegraf container mounts this volume and uses it as a config under /etc/telegraf/telegraf.conf
The influx container is started with a custom command

chmod +x /home/telegraf/create-or-update-influx-telegraf-config.sh
nohup /entrypoint.sh &
sleep 5
/home/telegraf/create-or-update-influx-telegraf-config.sh &
tail -f /dev/null

with

#!/bin/bash

# Check if any Telegraf configs exist
configs=$(influx telegrafs | tail -n +2)

if [ -z "$configs" ]; then
  echo No configs exist, create a new one
  influx telegrafs create --name 'Gitlab Telegraf Configuration' --token 'V2-token==' --org 'org' --description 'Configuration pulled from the Telegraf Volume which is pulled from the Gitlab' --file /home/telegraf/telegraf.conf &
else
  echo Update the first config
  config_id=$(echo "$configs" | head -n 1 | awk '{print $1}')
  influx telegrafs update --id "$config_id" --name 'Gitlab Telegraf Configuration' --description 'Configuration pulled from the Telegraf Volume which is pulled from the Gitlab' --file /home/telegraf/telegraf.conf
fi

which updates or creates the config from gitlab in the influx container.

This however doesnt mean that the config can now be edited in the influx web ui.

It does store the changes when I edit it, but it does not store it at the given file location but somewhere else.

This makes this unsuitable if one wants to update the config via the webui.

[default-student]

okay a personal fix, but nothing that changes the fundamental problem:

version: '3.9'

services:

  telegraf:
    image: telegraf
    container_name: influxdb-telegraf
    depends_on:
      setup:
        condition: service_completed_successfully

    entrypoint: ["/bin/bash", "-c", "/etc/telegraf/wait-for-id-then-start-with-config.sh"]

    environment:
      DOCKER_INFLUXDB_INIT_MODE: setup
      DOCKER_INFLUXDB_INIT_USERNAME: user
      DOCKER_INFLUXDB_INIT_PASSWORD: password
      DOCKER_INFLUXDB_INIT_ORG: org
      DOCKER_INFLUXDB_INIT_BUCKET: bucket
      DOCKER_INFLUXDB_INIT_ADMIN_TOKEN: V2-token==
      INFLUX_TOKEN: V2-token==
      DOCKER_INFLUXDB_INIT_CLI_CONFIG_NAME: configname2
    volumes:
      - telegraf:/etc/telegraf
      # - ./telegraf.conf:/etc/telegraf/telegraf.conf
    networks:
      - 'network'


  influxdb:
    image: influxdb
    container_name: influxdb2
    depends_on:
      setup:
        condition: service_completed_successfully

    command: >
      bash -c "
        nohup /entrypoint.sh &
        sleep 5
        chmod +x /home/telegraf/create-config-store-id-in-vol.sh
        /home/telegraf/create-config-store-id-in-vol.sh
        tail -f /dev/null
      "

    environment:
      DOCKER_INFLUXDB_INIT_MODE: setup
      DOCKER_INFLUXDB_INIT_USERNAME: user
      DOCKER_INFLUXDB_INIT_PASSWORD: password
      DOCKER_INFLUXDB_INIT_ORG: org
      DOCKER_INFLUXDB_INIT_BUCKET: bucket
      DOCKER_INFLUXDB_INIT_ADMIN_TOKEN: V2-token==
      DOCKER_INFLUXDB_INIT_CLI_CONFIG_NAME: configname2

    volumes:
      - influxdbv2:/var/lib/influxdb2:rw
      - influxconfig:/etc/influxdb2
      - telegraf:/home/telegraf
    ports:
      - "8086:8086"
    networks:
      - 'network'


# Service container
  # Purge any leftover files, clone repo, copy necessary files to the temporary volume
  setup:
    image: bitnami/git:latest
    container_name: setup
    command: >
      bash -c "
        echo clearing Volumes
          rm -rf /home/telegraf/*
          rm -rf /home/influxdb/* --force
          rm -rf /home/influxconfig/* --force
          
        echo emptied telegraf conf folder:
         ls /home/telegraf
        echo emptied influxdb database folder:
         ls /home/influxdb
        echo emptied influxdb database folder:
          ls /home/influxconfig

        echo Cloning git with config:
          git clone https://gitlab.de/my-repository/myrrepo.git
        echo downloaded:
          ls ./myrrepo
        echo copying git to telegraf config volume
          cp -R ./myrrepo/. /home/telegraf
        echo copied:
          ls /home/telegraf

        echo applying permissions:configs:
          chmod +x /home/telegraf/wait-for-id-then-start-with-config.sh
      "

      #   echo create data files in cleared folders:
      #     cd /home/telegraf
      #     touch '$$(date)'
      #     cd /home/influxdb
      #     touch '$$(date)'
      # cat /home/telegraf/telegraf.conf
      # tail -f /dev/null
      
    volumes:
      - telegraf:/home/telegraf
      - influxdbv2:/home/influxdb
      - influxconfig:/home/influxconfig

networks:
  network:
    driver: bridge  
    name: influxdb-telegraf-net

volumes:
  influxdbv2:
  influxconfig:
  telegraf:

with the create config and store id script

# check for config ids
configs=$(influx telegrafs | tail -n +2)

if [ -z "$configs" ]; then
  echo No configs exist, create a new one
  config_id=$(influx telegrafs create --name 'Gitlab Telegraf Configuration' --token 'V2-token==' --org 'org' --description 'Configuration pulled from the Telegraf Volume which is pulled from the Gitlab. It can be updated using the Influxdb WebUI and is pulled by the Telegraf container on restart. Please backup changes to the Github repo! Please restart the Telegraf container when changed.' --file /home/telegraf/telegraf.conf | awk '{if (NR == 2) print $1}')
  echo created config with id: '$config_id'
  touch /home/telegraf/telegraf.id
  echo "$config_id" >> /home/telegraf/telegraf.id
  echo stored id in '/home/telegraf/telegraf.id'
else
  echo config already exists just storing the id
  config_id=$(echo "$configs" | head -n 1 | awk '{print $1}')
  touch /home/telegraf/telegraf.id
  echo "$config_id" >> /home/telegraf/telegraf.id
  echo stored id: '$config_id' in '/home/telegraf/telegraf.id'
fi

and the script that waits for the config id and starts the telegraf agent.

file="/etc/telegraf/telegraf.id"

echo waiting for file: /etc/telegraf/telegraf.id
while [ ! -f "/etc/telegraf/telegraf.id" ]; 
do
	echo waiting for file: /etc/telegraf/telegraf.id
    sleep 1
done

id=$(cat "/etc/telegraf/telegraf.id")
echo running telegraf from the config with the id: '$id'
telegraf --config http://influxdb:8086/api/v2/telegrafs/$id

This docker compose works as a portainer stack, which requires there be no other files. Thus all files are pulled from a git repo by the service container. Then the influxdb container is initialized and the telegraf config pulled from the repo is specified as the telegraf config. The id of this config is stored in a shared volume file. the telegraf container waits for this file and uses the id to access the config stored in influx.

This enables us to modifiy the telegraf config in the influx web ui and after a restart of the telegraf container these changes are used.

I would greatly appreciate if there was a simpler way, but this is the solution for our organization for now.

[default-student]

Note the same issue on Stackoverflow
https://stackoverflow.com/questions/62282117/best-way-to-reload-telegraf-configuration-when-running-in-a-container