retemplate

A module to execute a Jinja template on a schedule, supporting multiple backends for value storage.

Currently supported backends:

• AWS Local Metadata Server
• AWS Secrets Manager Plaintext Secrets
• AWS Systems Manager Parameters
• Local Command Execution
• Redis

The included config.yml.example file provides a sample configuration for Retemplate.

This code almost certainly does not work on non-Unix systems, as it relies on Unix-style permissions and file ownership to operate. I have no plans to make this work on Windows.

Installing It

pip install retemplate

Running It

Place a config.yml file in the cwd, or specify a config file with -c.

rtpl -c /etc/retemplate.yml

rtpl also supports the -l or --logfile option, which sets the initial log file for the software. A very small number of startup messages are logged before the log configuration is processed. On Python versions prior to 3.8, the basicConfig function does not accept the force option, which allows rtpl to reconfigure an existing logger. So if you run Python 3.8 or later, you can just configure logging in the config file, and you'll only lose a couple mostly insignificant messages. On older Pythons, all logging will be lost if you don't establish the filename at launch time. Older Pythons also will not allow custom log formats, etc. It is strongly recommended that you run Retemplate on Python 3.8 or later.

Use Case

Let's say you have an instance of PyHiAPI running out of /opt/hiapi and it is kept alive by a supervisord configuration. Furthermore, let's say the message returned by HiAPI should match the value stored in a Redis server in a key called hiapi.message. So you feed -c /opt/hiapi/config.txt to hiapi in your supervisord config so it cares about the content of that file. Then you configure Retemplate to generate that file based on a template. You might create a template at /etc/retemplate/hiapi.config.j2 that looks like this:

rtpl://local-redis/hiapi_message

Then create a config file for retemplate that contains these elements:

stores:
  local-redis:
    type: redis
    host: localhost
    port: 6379
    db: 0
    ssl: False
templates:
  /opt/hiapi/config.txt:
    template: /etc/retemplate/hiapi.config.j2
    owner: root
    group: root
    chmod: "0600"
    frequency: 60
    onchange: supervisorctl restart hiapi

This would lead to the following behavior:

• Every 60 seconds, retemplate will attempt to parse the template at /etc/retemplate/hiapi.config.j2.
• The newly generated template will be compared to the existing one at /opt/hiapi/config.txt.
  • If the generated file differs from what is already there, that file will be replaced with the new version. Its ownership and permissions settings will be updated (root:root, 0600). The command supervisorctl restart hiapi will be executed.
  • If the two files are the same, retemplate will do nothing.

Configuration

Retemplate is configured with a YAML file consisting of three main sections:

• retemplate (global settings)
• stores (data store configuration)
• templates (which files get worked over)

Global Settings

Global settings come under the retemplate section.

logging

In the logging section, you can supply options to pass into the Python logger library's basicConfig function. config.yml.example shows a few simple options.

include

This is a list of other YAML config files to include. Regardless of where in your config file this option appears, Retemplate will always interpret them in the order listed, after completing the interpretation of the config file the directive is found in. Options found in those config files will override conflicting options discovered beforehand. In other words, configs read by include directives will take precedence.

You can use regular expressions to indicate inclusion of all files matching the string. For example, to include all YAML files in a directory, you could use /etc/retemplate/conf.d/*.yml. Any value supported by Python's glob.glob function is valid here.

Data Stores

The stores section defines your data stores, which are services or functions that retrieve data for templating purposes. There are currently five types of data stores, each with their own configuration options. In the configuration file, these are defined by a dictionary entry where the key is the name of the data store as it will be referenced later in templates and the value is a dictionary of configuration options to be passed into that data store. Although specific configurations may vary between data stores, they all have a type, defined below.

AWS EC2 Instance Metadata Server

type: aws-local-meta

EC2 instances in the AWS cloud have access to a metadata server that reveals a handful of useful values for introspection by processes running on those hosts. Configuration is minimal, requiring only the name and type:

ec2-metadata:
  type: aws-local-meta

Data is referenced by the URL path used to access it. For example:

rtpl://ec2-metadata/latest/meta-data/hostname

AWS Secrets Manager

type: aws-secrets-manager

Secrets Manager is an AWS hosted service for managing access to encrypted secrets. This data store is configured by passing in options to the boto3 client constructor.

secrets:
  type: aws-secrets-manager
  region_name: us-west-2
  aws_access_key_id: ABCDEFG
  aws_secret_access_key: ABCDEFG

Data is referenced by the name of the secret. To retreive the value, the get_secret_value function of boto3 is called, and values referenced in the query portion of the rtpl URI are passed as arguments. For example, you can retrieve a specific secret version like this:

rtpl://secrets/my.super.duper.secret?VersionId=TheOldOne

AWS Systems Manager

type: aws-systems-manager

AWS Systems Manager is a service intended for making configuration information available to various infrastructure resources in an AWS environment. Retemplate supports using its parameter store as a data source. Like the Secrets Manager store, this is configured using options to the boto3 client constructor.

params:
  type: aws-systems-manager
  region_name: us-west-2

When referencing values, you can pass parameters to the get_parameter function in the rtpl query string:

rtpl://params/my_system_parameter?WithDecryption=True

Local Execution

type: local-exec

This allows Retemplate to run a command on the server it lives on and use the resulting stdout text as a value. This is useful for when the data you need to produce is more complex than a single stored value. It does run as the same user Retemplate does, so be cautious when using this, since Retemplate often needs to run as root. Registering a local execution data store does require that you define the command it's allowed to run. This is a very basic safety measure, but you don't get any security from this aside from deliberation. Use with caution.

timestamp:
  type: local-exec
  command: date

You can supply additional arguments to the command with a series of arg variables in the query string. For example, given the above config, the following URI will run the command: date --rfc-3339 seconds

rtpl://timestamp/?arg=--rfc-3339&arg=seconds

Redis

type: redis

Retemplate can talk to a Redis server to retrieve keys from it. This data store assumes Redis is running on the same machine (localhost), on the default port (6379), using the default database ID (0), without SSL, and with no authentication required. To override these settings, configure the data store:

hosted-redis:
  type: redis
  host: redis.myenvironment.com
  port: 6372
  db: 1
  ssl: True
  auth_token: a_secret_password

The path of the rtpl URI becomes the key to look up in Redis. So if you have previously...

SET foo bar

...you can retrieve that in an rtpl template with:

rtpl://hosted-redis/foo

Templates

Retemplate needs to know what files to template and various other options about each one. Here is a sample config:

templates:
  /opt/hiapi/config.txt:
    template: /etc/retemplate/hiapi.config.j2
    owner: root
    group: root
    chmod: "0600"
    onchange: supervisorctl restart hiapi
    frequency: 60
    random_offset_max: 30

Here, templates is the root-level option underneath which all of your templates will be defined. The filename after that (/opt/hiapi/config.txt) is the destination for the rendered file. Referencing this config, here is what the rest of the values mean.

• template: The source file, the unrendered template itself.
• owner: After rendering the template, the file will be owned by this user.
• group: After rendering the template, the file will be owned by this group.
• chmod: After rendering the template, this is the octal-format permission for the file. It must be a string, and due to how YAML interprets numerical values, you must explicitly surround the value with quotes to ensure it is interpreted as a string.
• onchange: If the rendered file differs from what was there before, this is a command that will be executed. This should be used for reloading services after their config files have changed. This value can be omitted if it is unnecessary. Retemplate will emit a log event saying no onchange command is to be run.
• frequency: The number of seconds to wait between template renders.
• random_offset_max: When present, this causes the rendering process to wait an additional amount of time - up to this number of seconds - before it gets underway. This is designed to prevent the alignment of jobs such that they all make API calls or disk access requests simultaneously. If not set, there is no additional time offset.

The owner, group, and chmod options are technically optional. If you do not supply them, the OS will leave these at the defaults for the user that Retemplate is running as. Because this software often runs as root, that could leave your files unreadable by the programs that need to use them. It is recommended that you explicitly set these options.

Writing Templates

The template rendering process is a three-stage one:

1. The template is pre-processed for template variable assignments. These are first looked up, and then references to the variables are substituted. This happens per-line, in a scan-ahead manner, so that variables defined at the top of a file can be used as part of an rtpl:// URI later in the file (see example below).
2. The template is processed to do non-variable lookups of values. These references are then substituted.
3. The template is run through the Jinja2 rendering process.

Preprocessor Variables

These are contained by curly and angle braces and look like this in a template:

{< var_name = rtpl://store/key?param=val >}

They are then referenced in the same way, but without the assignment portion:

{<var_name>}

Referenced variables must be absent of whitespace.

Default Values

If you want a failure to retrieve a value to not cause a failure to render a template, you can provide a default value by using the special rtpl_default argument in the URI. Consider this bit of template:

rtpl://secrets-manager/key?rtpl_default=Blah

In this case, if the secrets-manager store fails to find the key, it will use the provided default value of "Blah" instead.

Example

Let's consider a case where you need to configure an agent with an API key, but that API key varies depending on which environment your system runs in. Further, let's say you have a script on the server that, when run, emits the name of the environment, and that environment name is then used to look up the API key in AWS Secrets Manager.

To accomplish this, you might start with a configuration that looks like this:

retemplate:
  logging:
    filename: /var/log/retemplate.log
  stores:
    environment:
      type: local-exec
      command: /opt/get-environment.sh
    secrets:
      type: aws-secrets-manager
      region: us-west-2
  templates:
    /etc/agent/config.ini:
      template: /etc/retemplate/agent.config.ini.j2
      owner: agent
      group: agent
      chmod: "0600"
      onchange: systemctl restart agent
      frequency: 60
      random_offset_max: 30

You would then write the template that lives at /etc/retemplate/agent.config.ini.j2 to look something like this:

{< env = rtpl://environment/ >}
[agent]
api_key: rtpl://secrets/{<env>}.agent.api_key

When Retemplate processes this, it goes through the preprocessing stage, storing the output of the /opt/get-environment.sh script as the env variable. The assignment line is removed from the template entirely. The reference to {<env>} on the third line gets replaced with that value. Let's say the value turns out to be stage. At this point, when Retemplate goes into its second pass, the template looks like this:

[agent]
api_key: rtpl://secrets/stage.agent.api_key

On the second pass, the rtpl URI gets replaced with the value it finds in AWS Secrets Manager. So the template might look like this now:

[agent]
api_key: abcd1234

This is then passed to Jinja, which will not detect anything special. So the contents above will be written to /etc/agent/config.ini, and the command systemctl restart agent will be run if the file produced differs from the file already on disk.