out_cloudwatch_logs: support record_accessor

[PettitWesley]

#3246

Doc PR: fluent/fluent-bit-docs#834

---

Enter [N/A] in the box, if an item is not applicable to your change.

Testing
Before we can approve your change; please submit the following in a comment:

• Example configuration file for the change
• Debug log output from testing the change

• Attached Valgrind output that shows no leaks or memory corruption was found

If this is a change to packaging of containers or native binaries then please confirm it works for all targets.

• Attached local packaging test output showing all targets (including any new ones) build.

Documentation

• Documentation required for this feature

Backporting

• Backport to latest stable release.

---

Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.

[PettitWesley]

So this feature uses record accessor to template CW log stream and group names: https://docs.fluentbit.io/manual/v/1.5/administration/configuring-fluent-bit/record-accessor

This is a higher perf version of the golang feature: https://github.com/aws/amazon-cloudwatch-logs-for-fluent-bit#templating-log-group-and-stream-names

It is not exactly the same feature however, since record accessor has a different syntax and works differently.

We have two new config items:

• log_group_template
• log_stream_template

The existing log group and stream config options are used as fallbacks.

Here's an example config:

[PARSER]
    Name   json
    Format json

[SERVICE]
    Parsers_File      /home/ec2-user/configs/parse-json.conf

[INPUT]
    Name              tail
    Path              /home/ec2-user/configs/logs/*
    refresh_interval  2
    rotate_wait       5
    db                /home/ec2-user/configs/fb.db
    db.sync           normal
    db.locking        true

[FILTER]
    Name parser
    Match *
    Key_Name log
    Parser json
    Reserve_Data True

#[OUTPUT]
#    Name stdout
#    Match *

[OUTPUT]
    Name cloudwatch_logs
    Match   *
    region us-east-1
# these are the default/fallback names
    log_group_name fallback-group
    log_stream_prefix fallback-stream
    auto_create_group On
# templates which are used if the templating succeeds, otherwise an error is logged
    log_group_template group.$TAG.$kubernetes['host'].$kubernetes['namespace_name'].$kubernetes['pod_name'].$kubernetes['container_name']
    log_stream_template $counter.stream.$TAG.$kubernetes['host'].$kubernetes['namespace_name'].$kubernetes['pod_name'].$kubernetes['container_name'].$metadata['value']

And here is my logger script which creates fake logs that have fake k8s metadata for use with the template:

import datetime
from collections import OrderedDict
import time
import os
import json
import signal
import random
import sys
import requests

word_site = "https://www.mit.edu/~ecprice/wordlist.10000"

response = requests.get(word_site)
WORDS = response.content.splitlines()

ID = 0

def print_log():
    global ID
    obj = OrderedDict({
            "kubernetes": {
                "annotations": {
                    "kubernetes.io/psp": "eks.privileged"
                },
                "container_hash": "<some hash>",
                "container_name": "myapp",
                "docker_id": "<some id>",
                "host": "ip-10-1-128-166.us-east-2.compute.internal",
                "labels": {
                    "app": "myapp",
                    "pod-template-hash": "<some hash>"
                },
                "namespace_name": "default",
                "pod_id": "198f7dd2-2270-11ea-be47-0a5d932f5920",
                "pod_name": "myapp-5468c5d4d7-n2swr"
            }
    })
    ID += 1
    obj['counter'] = ID
    if ID % 2 == 0:
        word = random.choice(WORDS)
        obj['metadata'] = {}
        obj['metadata']['value'] = word.decode("utf-8")
    s = json.dumps(obj)
    print(s)

for i in range(200):
    print_log()

[PettitWesley]

@edsiper @nokute78 This change makes a small addition to the record_accessor lib. So I need review and approval from a core maintainer.

[nokute78]

@PettitWesley I reviewed flb_record_accessor.[ch].
Could you add a test code to tests/internal/record_accessor.c ?

[nokute78]

How about not using check variable like return flb_ra_translate_check(ra, tag, tag_len, map, result, FLB_FALSE); ?

[PettitWesley]

fixed

[nokute78]

Is loopup a typo ?

[PettitWesley]

yeah oops... thanks

[PettitWesley]

fixed

[edsiper]

#11 134.8 /src/tests/internal/record_accessor.c:1458:5: error: expected ',' or ';' before 'json'
#11 134.8      json = "{\"root.with/symbols\": \"something\"}";

[PettitWesley]

@edsiper fixed now

[PettitWesley]

@nokute78 I added a unit test for flb_ra_translate_check

[edsiper]

NOTE: CI was failing due upstream tests, that test has been reverted.