Enhancement Proposal: API to Forward Logs to CloudWatch

[alanconway]

See:

• Epic https://issues.redhat.com/browse/LOG-998
• API issue: https://issues.redhat.com/browse/LOG-1001

/cc @jcantrill
/cc @jeremyeder
/cc @sichvoge

[openshift-ci-robot]

@alanconway: GitHub didn't allow me to assign the following users: jcantril.

Note that only openshift members, repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time.
For more information please see the contributor guide

In response to this:

/cc @jcantrill
/cc @jeremyeder
/cc @sichvoge
/assign @jcantril

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[alanconway]

/assign @jcantrill

[portante]

This seems a bit complicated for a first pass with the ability to set groups and streams arbitrarily.

Is there a way to just support 3 groups (app, infra, audit) and/or 2 groups (infra & audit) and apps grouped by namespace? Seems like those would be easier to get right in the very, very short amount of time we have to implement this.

[alanconway]

@jewzaam @portante @jcantrill update taking account of all your comments (thanks for that) but may still be some questions outstanding.

[jcantrill]

/hold

[alanconway]

@jcantrill @portante I think the latest draft may be a winner, please comment if you agree so we can LGTM and unhold.

[alanconway]

Great feedback, I think we're close. @portante @jcantrill please scan again. I've clarified the role of log stream names, otherwise its small changes.

[sichvoge]

I added an additional comment for retentionDays since I am not comfortable to actually expose this in our API. See explanation in my comment.

[alanconway]

@jcantrill @portante @sichvoge all comments taken on board, properly updated this time (I managed to check this out twice so brief confusion) Please comment.

[jcantrill]

/approve

[openshift-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: jcantrill

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [jcantrill]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[jcantrill]

Should we call this source to be consistent with the rest of logforwarding

[jcantrill]

I don't know what or how to get node-id but the node name is same as host name. Landing on this which would give you journal and all audit logs grouped by node hostname:

<label {{.LabelName}}>
  <filter kubernetes.**>
    @type record_transformer
    <record>
      cw_group_name {{.LogGroupName }}
      cw_stream_name ${record["kubernetes"]["container_image_id"]}
    </record>
  </filter>
  <filter journal.** *audit.log>
    @type record_transformer
    <record>
      cw_group_name ${record["hostname"]}
      cw_stream_name ${tag}
    </record>
  </filter>

I can remove the container streams to use tag instead of the hash which would include NS/Podname/containerhash some thing like var.log.containers.NS.podname.hash or the like

[jcantrill]

/hold cancel
/lgtm