[Elastic Log Driver] Create a config shim between libbeat and the user #18605
Conversation
fearful-symmetry
added
enhancement
Team:Integrations
|
Pinging @elastic/integrations (Team:Integrations) |
botelastic
bot
added
needs_team
💚 Build Succeeded
Expand to view the summary
Build stats
Test stats 🧪
|
|
+1 on only exposing a small set of settings via CLI. Do we only want to support Elasticsearch output, or other outputs as well? Having only a small number of settings, how about making them configurable via environment variables as well? |
|
@urso I want to support additional inputs later, yah. Right now I'm hoping to have one small-ish PR to add support for the new CLI scheme, then later on add more inputs as needed. Environment variable support could work, although it adds a great deal of ugliness to the config, as we need to hard-code any option that a user might want to fiddle with. |
fearful-symmetry
requested review from
urso,
bpintea and
a team
and removed request for
bpintea
|
Just added support for the additional ES settings. Not sure if it's worth adding new outputs now, or just getting this through so we can move on to other components, since we're hoping this is just a shim until we can get this integrated with fleet. |
💚 Build Succeeded
Expand to view the summary
Build stats
Test stats 🧪
|
💚 Build Succeeded
Expand to view the summary
Build stats
Test stats 🧪
|
elastic#18605) * init commit of config shim * update docs * make check * add timeout * move config system to use typeconv * add rest of backoff settings. * make fmt * some cleanup * use uint64 hash for structs * make fmt * create custom index manager, remove ILM support * add support for multiple endpoints * update tests * update docs * remove setup options * remove old tests * try to update asciidocm change 'endpoint' to 'hosts' * trying to fix CI * update docs * fix backtics (cherry picked from commit 73150c2)
#18605) (#18864) * init commit of config shim * update docs * make check * add timeout * move config system to use typeconv * add rest of backoff settings. * make fmt * some cleanup * use uint64 hash for structs * make fmt * create custom index manager, remove ILM support * add support for multiple endpoints * update tests * update docs * remove setup options * remove old tests * try to update asciidocm change 'endpoint' to 'hosts' * trying to fix CI * update docs * fix backtics (cherry picked from commit 73150c2)
…-stage-level * upstream/master: (30 commits) Add a GRPC listener service for Agent (elastic#18827) Disable host.* fields by default for iptables module (elastic#18756) [WIP] Clarify capabilities of the Filebeat auditd module (elastic#17068) fix: rename file and remove extra separator (elastic#18881) ci: enable JJBB (elastic#18812) Disable host.* fields by default for Checkpoint module (elastic#18754) Disable host.* fields by default for Cisco module (elastic#18753) Update latest.yml testing env to 7.7.0 (elastic#18535) Upgrade k8s.io/client-go and k8s keystore tests (elastic#18817) Add missing Jenkins stages for Auditbeat (elastic#18835) [Elastic Log Driver] Create a config shim between libbeat and the user (elastic#18605) Use indexers and matchers in config when defaults are enabled (elastic#18818) Fix panic on `metricbeat test modules` (elastic#18797) [CI] Fix permissions in MacOSX agents (elastic#18847) [Ingest Manager] When not port are specified and the https is used fallback to 443 (elastic#18844) [Ingest Manager] Fix install service script for windows (elastic#18814) [Metricbeat] Fix getting compute instance metadata with partial zone/region config (elastic#18757) Improve error messages in s3 input (elastic#18824) Add memory metrics into compute googlecloud (elastic#18802) include bucket name when logging error (elastic#18679) ...
What does this PR do?
This PR adds a "config shim" for the plugin. This is currently a draft that hard-codes a number of fundamental configs needed for the plugin to run, as opposed to letting the user specify unlimited dot-config options that we may or may not have tested.
Why is it important?
Right now, the config system for the plugin is one of the most user-hostile components of the plugin. We expose "raw" libbeat config options similar to filebeat command line options. In the long-term, a config system like this is untenable, as specifying complex config logic, nested options, and lists is incredibly awkward and verbose. This means that right now, we have none of the upsides of using libbeat's config, and the downside of relatively verbose config flags for basic options, i.e,
output.elasticsearch.host.Checklist
CHANGELOG.next.asciidocorCHANGELOG-developer.next.asciidoc.How to test this PR locally
All supported config flags are documented and in
config.go. Pull down the change, build the plugin on a docker host withmage BuildAndInstall, then test with a command likedocker run -it --log-driver elastic/elastic-logging-plugin:8.0.0 --log-opt endpoint=griffon.nest:9200 debian:latest /bin/bash