Doc: Clarify where input settings should be placed when using filebeat.config.inputs

[ppf2]

6.8.9

When using external config files, it is unclear what input settings are actually allowed under the filebeat.config.inputs construct. For example, if the user configures clean_inactive under filebeat.config.inputs, it does not throw a configuration error and treats it as a valid config, but it doesn't appear to apply to the inputs that are configured within the external files.

filebeat.config.inputs:
  enabled: true
  path: prospectors/*.yml
  clean_inactive: 24h

Specifying these settings within the external configuration files work. While the example on this page does show that scan_frequency is configured as part of the input block, because we are accepting these settings at the filebeat.config.inputs level, it will be helpful to clarify that these settings must all go into the external files, etc..

[elasticmachine]

Pinging @elastic/integrations-services (Team:Services)

[ycombinator]

I looked through the Filebeat code, specifically this path:

1. beats/filebeat/beater/filebeat.go

   Line 292 in 93a4591

   err = crawler.Start(fb.pipeline, config.ConfigInput, config.ConfigModules)

2. beats/filebeat/beater/crawler.go

   Lines 81 to 87 in 93a4591

   	if configInputs.Enabled() {
   	c.inputReloader = cfgfile.NewReloader(pipeline, configInputs)
   	if err := c.inputReloader.Check(c.inputsFactory); err != nil {
   	return fmt.Errorf("creating input reloader failed: %+v", err)
   	}
   	}

   .

3. beats/libbeat/cfgfile/reload.go

   Lines 107 to 123 in 93a4591

   	// NewReloader creates new Reloader instance for the given config
   	func NewReloader(pipeline beat.PipelineConnector, cfg *common.Config) *Reloader {
   	config := DefaultDynamicConfig
   	cfg.Unpack(&config)
   	path := config.Path
   	if !filepath.IsAbs(path) {
   	path = paths.Resolve(paths.Config, path)
   	}
   	return &Reloader{
   	pipeline: pipeline,
   	config: config,
   	path: path,
   	done: make(chan struct{}),
   	}
   	}

4. beats/libbeat/cfgfile/reload.go

   Lines 38 to 44 in 93a4591

   	// DefaultDynamicConfig provides default behavior for a Runner.
   	DefaultDynamicConfig = DynamicConfig{
   	Reload: Reload{
   	Period: 10 * time.Second,
   	Enabled: false,
   	},
   	}

5. beats/libbeat/cfgfile/reload.go

   Lines 58 to 64 in 93a4591

   	// DynamicConfig loads config files from a given path, allowing to reload new changes
   	// while running the beat
   	type DynamicConfig struct {
   	// If path is a relative path, it is relative to the ${path.config}
   	Path string `config:"path"`
   	Reload Reload `config:"reload"`
   	}

6. beats/libbeat/cfgfile/reload.go

   Lines 125 to 160 in 93a4591

   	// Check configs are valid (only if reload is disabled)
   	func (rl *Reloader) Check(runnerFactory RunnerFactory) error {
   	// If config reload is enabled we ignore errors (as they may be fixed afterwards)
   	if rl.config.Reload.Enabled {
   	return nil
   	}
   	debugf("Checking module configs from: %s", rl.path)
   	gw := NewGlobWatcher(rl.path)
   	files, _, err := gw.Scan()
   	if err != nil {
   	return errors.Wrap(err, "fetching config files")
   	}
   	// Load all config objects
   	configs, err := rl.loadConfigs(files)
   	if err != nil {
   	return errors.Wrap(err, "loading configs")
   	}
   	debugf("Number of module configs found: %v", len(configs))
   	// Initialize modules
   	for _, c := range configs {
   	// Only add configs to startList which are enabled
   	if !c.Config.Enabled() {
   	continue
   	}
   	if err = runnerFactory.CheckConfig(c.Config); err != nil {
   	return err
   	}
   	}
   	return nil
   	}

According to the code, we are not expecting any input settings to be specified directly under filebeat.config.inputs. They must be specified under each input defined in the external input configuration files. +1 to clarifying this in the documentation.

[ppf2]

Thanks for confirming! Submitted a PR to clarify documentation until we start validating configuration settings for Beats.

[dedemorton]

Closed by #19284