Metricbeat fails to publish elasticsearch metricbeat documents due to require_alias error

[matschaffer]

We observed in this deployment from @jasonrhodes https://admin.found.no/deployments/1df5d5b91b5a49fd9f9fa6fc2e93b51a that metricbeat was failing to publish ES monitoring docs with the following error:

Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Date(2022, time.January, 27, 0, 22, 0, 50458769, time.Local), Meta:{"index":".monitoring-es-7-mb"}, Fields:{"agent":{"ephemeral_id":"9f635268-bace-41b6-a2bd-6f3eab0f19ab","hostname":"5d886d0ad862","id":"649e5cfd-66dc-4ab0-88da-12a66a65f81e","name":"5d886d0ad862","type":"metricbeat","version":"7.17.0"},"cluster_uuid":"1G-2qN3BS6-UcYX8I9DHpw","ecs":{"version":"1.12.0"},"enrich_coordinator_stats":{"executed_searches_total":0,"node_id":"MIcZUsjeSqC90MU5trc2wQ","queue_size":0,"remote_requests_current":0,"remote_requests_total":0},"event":{"dataset":"elasticsearch.enrich","duration":6114284,"module":"elasticsearch"},"host":{"name":"5d886d0ad862"},"interval_ms":10000,"metricset":{"name":"enrich","period":10000},"service":{"address":"https://5d886d0ad862:18669","type":"elasticsearch"},"timestamp":"2022-01-27T00:22:00.056Z","type":"enrich_coordinator_stats"}, Private:interface {}(nil), TimeSeries:false}, Flags:0x0, Cache:publisher.EventCache{m:common.MapStr(nil)}} (status=404): {"type":"index_not_found_exception","reason":"no such index [.monitoring-es-7-mb-2022.01.27] and [require_alias] request flag is [true] and [.monitoring-es-7-mb-2022.01.27] is not an alias","index_uuid":"_na_","index":".monitoring-es-7-mb-2022.01.27"}, dropping event!

This is unexpected since the .monitoring-es-7-* indices are not expected to have aliases.

@lucasmoore provided the metricbeat config for the master instance of the deployment, which contain no reference to alias requirements.

metricbeat.yml

output.elasticsearch:
  hosts: ["http://containerhost:9244"]
  username: "elastic-observability-agent"
  password: "redacted"
  headers:
    X-Found-Cluster: d733939974c6401d95c05b2696664ac1
    X-Elastic-App-Auth: redacted


  ssl.verification_mode: full



setup.template.overwrite: false

metricbeat.config.modules:
  enabled: true
  path: ${path.config}/modules.d/*.yml
  reload.enabled: true
  reload.period: 30s

logging.level: info
logging.to_files: true
logging.json: true
logging.files:
  path: /app/logs/beats
  name: metricbeat.log
  keepfiles: 3



queue.disk:
  max_size: "1GB"
  read_ahead: 512
  write_ahead: 2048

modules.d/elasticsearch.yml

- module: elasticsearch
  metricsets:
    - ccr
    - enrich
    - cluster_stats
    - index
    - index_recovery
    - index_summary
    - ml_job
    - node_stats
    - shard
  period: 10s
  hosts: ["https://${HOSTNAME:REDACTED}:REDACTED"]

  username: "ec-local-beats-monitor"


  password: "redacted"

  headers:


  ssl.verification_mode: none
  xpack.enabled: true

I found #29879 which involves require_alias and was recently merged to 7.17. So at the moment I suspect it's what's causing the incorrect expectation, but I haven't gone over the PR in detail yet.

cc @kvch who opened the PR and may have additional context

Rel: #29920 (probably same issue, but that issue seemed ECK focused so I didn't see the connection until I'd already opened this one)

[elasticmachine]

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

[jasonrhodes]

Nice work, @matschaffer! I suspect that the client.index.IsAlias() function is returning the wrong thing:

https://github.com/elastic/beats/pull/29879/files#diff-d011e52e4fe701c353d9ac635e322c5c2bb55e35c0eaaff9d1f56bbce7d67a7aR321

I've honestly gotten myself lost on alias vs not alias at the index level. I assume this is because MB in 8.0 writes to data streams which are not write aliases? Can you just confirm my understanding there?

[matschaffer]

I assume this is because MB in 8.0 writes to data streams which are not write aliases? Can you just confirm my understanding there?

You are correct as far as my understanding goes as well. On 7.x metricbeat-collected monitoring data should be raw indices (no aliases) and on 8.0 they should be data streams (also not aliases, at least as far as beats is concerned).

Internal collection will be raw indices in both 7 & 8.

[kvch]

The revert is merged.