Logs Data for "elastic_agent" dataset not generated by Agent

[amolnater-qasource]

Kibana version: 7.14.0 Snapshot Kibana self-managed environment

Host OS and Browser version: All, All

Build details:

Build: 42089
Commit: 67a71c75d2da40e49fba2620f488c9b4ce2467d2
Artifact Links: 
https://snapshots.elastic.co/7.14.0-15b00b37/downloads/elasticsearch/elasticsearch-7.14.0-SNAPSHOT-windows-x86_64.zip
https://snapshots.elastic.co/7.14.0-15b00b37/downloads/kibana/kibana-7.14.0-SNAPSHOT-windows-x86_64.zip
https://snapshots.elastic.co/7.14.0-15b00b37/downloads/beats/elastic-agent/elastic-agent-7.14.0-SNAPSHOT-windows-x86_64.zip

Preconditions:

1. 7.14.0 Snapshot Kibana self-managed environment should be available.
2. Elastic Agent must be installed with Default Fleet Server Policy having Fleet Server integration.

Steps to reproduce:

1. Login to Kibana environment.
2. Navigate to Data Streams tab and observe no logs type data for "elastic_agent" dataset.

Expected Result:
Logs data for "elastic_agent" dataset should be generated under Data Streams tab.

Logs:
logs.zip

Screenshot:

[elasticmachine]

Pinging @elastic/agent (Team:Agent)

[amolnater-qasource]

@manishgupta-qasource Please review.

[manishgupta-qasource]

Reviewed & Assigned to @EricDavisX

[EricDavisX]

@amolnater-qasource can you attach the policy the Agent was on? I don't want to make assumptions, please. It may be the expected result - but maybe not, the policy will help inform. Thank you.

[EricDavisX]

I think I have another scenario impacted by the cause of this issue: deploy an Agent and go to the Fleet 'logs' tab for the Agent and no logs are shown (because elastic_agent dataset is pre-selected, but no logs exist and it cannot then be unselected!). If you select other datasets and then also select a different time frame (to make the data pull in anew) then it will show some logs - very scary at first. The Fleet UI is acting up too, but that is follow up separately.

[EricDavisX]

note how the UI shows *something as selected, but it isn't in the list check-marked (that is the elastic_agent.elastic_agent logs dataset), and any UI concern is separate, this is just showing the data isn't there to be seen:

[EricDavisX]

The above may be the exact cause of this, too - it has additional logs if helpful.
I found this challenging to triage due to a Kibana Fleet side bug, fyi - elastic/kibana#103563 if you're using the UI as you work through this.

[amolnater-qasource]

Hi @EricDavisX

Can you attach the policy the Agent was on? I don't want to make assumptions, please. It may be the expected result - but maybe not, the policy will help inform.

Please find Default Fleet Server Policy attached below:
elastic-agent.zip

Further, we have revalidated your observations and also found it reproducible on cloud environments.

Build details:

Build: 42091
Commit: fe3ea8ed1fc80caa3acd805d24a1e9ea882aa30f
Artifact Link: https://snapshots.elastic.co/7.14.0-b3f1839d/downloads/beats/elastic-agent/elastic-agent-7.14.0-SNAPSHOT-windows-x86_64.zip

Please let us know if anything else is required from our end.
Thanks
QAS

[michalpristas]

did something changed fleet/permissions side?
i'm looking at this locally
we havent changed much recently but what i'm looking at is that agent generates config with correct path and settings, this config is then picked up by monitoring filebeat which processes files and sends events.
dataset is not populated though

when i change dataset from elastic_agent to elastic_agent.elastic_agent and index from logs-elastic_agent-default to logs-elastic_agent.elastic_agent-default i see events in discovery view, agent logs tab is not populated though (which is expected as it filters by dataset and agent id)

when i keep dataset to elastic_agent so filtering works inside Logs tab in Agent detail page and keep index modified to logs-elastic_agent.elastic_agent-default no events are propagated as well.

cc @ruflin @blakerouse

[ruflin]

@afgomez Can you chime in here? My guess is that this is related to the permissions in Kibana. It might be that we miss something in the package itself?

[afgomez]

The permissions generated for the agent are as follows:

output_permissions:
  default:
    _elastic_agent_checks:
      cluster:
        - monitor
      indices:
        - names:
            - logs-elastic_agent.*-default
            - metrics-elastic_agent.*-default
          privileges:
            - auto_configure
            - create_doc

My guess is that the . after logs-elastic_agent. should go away. I can open a PR for it

[michalpristas]

yeah this seems incorrect, please let me know when you have a PR, this will also need to be be backported to 7.14

[michalpristas]

@amolnater-qasource please retest when elastic/kibana#104447 goes in

[amolnater-qasource]

Hi @michalpristas
Thanks, we will revalidate this once build with latest merges will be available.

[amolnater-qasource]

Hi @EricDavisX
We have revalidated this issue on 7.14.0 Snapshot and found it fixed.

• "elastic_agent" dataset are generated for Agent under Data Streams tab and Logs tab

Build details:

Build: 42366
Commit: 22dee04008b9936be37225b97a6456e750d559a7
Artifact Link: https://snapshots.elastic.co/7.14.0-ef1f955b/downloads/beats/elastic-agent/elastic-agent-7.14.0-SNAPSHOT-windows-x86_64.zip

Screenshot:

Hence we are closing this out.
cc: @michalpristas
Thanks
QAS