[Fleet] Integrate the <LogStream /> component in the Agent details page.

[ph]

It is now possible to embed the logs component in other places in the UI, I think it would be great to be able to see the log of the Agent in the details page.

@jen-huang @hbharding WDYT?

Questions:

@ruflin Should we display only agent logs or we should include filebeat or metricbeat logs in the agent page? I think ideally we would just have the agent, but I think for now there is values in having the filebeat, metricbeat or endpoint logs.

[elasticmachine]

Pinging @elastic/ingest-management (Team:Ingest Management)

[ph]

@blakerouse Are we sending the agent uuid in the logs of the agent?

[jen-huang]

Just to clarify, this refers to Infra's LogStream component, right? https://github.com/elastic/kibana/blob/master/x-pack/plugins/infra/public/components/log_stream/README.md

👍 from me, all for reusing existing components.

[ph]

Looking at the doc above, can it display logs from multiple datastream? Initially we can show agent log but it would be nice to see metricbeat/filebeat logs too. cc @ruflin

[ruflin]

There is a query part and as all our indices use constant keywords for the dataset, we can add there multiple, not only for agent.

[jen-huang]

I'm looking into this and a few questions came up:

1. This does not replace the existing Activity Logs table, right? Rather it would just be another tab/table to view raw agent logs? The current activity logs table show agent event saved objects, which are created when the agent checks in with Fleet and does not seem to be 1:1 with the raw agent logs.

2. I took a look at the data that is ingested into logs-elastic.agent-* data stream to see which identifier field I can use to query for the agent that the user is looking at. But I couldn't find an overlap between agent IDs in the logs versus the agent saved object.

@ph @michalpristas Could you help me answer why the field agent.id in agent logs, is not the same as the local_metadata.elastic.agent.id that is sent to Fleet upon enrollment/check in?

Here is a redacted logs document:

{
  "_index": ".ds-logs-elastic.agent-default-000001",
  "_id": "SIn1wXQBg8juVJpHI2NE",
  "_version": 1,
  "_score": null,
  "_source": {
    "@timestamp": "2020-09-24T21:11:41.884Z",
    "host": {
      "id": "88769239-A9C7-5D8C-8907-ADA0E530270F",
      "ip": [...],
      "mac": [...],
      "hostname": "jens-mbp.lan",
      "architecture": "x86_64",
      "os": {
        "platform": "darwin",
        "version": "10.14.6",
        "family": "darwin",
        "name": "Mac OS X",
        "kernel": "18.7.0",
        "build": "18G2022"
      },
      "name": "jens-mbp.lan"
    },
    "log": {
      "offset": 21902,
      "file": {
        "path": "/Users/jen/Projects/elastic/elastic-agent-8.0.0-SNAPSHOT-darwin-x86_64 5/data/logs/elastic-agent-json.log"
      }
    },
    "log.level": "error",
    "ecs.version": "1.5.0",
    "input": {
      "type": "log"
    },
    "data_stream": {
      "namespace": "default",
      "type": "logs",
      "dataset": "elastic.agent"
    },
    "event": {
      "dataset": "elastic.agent"
    },
    "ecs": {
      "version": "1.6.0"
    },
    "log.origin": {
      "file.name": "application/fleet_gateway.go",
      "file.line": 176
    },
    "message": "Could not communicate with Checking API will retry, error: fail to checkin to fleet: Post \"http://localhost:5603/jen/api/ingest_manager/fleet/agents/387d125d-6931-4fc2-a3f6-9077ea27f48e/checkin?\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)",
    "agent": {
      "type": "filebeat",
      "version": "8.0.0",
      "ephemeral_id": "1b300a3c-9c68-4665-b560-aa53f1023fb0",
      "id": "7194bfb0-1281-47e2-8cc0-21d7499b4b15",
      "name": "jens-mbp.lan"
    },
    "container": {
      "id": "elastic-agent-8.0.0-SNAPSHOT-darwin-x86_64 5"
    }
  },
  "fields": {
    "@timestamp": [
      "2020-09-24T21:11:41.884Z"
    ]
  },
  "sort": [
    1600981901884
  ]
}

And here is the agent SO document:

{
  "_index" : ".kibana_2",
  "_id" : "fleet-agents:387d125d-6931-4fc2-a3f6-9077ea27f48e",
  "_score" : 2.9519293,
  "_source" : {
    "fleet-agents" : {
      "active" : true,
      "policy_id" : "d9413650-fea5-11ea-aac0-17011452a646",
      "type" : "PERMANENT",
      "enrolled_at" : "2020-09-24T21:03:20.480Z",
      "user_provided_metadata" : { },
      "local_metadata" : {
        "elastic" : {
          "agent" : {
            "id" : "387d125d-6931-4fc2-a3f6-9077ea27f48e",
            "version" : "8.0.0"
          }
        },
        "host" : {
          "architecture" : "x86_64",
          "hostname" : "jens-mbp.lan",
          "name" : "jens-mbp.lan",
          "id" : "88769239-A9C7-5D8C-8907-ADA0E530270F",
          "ip" : [...],
          "mac" : [...]
        },
        "os" : {
          "family" : "darwin",
          "kernel" : "18.7.0",
          "platform" : "darwin",
          "version" : "10.14.6",
          "name" : "Mac OS X",
          "full" : "Mac OS X(10.14.6)"
        }
      },
      "access_api_key_id" : "QontwXQBg8juVJpHfDAa",
      "default_api_key" : "4Pr0W/9nfQggLJ/Q7Y39k4Ho8SX9eE3JS+2b9XwElxR8mfTvfxdYgwRkdsZbCHgHj9ZdzEjtkPxPGSFCJL7FrlbmLDQ+uQQkHammRIXxVF/B8NdQMmLd3BiuQPlLq/RIBJ0So6Hcmqs09IDNwfcCqG5xSMNayiZIxYanpPglfu0e0O3nfrgCDd0=",
      "default_api_key_id" : "SontwXQBg8juVJpHlTA2",
      "last_checkin" : "2020-09-24T21:35:32.135Z",
      "policy_revision" : 1,
      "packages" : [
        "system"
      ]
    },
    "type" : "fleet-agents",
    "references" : [ ],
    "migrationVersion" : {
      "fleet-agents" : "7.10.0"
    },
    "updated_at" : "2020-09-24T21:35:32.136Z"
  }
}

[ruflin]

I think there is an other issue here which I also stumbled over and it is that elastic.agent.id is not reported as part of the events currently and I think it should. It is up to each process to ship agent.id or not, but elastic.agent.id should be in all events and log entries we ship to figure out from which agent the data is coming.

[jen-huang]

The missing elastic.agent.id will be fixed by elastic/beats#21543.

[jen-huang]

Work on this is currently blocked by elastic/beats#21864.

[jen-huang]

Tested today with 7.10 BC3 agent build and expected fields are now present 🎉 No longer blocked.