[Fleet] "Package policy is invalid: inputs.logfile.streams.log.log.vars.paths: Invalid format" when preconfiguring custom logs integration

[kpollich]

Kibana version: 7.15

Ref elastic/cloud-on-k8s#4873 (comment)

Summary

When attempting to preconfigure the custom logs integration, Kibana throws an error during Fleet setup: Package policy is invalid: inputs.logfile.streams.log.log.vars.paths: Invalid format:

Steps to Reproduce

Add preconfigured values for xpack.fleet.packages and xpack.fleet.agentPolicies as below:

xpack.fleet.packages:
  - name: log
    version: latest
xpack.fleet.agentPolicies:
  - name: Custom Logs Policy
    id: custom-logs-123
    namespace: default
    package_policies:
      - package:
          name: log
        name: log-1-preconfigured
        inputs:
          - type: logfile
            enabled: true
            streams:
              - data_stream:
                  dataset: log.log
                enabled: true
                vars:
                  - name: paths
                    value: '/var/log/wifi.log'

Then, attempt to load the Fleet or Integrations apps. Observe the "Error during Fleet setup" as above.

[kpollich]

Interestingly, I'm able to successfully create a custom logs policy via the Fleet UI, but it generates a package policy where paths is a text array field.

{
            "name" : "log-1",
            "description" : "",
            "namespace" : "default",
            "policy_id" : "custom-logs-123",
            "enabled" : true,
            "output_id" : "",
            "inputs" : [
              {
                "type" : "logfile",
                "policy_template" : "logs",
                "enabled" : true,
                "streams" : [
                  {
                    "enabled" : true,
                    "data_stream" : {
                      "type" : "logs",
                      "dataset" : "log.log"
                    },
                    "vars" : {
                      "paths" : {
                        "type" : "text",
                        "value" : [
                          "/var/log/wifi.log"
                        ]
                      },
                      "data_stream.dataset" : {
                        "value" : "generic",
                        "type" : "text"
                      },
                      "custom" : {
                        "value" : "",
                        "type" : "yaml"
                      }
                    },
                    "id" : "logfile-log.log-1933315a-69ce-46d2-b756-178ec33d706f",
                    "compiled_stream" : {
                      "paths" : [
                        "/var/log/wifi.log"
                      ],
                      "data_stream" : {
                        "dataset" : "generic"
                      }
                    }
                  }
                ]
              }
            ],
            "package" : {
              "name" : "log",
              "title" : "Custom logs",
              "version" : "0.5.0"
            },
            "revision" : 1,
            "created_at" : "2021-09-29T13:26:48.142Z",
            "created_by" : "elastic",
            "updated_at" : "2021-09-29T13:26:48.142Z",
            "updated_by" : "elastic"
          },
          "type" : "ingest-package-policies",
          "references" : [ ],
          "migrationVersion" : {
            "ingest-package-policies" : "7.15.0"
          },
          "coreMigrationVersion" : "8.0.0",
          "updated_at" : "2021-09-29T13:26:48.147Z"
        }

This makes sense, considering paths is configured with multi: true per https://github.com/elastic/integrations/blob/a818e388efe60ca0dd2f88026a6fa5a2d742f76a/packages/log/data_stream/log/manifest.yml#L13

Updating the YML in the preconfiguration file to specify paths as an array value fixed the issue for me, e.g.:

xpack.fleet.packages:
  - name: log
    version: latest
xpack.fleet.agentPolicies:
  - name: Custom Logs Policy
    id: custom-logs-123
    namespace: default
    package_policies:
      - package:
          name: log
        name: log-1-preconfigured
        inputs:
          - type: logfile
            enabled: true
            streams:
              - data_stream:
                  dataset: log.log
                enabled: true
                vars:
                  - name: paths
                    value:
                      - /var/log/wifi.log

@david-kow - Are you able to share the config you're using and still seeing issues with in Kibana? I'd like to try and translate one-to-one into my kibana.dev.yml and attempt to reproduce that way.

[kpollich]

It looks like this error is expected behavior on Fleet's side. Previously, we weren't validating package policy inputs during the preconfiguration/setup process, but as of this PR , we validate package policies any time we perform our input overriding logic whether it's on the client as part of the policy editor UI or on the server as part of preconfiguration/setup.

b1253db#diff-3c2c778705d1b16042158c84def8fa9129aaf694f05217944a8231434d5104c1R959-R971

So @david-kow and @barkbay - you should be able to provide an array value for paths now. There was another related issue with overriding during preconfiguration in general that was fixed in #113204. So this PR fixed the root issue with preconfigured variables not applying in some situations, and the error message reported in this issue is expected behavior when a non-array value is provided for a variable configured with multi: true.

Closing this issue out.

[thbkrkr]

Specifying paths as an array value like this:

      package_policies:
      - package:
          name: log
        name: log-1
        inputs:
        - type: logfileou 
          enabled: true
          streams:
          - data_stream:
              dataset: log.log
            enabled: true
            vars:
            - name: paths
              value:
              - '/var/log/containers/*${kubernetes.container.id}.log'
            - name: custom
              value: |
                symlinks: true
                condition: ${kubernetes.namespace} == 'default'

fails in 7.15.2 with a slightly different error:

Package policy is invalid: inputs.logfile.streams.log.log.vars.paths: Log file path is required

@kpollich Are you aware of a change in the custom log integration in 7.15.2?

[kpollich]

@kpollich Are you aware of a change in the custom log integration in 7.15.2?

I am not aware of any such changes, no. The log integration doesn't seem to change much per its changelog here: https://github.com/elastic/integrations/blob/master/packages/log/changelog.yml

In the YML you pasted above, you've got type: logfileou - is that an actual typo or a copy/paste error? I think the issue might be that we're looking for an input w/ type logfile here and that's why the error about Log file path not being supplied is surfacing.

I ran through Kibana setup successfully with the following in my kibana.dev.yml and confirmed the values I expected in Fleet UI:

xpack.fleet.packages:
  - name: log
    version: latest
xpack.fleet.agentPolicies:
  - name: Custom Logs Policy
    id: custom-logs-123
    namespace: default
    package_policies:
    - package:
        name: log
      name: log-1
      inputs:
      - type: logfile
        enabled: true
        streams:
        - data_stream:
            dataset: log.log
          enabled: true
          vars:
          - name: paths
            value:
            - '/var/log/containers/*${kubernetes.container.id}.log'
          - name: custom
            value: |
              symlinks: true
              condition: ${kubernetes.namespace} == 'default'

[thbkrkr]

Oh sorry for the typo but it is only in GitHub, I test well with type: logfile. Thanks for checking out.

I tested with the 'Custom Logs Policy' you shared below (gist), Kibana and Fleet start but Elastic Agents fail with:

> k logs elastic-agent-agent-jqhjl 
Performing setup of Fleet in Kibana

Error: unable to find policy named ""
For help, please see our troubleshooting guide at https://www.elastic.co/guide/en/fleet/7.15/fleet-troubleshooting.html

Also I see the agent policy (https://localhost:5601/app/fleet/policies/custom-logs-123) but it does not have any integrations and I don't see any custom logs integration in https://localhost:5601/app/integrations/detail/log-0.4.6/policies compared to you. There must be something else wrong on my side.

I noticed that if I removed: is_default: true set at the agent policy level, I get the same behaviour, Kibana and Fleet start but the Elastic Agent fail.

If I add it to your example, I get the package policy error Log file path is required when loading integrations in Kibana. Could you try?

   - name: Custom Logs Policy
      is_default: true
      id: custom-logs-123
      namespace: default

[thbkrkr]

I haven't made much progress. Just saw it works fine in 7.15.1 and 7.16.0-SNAPSHOT but not in 7.15.2.

@kpollich Did you test in 7.15.2 or 7.16.0-SNAPSHOT in #113400 (comment)?.

[kpollich]

I haven't made much progress. Just saw it works fine in 7.15.1 and 7.16.0-SNAPSHOT but not in 7.15.2.

@kpollich Did you test in 7.15.2 or 7.16.0-SNAPSHOT in #113400 (comment)?.

Think I tested in 7.16.0-SNAPSHOT. I can spin up a 7.15.2 environment and test again shortly. Will report back.

[kpollich]

Confirming I see the same issue on 7.15.2

I created a 7.15.2 local dev environment and spent an hour or so trying to track down why this is happening in 7.15.2 without any real luck.

[MachLearnPort]

Hi, this issue is still persisting on 8.4.1 - has a fix or solution been proposed? I am looking to leverage this to fully automate and config my stack.

Thanks

[kpollich]

Hi @MachLearnPort - I am going to reopen this and we'll take a look soon on 8.4.1.

[criamico]

I tried to reproduce this bug locally on current main (8.7.0-snapshot) with this preconfiguration on my kibana.dev.yml

xpack.fleet.packages:
  - name: log
    version: latest
xpack.fleet.agentPolicies:
  - name: Custom Logs Policy
    id: custom-logs-123
    namespace: default
    package_policies:
      - package:
          name: log
        name: log-1-preconfigured
        inputs:
          - type: logfile
            enabled: true
            streams:
              - data_stream:
                  dataset: log.log
                enabled: true
                vars:
                  - name: paths
                    value: '/var/log/containers/*${kubernetes.container.id}.log'

For me fleet is starting normally and the agent is working as expected, I see no failures or other strange behaviors.

@kpollich I believe that there have been several changes done to the preconfiguration logic, am I correct?

[kpollich]

@criamico - Yes you are correct. This was likely fixed elsewhere in updates to preconfiguration logic/policy validation. I'm closing this for now.

Please re-open if this occurs on a particular stack version of custom logs integration version. Many thanks!

[aalagia90]

Hi,
i have tried to install this solution cloud-on-k8s/config/recipes/elastic-agent/fleet-custom-logs-integration.yaml at main · elastic/cloud-on-k8s · GitHub but the package custom logs is not present in the elk policy. When the kibana start i have seen this error

Cattura

The version that i use of components are

kibana,elastic,fleet,elastic-agent : 8.13.2
eck operator: 2.12
log: 2.3.1

The configuration works until log 1.1.2 from 2.0.0 doesn't works

This is my yaml file

apiVersion: kibana.k8s.elastic.co/v1
kind: Kibana
metadata:
  name: kibana
spec:
  version: 8.13.2
  count: 1
  elasticsearchRef:
    name: elasticsearch
  config:
    xpack.fleet.agents.elasticsearch.hosts: ["https://elasticsearch-es-http.default.svc:9200"]
    xpack.fleet.agents.fleet_server.hosts: ["https://fleet-server-agent-http.default.svc:8220"]
    xpack.fleet.packages:
    - name: system
      version: latest
    - name: elastic_agent
      version: latest
    - name: fleet_server
      version: latest
    - name: log
      version: latest  
    xpack.fleet.agentPolicies:
    - name: Fleet Server on ECK policy
      id: eck-fleet-server
      namespace: default
      monitoring_enabled:
      - logs
      - metrics
      unenroll_timeout: 900
      package_policies:
      - name: fleet_server-1
        id: fleet_server-1
        package:
          name: fleet_server
    - name: Elastic Agent on ECK policy
      id: eck-agent
      namespace: default
      monitoring_enabled:
      - logs
      - metrics
      unenroll_timeout: 900
      package_policies:
      - name: system-1
        id: system-1
        package:
          name: system
      - package:
          name: log
        name: log-1
        inputs:
        - type: logfile
          enabled: true
          streams:
          - data_stream:
              dataset: log.log
            enabled: true
            vars:
            - name: paths
              value:
              - '/var/log/containers/*${kubernetes.container.id}.log'
            - name: custom
              value: |
                symlinks: true
                condition: ${kubernetes.namespace} == 'default'
---

Thank you very much