Datadog Filebeat integration: Registry denied for datadog to read

[kalilou]

Output of the info page

Redirecting to /bin/systemctl status datadog-agent.service
● datadog-agent.service - Datadog Agent
   Loaded: loaded (/usr/lib/systemd/system/datadog-agent.service; enabled; vendor preset: disabled)
   Active: active (running) since Mon 2020-04-27 20:25:32 UTC; 13h ago
 Main PID: 20068 (agent)
   CGroup: /system.slice/datadog-agent.service
           └─20068 /opt/datadog-agent/bin/agent/agent run -p /opt/datadog-agent/run/agent.pid

Apr 28 09:58:49 HOSTNAME sudo[27519]: dd-agent : TTY=unknown ; PWD=/ ; USER=asterisk ; COMMAND=/sbin/asterisk -rx core ...annels
Apr 28 09:58:56 HOSTNAME agent[20068]: 2020-04-28 09:58:56 UTC | CORE | ERROR | (pkg/collector/python/datadog_agent.go:....json'
Apr 28 09:58:56 HOSTNAME agent[20068]: 2020-04-28 09:58:56 UTC | CORE | ERROR | (pkg/collector/python/datadog_agent.go:...l/6455
Apr 28 09:59:04 HOSTNAME sudo[27590]: dd-agent : TTY=unknown ; PWD=/ ; USER=asterisk ; COMMAND=/sbin/asterisk -rx core ...annels
Apr 28 09:59:11 HOSTNAME agent[20068]: 2020-04-28 09:59:11 UTC | CORE | ERROR | (pkg/collector/python/datadog_agent.go:....json'
Apr 28 09:59:11 HOSTNAME agent[20068]: 2020-04-28 09:59:11 UTC | CORE | ERROR | (pkg/collector/python/datadog_agent.go:...l/6455
Apr 28 09:59:19 HOSTNAME sudo[27664]: dd-agent : TTY=unknown ; PWD=/ ; USER=asterisk ; COMMAND=/sbin/asterisk -rx core ...annels
Apr 28 09:59:26 HOSTNAME agent[20068]: 2020-04-28 09:59:26 UTC | CORE | ERROR | (pkg/collector/python/datadog_agent.go:....json'
Apr 28 09:59:26 HOSTNAME agent[20068]: 2020-04-28 09:59:26 UTC | CORE | ERROR | (pkg/collector/python/datadog_agent.go:...l/6455
Apr 28 09:59:34 HOSTNAME sudo[27736]: dd-agent : TTY=unknown ; PWD=/ ; USER=asterisk ; COMMAND=/sbin/asterisk -rx core ...annels
Hint: Some lines were ellipsized, use -l to show in full.

Additional environment details (Operating System, Cloud provider, etc):
Cloud provider: AWS
OS: CentOS 7
Datadog agent version: Agent 6.13.0 - Commit: df8e880 - Serialization version: 4.7.1 - Go version: go1.11.5

Steps to reproduce the issue:

1. Install Datadog agent
2. Setup filebeat integration
3. Tail -f /var/log/datadog-agent/agent.log

Describe the results you received:
The metrics are being sent to Datadog but for some reasons in the logs, there is a permission issue happening regarding filebeat registry not being allowed to be read.

2020-04-28 09:59:11 UTC | CORE | ERROR | (pkg/collector/python/datadog_agent.go:106 in LogMessage) | (filebeat.py:245) | Cannot read the registry log file at /var/lib/filebeat/registry/filebeat/data.json: [Errno 13] Permission denied: '/var/lib/filebeat/registry/filebeat/data.json'
2020-04-28 09:59:11 UTC | CORE | ERROR | (pkg/collector/python/datadog_agent.go:106 in LogMessage) | (filebeat.py:249) | You might be interesting in having a look at https://github.com/elastic/beats/pull/6455
2020-04-28 09:59:26 UTC | CORE | ERROR | (pkg/collector/python/datadog_agent.go:106 in LogMessage) | (filebeat.py:245) | Cannot read the registry log file at /var/lib/filebeat/registry/filebeat/data.json: [Errno 13] Permission denied: '/var/lib/filebeat/registry/filebeat/data.json'
2020-04-28 09:59:26 UTC | CORE | ERROR | (pkg/collector/python/datadog_agent.go:106 in LogMessage) | (filebeat.py:249) | You might be interesting in having a look at https://github.com/elastic/beats/pull/6455

Describe the results you expected:
The confusion part is filebeat metrics are being sent to Datadog despite of the fact the error is being seen in the logs. We've tried to set the file permission via our automation (SCM) but did not succeed and we've even used the Filebeat configuration registry permission in the yml file.

Additional information you deem important (e.g. issue happens only occasionally):
We've seen this in the following issue elastic/beats#6455 but could not get rid of the issue.

[hithwen]

cc @wk8 as integration maintainer

[wk8]

Sorry @hithwen I no longer use Filebeat in my fleet, so I don't have anywhere to test any of this.

[hithwen]

Hi @kalilou, integrations-extra repository is community driven. Since the integration maintaner is no longer maintaining it we have two ways forward:

1. Investigate the issue yourseld and submit a PR fixing the issue
2. Submit a request to Datadog support to adopt Filebeat as a core-integration in which case it will have official support including bugfixing.