Container logs are not written in elastic-package stack dump using docker-compose v2

[mrodm]

Relates #1116
Blocks #1188

Logs cannot be retrieved when docker-compose v2 is used in the system.

Using docker-compose v1:

 $ docker-compose version --short
1.25.5

 $ elastic-package stack up -v -d --version 8.7.0
 ...

 $ elastic-package stack dump
2023/05/05 13:31:25 ERROR can't copy internal logs (service: elastic-agent): docker copy failed: could not copy files from the container (stderr="Error response from daemon: Could not find the file /usr/share/elastic-agent/state/data/logs/default in container elastic-package-stack_elastic-agent_1\n"): exit status 1
2023/05/05 13:31:27 ERROR can't copy internal logs (service: fleet-server): docker copy failed: could not copy files from the container (stderr="Error response from daemon: Could not find the file /usr/share/elastic-agent/state/data/logs/default in container elastic-package-stack_fleet-server_1\n"): exit status 1
Path to stack dump: elastic-stack-dump
Done

 $ ls -l elastic-stack-dump/logs
total 816
-rw-r--r-- 1 mariorodriguez mariorodriguez 147046 may  5 13:31 elastic-agent.log
-rw-r--r-- 1 mariorodriguez mariorodriguez 325907 may  5 13:31 elasticsearch.log
-rw-r--r-- 1 mariorodriguez mariorodriguez 286197 may  5 13:31 fleet-server.log
-rw-r--r-- 1 mariorodriguez mariorodriguez  15648 may  5 13:31 kibana.log
-rw-r--r-- 1 mariorodriguez mariorodriguez  54461 may  5 13:31 package-registry.log

 $ elastic-package stack down

Usign docker-compose v2:

 $ docker-compose version --short
2.17.2

 $ elastic-package stack up -v -d --version 8.7.0

 $ elastic-package stack dump
2023/05/05 13:38:39 ERROR can't fetch service logs (service: elasticsearch): running command failed: exit status 15
2023/05/05 13:38:39 ERROR can't fetch service logs (service: elastic-agent): running command failed: exit status 15
2023/05/05 13:38:39 ERROR can't copy internal logs (service: elastic-agent): docker copy failed: could not copy files from the container (stderr="Error response from daemon: Could not find the file /usr/share/elastic-agent/state/data/logs/default in container elastic-package-stack-elastic-agent-1\n"): exit status 1
2023/05/05 13:38:39 ERROR can't fetch service logs (service: fleet-server): running command failed: exit status 15
2023/05/05 13:38:39 ERROR can't copy internal logs (service: fleet-server): docker copy failed: could not copy files from the container (stderr="Error response from daemon: Could not find the file /usr/share/elastic-agent/state/data/logs/default in container elastic-package-stack-fleet-server-1\n"): exit status 1
2023/05/05 13:38:39 ERROR can't fetch service logs (service: kibana): running command failed: exit status 15
2023/05/05 13:38:39 ERROR can't fetch service logs (service: package-registry): running command failed: exit status 15
Path to stack dump: elastic-stack-dump
Done

 $ ls -l elastic-stack-dump/logs
total 0

 $ elastic-package stack down

For both versions, there is a common error related to some internal logs from fleet-server and elastic-agent:

• Path used by elastic-package does not exist in the container: /usr/share/elastic-agent/state/data/logs/default
• Logs are written to: /usr/share/elastic-agent/state/data/logs/elastic-agent-*.ndjson

[mrodm]

Internal logs that elastic-package tries to copy looks like it changed in 8.6.

Previously, (checked with 8.5.0) there was a default folder with filebeat/metricbeat/... and other logs there:

 $ tree -l elastic-stack-dump/logs/elastic-agent-internal/
elastic-stack-dump/logs/elastic-agent-internal/
├── default
│   ├── filebeat-20230505-1.ndjson
│   ├── filebeat-20230505-2.ndjson
│   ├── filebeat-20230505-3.ndjson
│   ├── filebeat-20230505.ndjson
│   ├── metricbeat-20230505-1.ndjson
│   └── metricbeat-20230505.ndjson
└── elastic-agent-20230505.ndjson

1 directory, 7 files

Since Elastic-agent >=8.6.0 , there is no default folder and it is just logs in /usr/share/elastic-agent/state/data/logs/:

 $ tree -l elastic-stack-dump/logs/elastic-agent-internal/
elastic-stack-dump/logs/elastic-agent-internal/
└── elastic-agent-20230505.ndjson

0 directories, 1 file

According to the release notes of elastic-agent 8.6.0

• Collect logs from sub-processes via stdout and stderr and write them to a single, unified Elastic Agent log file #1702 #221