Elastic Stack Up not working on Windows or Ubuntu WSL

[nicpenning]

I am wondering if something is amiss with the docker compose file.

When trying to follow these docs: https://github.com/elastic/integrations/blob/main/docs/developer_workflow_design_build_test_integration.md

And trying to run this command: elastic-package stack up -v -d

It just gets stuck at Build 0%

But trying to run the commands manually to start the stack up I am seeing this error:

service "fleet-server" has neither an image nor a build context specified: invalid compose project

Is it possible fleet was not included in the stack or should I be doing something to prevent fleet from being deployed to simply walk through building an integration?

I will post more details when I'm at a computer.

Tasks

Beta Give feedback

1. Add some mention to having a working docker installation to the Getting Started guide.
2. Throw an error instead of hanging forever when building images if docker is not available.

[jsoriano]

Hey Nic, thanks for reporting your issue.

What versions of elastic-package, docker and docker-compose are you using?

[nicpenning]

Looks like docker-compose: v2.18.1 and elastic-package: b57cc2d (which I believe is v0.80.0, but unsure).

Method for getting elastic-package was:

git clone https://github.com/elastic/elastic-package.git
make build

Welcome to Ubuntu 22.04.2 LTS (GNU/Linux 5.10.102.1-microsoft-standard-WSL2 x86_64)

 * Documentation:  https://help.ubuntu.com
 * Management:     https://landscape.canonical.com
 * Support:        https://ubuntu.com/advantage

napsta@el33t-b00k-1:~$ docker-compose --version
Docker Compose version v2.18.1
napsta@el33t-b00k-1:~$ cd elastic/elastic-package/
napsta@el33t-b00k-1:~/elastic/elastic-package$ ./elastic-package version
elastic-package version-hash b57cc2d (build time: 2023-05-28T17:50:42-05:00)

This is my issue:
(Building 0.0s is just stuck and never progresses and you can't control-C out of it, you have to end the process/terminal).

napsta@el33t-b00k-1:~/elastic/elastic-package$ ./elastic-package stack up -v
2023/05/31 21:35:32 DEBUG Enable verbose logging
2023/05/31 21:35:32 DEBUG Distribution built without a version tag, can't determine release chronology. Please consider using official releases at https://github.com/elastic/elastic-package/releases
Boot up the Elastic stack
Using profile /home/napsta/.elastic-package/profiles/default.
Remember to load stack environment variables using 'eval "$(elastic-package stack shellinit)"'.
Elasticsearch host: https://127.0.0.1:9200
Kibana host: https://127.0.0.1:5601
Username: elastic
Password: changeme
Local package-registry will serve packages from these sources:
- Proxy to https://epr.elastic.co
2023/05/31 21:35:32 DEBUG running command: /usr/local/bin/docker-compose version --short
2023/05/31 21:35:32 DEBUG Determined Docker Compose version: 2.18.1
2023/05/31 21:35:32 DEBUG running command: /usr/local/bin/docker-compose -f /home/napsta/.elastic-package/profiles/default/stack/snapshot.yml -p elastic-package-stack build
[+] Building 0.0s (0/0)

When running /usr/local/bin/docker-compose -f /home/napsta/.elastic-package/profiles/default/stack/snapshot.yml -p elastic-package-stack build this is the result that made me come to my conclusion with Fleet possibly being the issue:

napsta@el33t-b00k-1:~$ /usr/local/bin/docker-compose -f /home/napsta/.elastic-package/profiles/default/stack/snapshot.yml -p elastic-package-stack build
WARN[0000] The "ELASTIC_AGENT_IMAGE_REF" variable is not set. Defaulting to a blank string.
WARN[0000] The "PROFILE_NAME" variable is not set. Defaulting to a blank string.
WARN[0000] The "PROFILE_NAME" variable is not set. Defaulting to a blank string.
WARN[0000] The "ELASTIC_AGENT_IMAGE_REF" variable is not set. Defaulting to a blank string.
WARN[0000] The "ELASTICSEARCH_IMAGE_REF" variable is not set. Defaulting to a blank string.
WARN[0000] The "KIBANA_IMAGE_REF" variable is not set. Defaulting to a blank string.
service "fleet-server" has neither an image nor a build context specified: invalid compose project

[nicpenning]

Turns out I didn't have docker installed, so I installed it but no change:

napsta@el33t-b00k-1:~$ docker version
Client: Docker Engine - Community
 Version:           24.0.2
 API version:       1.43
 Go version:        go1.20.4
 Git commit:        cb74dfc
 Built:             Thu May 25 21:51:00 2023
 OS/Arch:           linux/amd64
 Context:           default
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?

[nicpenning]

Instead of pulling from master, I used the 0.80 release to be sure I have a stable one:

napsta@el33t-b00k-1:~$ elastic-package version
could not determine benchmark reports folder: locating build directory failed: can't create new build directory: package can be only built inside of a Git repository (.git folder is used as reference point)elastic-package v0.80.0 version-hash 151c71d (build time: 2023-05-15T04:22:47-05:00)

And the issue persists.

[jsoriano]

Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?

Umm, if this doesn't work I don't think docker-compose can work, please review your docker installation.

could not determine benchmark reports folder: locating build directory failed: can't create new build directory: package can be only built inside of a Git repository (.git folder is used as reference point)elastic-package v0.80.0 version-hash 151c71d (build time: 2023-05-15T04:22:47-05:00)

This looks like an actual issue, I can reproduce it too in Linux. I think this is related to recent changes.

[nicpenning]

Got it, I will review my docker install and get the docker daemon operational.

[jsoriano]

The error message should be fixed by #1294.

[nicpenning]

I got the docker service running but no change in functionality.

[nicpenning]

What I am seeing now:

napsta@el33t-b00k-1:~$ /usr/local/bin/docker-compose -f /home/napsta/.elastic-package/profiles/default/stack/snapshot.yml -p elastic-package-stack build
WARN[0000] The "PROFILE_NAME" variable is not set. Defaulting to a blank string.
WARN[0000] The "PROFILE_NAME" variable is not set. Defaulting to a blank string.
WARN[0000] The "KIBANA_IMAGE_REF" variable is not set. Defaulting to a blank string.
WARN[0000] The "ELASTIC_AGENT_IMAGE_REF" variable is not set. Defaulting to a blank string.
WARN[0000] The "ELASTIC_AGENT_IMAGE_REF" variable is not set. Defaulting to a blank string.
WARN[0000] The "ELASTICSEARCH_IMAGE_REF" variable is not set. Defaulting to a blank string.
service "elastic-agent" has neither an image nor a build context specified: invalid compose project

[jsoriano]

What I am seeing now:

napsta@el33t-b00k-1:~$ /usr/local/bin/docker-compose -f /home/napsta/.elastic-package/profiles/default/stack/snapshot.yml -p elastic-package-stack build
WARN[0000] The "PROFILE_NAME" variable is not set. Defaulting to a blank string.
WARN[0000] The "PROFILE_NAME" variable is not set. Defaulting to a blank string.
WARN[0000] The "KIBANA_IMAGE_REF" variable is not set. Defaulting to a blank string.
WARN[0000] The "ELASTIC_AGENT_IMAGE_REF" variable is not set. Defaulting to a blank string.
WARN[0000] The "ELASTIC_AGENT_IMAGE_REF" variable is not set. Defaulting to a blank string.
WARN[0000] The "ELASTICSEARCH_IMAGE_REF" variable is not set. Defaulting to a blank string.
service "elastic-agent" has neither an image nor a build context specified: invalid compose project

This docker compose configuration is not intended to be used on its own, you should use elastic-package stack up.

[nicpenning]

Of course, this was what I see if I ran the commands from what is displayed during the elastic-package run. (Good to know that probably won't work on its own).

I still see the Building 0% not progressing.

Are there any specific logs to see why this is hanging?

[nicpenning]

I found the issue. I was not running as admin/root to execute the application. So when in Linux/Ubuntu WSL2, you must run with sudo:

root@el33t-b00k-1:/home/napsta/elastic/elastic-package# ./elastic-package stack up -d -v
2023/06/07 21:27:24 DEBUG Enable verbose logging
2023/06/07 21:27:24 DEBUG Distribution built without a version tag, can't determine release chronology. Please consider using official releases at https://github.com/elastic/elastic-package/releases
Boot up the Elastic stack
Using profile /root/.elastic-package/profiles/default.
Remember to load stack environment variables using 'eval "$(elastic-package stack shellinit)"'.
Elasticsearch host: https://127.0.0.1:9200
Kibana host: https://127.0.0.1:5601
Username: elastic
Password: changeme
Local package-registry will serve packages from these sources:
- Proxy to https://epr.elastic.co
2023/06/07 21:27:24 DEBUG running command: /usr/local/bin/docker-compose version --short
2023/06/07 21:27:24 DEBUG Determined Docker Compose version: 2.18.1
2023/06/07 21:27:24 DEBUG running command: /usr/local/bin/docker-compose -f /root/.elastic-package/profiles/default/stack/snapshot.yml -p elastic-package-stack build
[+] Building 9.8s (8/8) FINISHED

Do we think it is worth noting in the documentation that this could require sudo?

(https://github.com/elastic/integrations/blob/main/docs/developer_workflow_design_build_test_integration.md#steps)

[jsoriano]

@nicpenning this is part of the installation steps of docker. It is mentioned several times in the docs https://docs.docker.com/engine/install/ubuntu/.

It is usual that after installing docker in Linux, you add the users you want to allow to use docker to the docker group. See https://docs.docker.com/engine/install/linux-postinstall/#manage-docker-as-a-non-root-user

elastic-package stack requires a working docker installation. It could be actually a good idea to add this to the getting started section: https://github.com/elastic/elastic-package#getting-started

But what worries me more is that elastic-package stack just hanged and didn't report any error, we should definitely handle this, I will add a couple of tasks to this issue.

Thanks!

[nicpenning]

Ah yes, you are correct on the docker install. I grazed past that, so my apologies!

It would be awesome to show an error though, that would have saved me a couple weeks time :) I suppose you can tell I am not a docler expert, nor did I pay close enough attention to dockers docs.

Let me know if there is anything else you need from me. Thanks for the help!!