Add fleet-server package

[ruflin]

NOTE: This is not working at all yet. No review please. This is used to collaborate on the changes.

This package contains all the assets needed by fleet-server.

[ruflin]

Here is an example on how this should look: https://github.com/elastic/package-storage/blob/production/packages/endpoint/0.17.0/elasticsearch/index_template/metrics-metadata-current.json

I wonder what happens if we prefix this template with a .. Not even sure it is possible.

[elasticmachine]

💔 Build Failed

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Build Cause: Pull request #540 updated

• • Start Time: 2021-01-19T15:41:20.033+0000

• Duration: 4 min 19 sec

• Commit: ada65c4

Steps errors

Expand to view the steps failures

Check integration: fleet_server

• Took 0 min 0 sec . View more details on here
• Description: ../../build/elastic-package check -v

Log output

Expand to view the last 100 lines of log output

[2021-01-19T15:45:19.748Z] go: downloading github.com/elastic/go-elasticsearch/v7 v7.9.0
[2021-01-19T15:45:19.748Z] go: downloading github.com/go-git/go-git/v5 v5.1.0
[2021-01-19T15:45:19.748Z] go: downloading github.com/go-git/go-billy/v5 v5.0.0
[2021-01-19T15:45:19.748Z] go: downloading github.com/Masterminds/semver v1.5.0
[2021-01-19T15:45:19.748Z] go: downloading gopkg.in/yaml.v3 v3.0.0-20200615113413-eeeca48fe776
[2021-01-19T15:45:19.748Z] go: downloading golang.org/x/oauth2 v0.0.0-20200902213428-5d25da1a8d43
[2021-01-19T15:45:19.748Z] go: downloading github.com/google/go-github/v32 v32.1.0
[2021-01-19T15:45:19.748Z] go: downloading github.com/cespare/xxhash v1.1.0
[2021-01-19T15:45:19.748Z] go: downloading github.com/jedib0t/go-pretty v4.3.0+incompatible
[2021-01-19T15:45:19.748Z] go: downloading github.com/cespare/xxhash/v2 v2.1.1
[2021-01-19T15:45:19.748Z] go: downloading github.com/kballard/go-shellquote v0.0.0-20180428030007-95032a82bc51
[2021-01-19T15:45:19.748Z] go: downloading github.com/magefile/mage v1.10.0
[2021-01-19T15:45:19.748Z] go: downloading github.com/spf13/pflag v1.0.5
[2021-01-19T15:45:19.748Z] go: downloading gopkg.in/yaml.v2 v2.3.0
[2021-01-19T15:45:19.748Z] go: downloading github.com/mgutz/ansi v0.0.0-20200706080929-d51e80ef957d
[2021-01-19T15:45:19.748Z] go: downloading golang.org/x/net v0.0.0-20201021035429-f5854403a974
[2021-01-19T15:45:19.748Z] go: downloading golang.org/x/crypto v0.0.0-20200820211705-5c72a883971a
[2021-01-19T15:45:19.748Z] go: downloading github.com/sergi/go-diff v1.1.0
[2021-01-19T15:45:19.748Z] go: downloading github.com/mitchellh/go-homedir v1.1.0
[2021-01-19T15:45:19.748Z] go: downloading github.com/imdario/mergo v0.3.11
[2021-01-19T15:45:19.748Z] go: downloading github.com/mattn/go-colorable v0.1.7
[2021-01-19T15:45:19.748Z] go: downloading github.com/mattn/go-runewidth v0.0.9
[2021-01-19T15:45:19.748Z] go: downloading github.com/go-git/gcfg v1.5.0
[2021-01-19T15:45:20.011Z] go: downloading github.com/mattn/go-isatty v0.0.12
[2021-01-19T15:45:20.011Z] go: downloading github.com/kevinburke/ssh_config v0.0.0-20190725054713-01f96b0aa0cd
[2021-01-19T15:45:20.011Z] go: downloading github.com/aymerick/raymond v2.0.2+incompatible
[2021-01-19T15:45:20.011Z] go: downloading github.com/kylelemons/godebug v1.1.0
[2021-01-19T15:45:20.011Z] go: downloading golang.org/x/sys v0.0.0-20200930185726-fdedc70b468f
[2021-01-19T15:45:20.011Z] go: downloading github.com/google/go-querystring v1.0.0
[2021-01-19T15:45:20.011Z] go: downloading golang.org/x/text v0.3.3
[2021-01-19T15:45:20.011Z] go: downloading gopkg.in/warnings.v0 v0.1.2
[2021-01-19T15:45:20.011Z] go: downloading github.com/jbenet/go-context v0.0.0-20150711004518-d14ea06fba99
[2021-01-19T15:45:20.011Z] go: downloading github.com/rakyll/statik v0.1.7
[2021-01-19T15:45:20.011Z] go: downloading github.com/Masterminds/semver/v3 v3.1.0
[2021-01-19T15:45:20.011Z] go: downloading github.com/go-openapi/strfmt v0.19.6
[2021-01-19T15:45:20.011Z] go: downloading github.com/xanzy/ssh-agent v0.2.1
[2021-01-19T15:45:20.011Z] go: downloading go.mongodb.org/mongo-driver v1.0.3
[2021-01-19T15:45:20.011Z] go: downloading github.com/go-openapi/errors v0.19.2
[2021-01-19T15:45:20.011Z] go: downloading github.com/creasty/defaults v1.5.1
[2021-01-19T15:45:20.011Z] go: downloading github.com/xeipuuv/gojsonreference v0.0.0-20180127040603-bd5ef7bd5415
[2021-01-19T15:45:20.011Z] go: downloading github.com/xeipuuv/gojsonpointer v0.0.0-20190905194746-02993c407bfb
[2021-01-19T15:45:20.011Z] go: downloading github.com/asaskevich/govalidator v0.0.0-20190424111038-f61b66f89f4a
[2021-01-19T15:45:20.272Z] go: downloading github.com/mitchellh/mapstructure v1.1.2
[2021-01-19T15:45:20.272Z] go: downloading github.com/xeipuuv/gojsonschema v1.2.0
[2021-01-19T15:45:20.272Z] go: downloading github.com/emirpasic/gods v1.12.0
[2021-01-19T15:45:20.533Z] go: downloading github.com/go-stack/stack v1.8.0
[2021-01-19T15:45:32.816Z] Running in /var/lib/jenkins/workspace/gest-manager_integrations_PR-540/src/github.com/elastic/integrations/packages/fleet_server
[2021-01-19T15:45:33.104Z] + ../../build/elastic-package check -v
[2021-01-19T15:45:33.104Z] elastic-package has been installed.
[2021-01-19T15:45:33.104Z] 2021/01/19 15:45:32 DEBUG Enable verbose logging
[2021-01-19T15:45:33.104Z] Format the package
[2021-01-19T15:45:33.104Z] Done
[2021-01-19T15:45:33.104Z] Lint the package
[2021-01-19T15:45:33.104Z] 2021/01/19 15:45:32 DEBUG Check if README.md is up-to-date
[2021-01-19T15:45:33.104Z] 2021/01/19 15:45:32 DEBUG Generate README.md file (package: /var/lib/jenkins/workspace/gest-manager_integrations_PR-540/src/github.com/elastic/integrations/packages/fleet_server)
[2021-01-19T15:45:33.104Z] 2021/01/19 15:45:32 DEBUG README file is static, can't be generated from the template file
[2021-01-19T15:45:33.104Z] Error: checking package failed: linting package failed: found 1 validation error:
[2021-01-19T15:45:33.104Z]    1. expecting to find [data_stream] folder in folder [/var/lib/jenkins/workspace/gest-manager_integrations_PR-540/src/github.com/elastic/integrations/packages/fleet_server]
[2021-01-19T15:45:33.104Z] 
[2021-01-19T15:45:33.130Z] Running in /var/lib/jenkins/workspace/gest-manager_integrations_PR-540/src/github.com/elastic/integrations
[2021-01-19T15:45:33.144Z] Archiving artifacts
[2021-01-19T15:45:33.297Z] ‘build/test-results/*.xml’ doesn’t match anything: ‘build’ exists but not ‘build/test-results/*.xml’
[2021-01-19T15:45:33.298Z] No artifacts found that match the file pattern "build/test-results/*.xml". Configuration error?
[2021-01-19T15:45:33.306Z] Recording test results
[2021-01-19T15:45:33.437Z] None of the test reports contained any result
[2021-01-19T15:45:33.437Z] [Checks API] No suitable checks publisher found.
[2021-01-19T15:45:33.722Z] + build/elastic-package stack dump -v --output build/elastic-stack-dump/fleet_server
[2021-01-19T15:45:33.722Z] 2021/01/19 15:45:33 DEBUG Enable verbose logging
[2021-01-19T15:45:33.722Z] 2021/01/19 15:45:33 DEBUG Dump Elastic stack data
[2021-01-19T15:45:33.722Z] 2021/01/19 15:45:33 DEBUG Dump stack logs
[2021-01-19T15:45:33.722Z] 2021/01/19 15:45:33 DEBUG Recreate the output location (path: build/elastic-stack-dump/fleet_server)
[2021-01-19T15:45:33.722Z] 2021/01/19 15:45:33 DEBUG Dump stack logs for elasticsearch
[2021-01-19T15:45:33.722Z] 2021/01/19 15:45:33 DEBUG running command: /usr/local/bin/docker-compose -f /var/lib/jenkins/workspace/gest-manager_integrations_PR-540/.elastic-package/stack/snapshot.yml -p elastic-package-stack logs elasticsearch
[2021-01-19T15:45:34.662Z] 2021/01/19 15:45:34 DEBUG Dump stack logs for elastic-agent
[2021-01-19T15:45:34.662Z] 2021/01/19 15:45:34 DEBUG running command: /usr/local/bin/docker-compose -f /var/lib/jenkins/workspace/gest-manager_integrations_PR-540/.elastic-package/stack/snapshot.yml -p elastic-package-stack logs elastic-agent
[2021-01-19T15:45:35.233Z] 2021/01/19 15:45:35 DEBUG Dump stack logs for kibana
[2021-01-19T15:45:35.233Z] 2021/01/19 15:45:35 DEBUG running command: /usr/local/bin/docker-compose -f /var/lib/jenkins/workspace/gest-manager_integrations_PR-540/.elastic-package/stack/snapshot.yml -p elastic-package-stack logs kibana
[2021-01-19T15:45:35.802Z] 2021/01/19 15:45:35 DEBUG Dump stack logs for package-registry
[2021-01-19T15:45:35.802Z] 2021/01/19 15:45:35 DEBUG running command: /usr/local/bin/docker-compose -f /var/lib/jenkins/workspace/gest-manager_integrations_PR-540/.elastic-package/stack/snapshot.yml -p elastic-package-stack logs package-registry
[2021-01-19T15:45:36.742Z] Path to stack dump: build/elastic-stack-dump/fleet_server
[2021-01-19T15:45:36.742Z] Done
[2021-01-19T15:45:36.758Z] Archiving artifacts
[2021-01-19T15:45:37.055Z] + build/elastic-package stack down -v
[2021-01-19T15:45:37.055Z] 2021/01/19 15:45:36 DEBUG Enable verbose logging
[2021-01-19T15:45:37.055Z] Take down the Elastic stack
[2021-01-19T15:45:37.055Z] 2021/01/19 15:45:36 DEBUG running command: /usr/local/bin/docker-compose -f /var/lib/jenkins/workspace/gest-manager_integrations_PR-540/.elastic-package/stack/snapshot.yml -p elastic-package-stack down
[2021-01-19T15:45:37.625Z] Removing network elastic-package-stack_default
[2021-01-19T15:45:37.625Z] Network elastic-package-stack_default not found.
[2021-01-19T15:45:37.625Z] Done
[2021-01-19T15:45:37.668Z] Failed in branch fleet_server
[2021-01-19T15:45:37.717Z] Stage "Update Package Storage" skipped due to earlier failure(s)
[2021-01-19T15:45:37.894Z] Running on worker-1244230 in /var/lib/jenkins/workspace/gest-manager_integrations_PR-540
[2021-01-19T15:45:37.967Z] [INFO] getVaultSecret: Getting secrets
[2021-01-19T15:45:38.040Z] Masking supported pattern matches of $VAULT_ADDR or $VAULT_ROLE_ID or $VAULT_SECRET_ID
[2021-01-19T15:45:39.948Z] + chmod 755 generate-build-data.sh
[2021-01-19T15:45:39.948Z] + ./generate-build-data.sh https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Ingest-manager/integrations/PR-540/ https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Ingest-manager/integrations/PR-540/runs/5 FAILURE 258531
[2021-01-19T15:45:39.948Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Ingest-manager/integrations/PR-540/runs/5/steps/?limit=10000 -o steps-info.json
[2021-01-19T15:45:40.647Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Ingest-manager/integrations/PR-540/runs/5/tests/?status=FAILED -o tests-errors.json
[2021-01-19T15:45:41.346Z] Retry 1/3 exited 22, retrying in 1 seconds...
[2021-01-19T15:45:42.793Z] Retry 2/3 exited 22, retrying in 2 seconds...

[ruflin]

Indication on the templates we need:

{"level":"info","name":".fleet-enrollment-api-keys","time":"Jan 19 15:21:37.411723","message":"Create template"}
{"level":"info","name":".fleet-policies","time":"Jan 19 15:21:37.648501","message":"Create template"}
{"level":"info","name":".fleet-policies-leader","time":"Jan 19 15:21:37.784996","message":"Create template"}
{"level":"info","name":".fleet-servers","time":"Jan 19 15:21:37.921092","message":"Create template"}
{"level":"info","name":".fleet-actions","time":"Jan 19 15:21:38.094132","message":"Create template"}
{"level":"info","policy":".fleet-actions-results-ilm-policy","time":"Jan 19 15:21:38.239051","message":"ILM policy is not found. Create a new one."}
{"level":"info","name":".fleet-actions-results","time":"Jan 19 15:21:38.286755","message":"Create template"}
{"level":"info","name":".fleet-agents","time":"Jan 19 15:21:38.588401","message":"Create template"}

One of the questions will be if we can make the templates for it private.

@aleksmaus Which of the above are indices and which are data streams?

[aleksmaus]

@ruflin we only have one datastream for now .fleet-actions-results
and btw we can't use this on elasticsearch 7.10 since the datastream are not allowed to start with ".", only 7.11

[ruflin]

@aleksmaus This package will only be compatible with ^7.12 so should not be an issue. I suggest for the first iteration we put all templates globally and don't use the data_stream directories? This does not mean we can't use data-streams.

[aleksmaus]

Closing this draft. Replaced with #544.