Fleet Server package #544

aleksmaus · 2021-01-19T22:05:57Z

Bootstraps all the indexes needed for the Fleet Server once the integration is installed.

@ruflin @blakerouse
Since it creates only ILM policies and templates, will have another PR for the Fleet Server to make it resilient to the situation where the indexes do not exists yet.

Closing the initial draft #540 that @ruflin started.

elasticmachine · 2021-01-19T22:11:33Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Build Cause: Pull request #544 updated
- Start Time: 2021-01-20T16:19:22.222+0000
Duration: 7 min 11 sec
Commit: 202c4bc

Test stats 🧪

Test	Results
Failed	0
Passed	1
Skipped	0
Total	1

elasticmachine · 2021-01-20T06:12:00Z

Pinging @elastic/agent (Team:Agent)

ruflin · 2021-01-20T13:37:34Z

@ycombinator This is a package without a data_stream which I think is valid?

Update: change would need to happen here? https://github.com/elastic/package-spec/blob/master/versions/1/spec.yml#L25

ruflin · 2021-01-20T13:38:30Z

packages/fleet_server/elasticsearch/ilm_policy/.fleet-actions-results.json

@@ -0,0 +1,23 @@
+{


@aleksmaus Very interesting that it just works with the dot prefix. It will make it fun to develop as some OS hide these files :-)

you mean windowz? :-)

ruflin · 2021-01-20T13:39:38Z

packages/fleet_server/elasticsearch/index_template/.fleet-servers.json

+    "index_patterns": [
+        ".fleet-servers"
+    ],
+    "template": {


I wonder how restrictive we should be in these templates. Should we allow additional fields or not? If yes, keyword as default?

For upgrades would that be good to add to all the templates? Just incase that Fleet Server itself is upgraded before the package?

Probably. In the ideal scenario the template would of course always be first and we could even use strict mappings, but not the case for now.

ruflin · 2021-01-20T13:43:43Z

@aleksmaus What I suggest to keep moving here is adding a "fake" data_stream to pass validation, merge PR, promote it to snapshot so we can start using this for testing in Kibana / fleet-server.

I think we will have follow up discussions around more details in the template and ILM policy but this can come later.

blakerouse

Looking good, some questions and comments.

blakerouse · 2021-01-20T14:16:27Z

packages/fleet_server/manifest.yml

+    title: Fleet Server
+    description: Fleet Server setup
+    inputs:
+      - type: fleet_server


Change type to fleet-server. Fleet Server already expects the configuration file format to be:

inputs: - type: fleet-server host: 0.0.0.0 port: 8000

blakerouse · 2021-01-20T14:17:22Z

packages/fleet_server/manifest.yml

+    inputs:
+      - type: fleet_server
+        vars:
+          # TODO: Mainly for testing, probably no config options at first


I think we should do host binding and port binding.

Which are the fields host and port.

blakerouse · 2021-01-20T14:18:58Z

packages/fleet_server/manifest.yml

+            required: true
+            show_user: true
+            default:
+              - 127.0.0.1


Should default to 0.0.0.0.

blakerouse · 2021-01-20T14:19:36Z

packages/fleet_server/manifest.yml

+            default:
+              - 127.0.0.1
+        title: "Fleet Server parts"
+        description: "Hello World Fleet Server"


Should add port with default set to 8000 (as it's the default now). We should probably change that.

yes elastic/fleet-server#47

aleksmaus · 2021-01-20T16:17:16Z

ruflin

LGTM. It doesn't make it into the policy yet but lets get it in so we have all the assets.

ruflin · 2021-01-20T20:10:48Z

packages/fleet_server/manifest.yml

+    title: Fleet Server
+    description: Fleet Server setup
+    inputs:
+      - type: fleet-server


This does not end up in the policy currently as we don't use it in any of the data streams and have a template for it. Maybe a good use for our fake data stream?

We had a discussion around this for pure input packages. I'm trying to find the issue.

Ah, found it: elastic/kibana#83878 We can take inspiration from apm package: https://github.com/elastic/package-storage/blob/snapshot/packages/apm/0.1.0-dev.4/agent/input/template.yml.hbs

ycombinator · 2021-01-25T23:35:02Z

Hi @aleksmaus, @ruflin, I was on PTO all last week and am catching up on PRs today.

I see that this PR was merged with a fake data_stream folder to keep moving forward (👍 ). But now that we have elastic/package-spec#113 merged (along with elastic/elastic-package#232 and #563), could we get a follow up PR to remove the fake data_stream folder from the fleet_server package in the integrations repo? Thanks!

* Add fleet-server package * add ilm policy examples * fix template formatting * fix priority so it does not conflict * apply proper formatting * Functional indices boostrap * Make lint happier * Address review feedback. Add fake datastream, to pass linter * Better Fleet Server configuration description Co-authored-by: ruflin <spam@ruflin.com>

ruflin and others added 6 commits January 19, 2021 15:26

Add fleet-server package

7d2ce43

add ilm policy examples

7017a9d

fix template formatting

806f950

fix priority so it does not conflict

11cfca3

apply proper formatting

ada65c4

Functional indices boostrap

096579d

aleksmaus requested review from ruflin, nchaulet and blakerouse January 19, 2021 22:05

aleksmaus mentioned this pull request Jan 19, 2021

Add fleet-server package #540

Closed

Make lint happier

4dd2e9a

andresrc added the Team:Elastic-Agent Label for the Agent team label Jan 20, 2021

andresrc assigned aleksmaus Jan 20, 2021

ruflin reviewed Jan 20, 2021

View reviewed changes

blakerouse reviewed Jan 20, 2021

View reviewed changes

Address review feedback. Add fake datastream, to pass linter

565c231

This was referenced Jan 20, 2021

Need to be able to create a package without data_stream. elastic/package-spec#112

Closed

Make Fleet Server components resilient if the indices do not exist elastic/fleet-server#68

Merged

aleksmaus requested review from blakerouse and ruflin January 20, 2021 16:05

Better Fleet Server configuration description

202c4bc

ruflin approved these changes Jan 20, 2021

View reviewed changes

aleksmaus merged commit c8027b0 into elastic:master Jan 20, 2021

andrewkroh added Integration:fleet_server Fleet Server New Integration Issue or pull request for creating a new integration package. labels Aug 13, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fleet Server package #544

Fleet Server package #544

aleksmaus commented Jan 19, 2021

elasticmachine commented Jan 19, 2021 •

edited

Loading

Build stats

Test stats 🧪

elasticmachine commented Jan 20, 2021

ruflin commented Jan 20, 2021 •

edited

Loading

ruflin Jan 20, 2021

aleksmaus Jan 20, 2021

ruflin Jan 20, 2021

ruflin Jan 20, 2021

blakerouse Jan 20, 2021

ruflin Jan 20, 2021

ruflin commented Jan 20, 2021

blakerouse left a comment

blakerouse Jan 20, 2021

blakerouse Jan 20, 2021

blakerouse Jan 20, 2021

blakerouse Jan 20, 2021

ruflin Jan 20, 2021

aleksmaus commented Jan 20, 2021

ruflin left a comment

ruflin Jan 20, 2021

ruflin Jan 20, 2021

ycombinator commented Jan 25, 2021 •

edited

Loading

Fleet Server package #544

Fleet Server package #544

Conversation

aleksmaus commented Jan 19, 2021

elasticmachine commented Jan 19, 2021 • edited Loading

💚 Build Succeeded

Build stats

Test stats 🧪

elasticmachine commented Jan 20, 2021

ruflin commented Jan 20, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ruflin commented Jan 20, 2021

blakerouse left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

aleksmaus commented Jan 20, 2021

ruflin left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ycombinator commented Jan 25, 2021 • edited Loading

elasticmachine commented Jan 19, 2021 •

edited

Loading

ruflin commented Jan 20, 2021 •

edited

Loading

ycombinator commented Jan 25, 2021 •

edited

Loading