Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Enable beats stack monitoring configuration #5878

Merged
merged 69 commits into from
Sep 26, 2022
Merged

Enable beats stack monitoring configuration #5878

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
69 commits
Select commit Hold shift + click to select a range
c6b7287
Add stack monitoring for Beats
naemono Jun 22, 2022
165decd
When calling GetAssociations, also add the monitoring objects.
naemono Jun 29, 2022
c31c29d
Only access info after chedcking error
naemono Jun 30, 2022
de1cddc
Add debugging
naemono Jun 30, 2022
e14849d
Removing debugging logging statements
naemono Jul 11, 2022
23de080
Handling external, and internal monitoring refs the same.
naemono Jul 12, 2022
ec13b90
Fix unit test.
naemono Jul 15, 2022
a21aa92
Fix some issues with errorf formatting
naemono Jul 15, 2022
6101d4c
Merge branch 'main' into 5563-enable-beats-monitoring
naemono Jul 15, 2022
8991412
Add missing '/v2' to association package.
naemono Jul 18, 2022
02b0489
Optimize ssl/ca bits when creating monitoring configuration
naemono Jul 18, 2022
e517733
Add copyright to beat/common/monitoring.go
naemono Jul 18, 2022
42829d9
remove duplicative checks in beat e2e test
naemono Jul 18, 2022
94ebcf3
remove unused monitoringAssociationConnectionInfo
naemono Jul 18, 2022
77a205e
re-running generation, since wording changed in crds
naemono Jul 18, 2022
0ea5dfd
Avoid extra var in range in pkg/controller/beat/common/pod_test.go
naemono Jul 26, 2022
9831c38
Use new func in existing checks package to return beats monitoring st…
naemono Jul 26, 2022
b155d9d
Enabling webhook validation for beats stack monitoring.
naemono Jul 27, 2022
25702b7
Add unit tests for beat validation.
naemono Jul 27, 2022
2779620
Minor change in pod_test.go.
naemono Jul 27, 2022
2994d5f
Removing unneeded BeatsStep function.
naemono Jul 29, 2022
2c19f81
Moving to utilizing sidecars for beat monitoring instead of internal …
naemono Aug 2, 2022
4ae5d27
revert config/recipes changes
naemono Aug 2, 2022
06a3396
Fix early return in beats validation so that invalid monitoring refs …
naemono Aug 3, 2022
a175029
Ensure that beats http.port setting finds it's way all the way to the…
naemono Aug 3, 2022
ccf1323
Add missing metricbeat template yaml file.
naemono Aug 3, 2022
b9a0c6a
Adding missing header to new file.
naemono Aug 3, 2022
f21ff9c
running make generate crds
naemono Aug 4, 2022
7dc4bc9
Updating api docs
naemono Aug 4, 2022
d14893a
Update CRD documentation
naemono Aug 12, 2022
d901228
Correcting spelling
naemono Aug 15, 2022
3ae14e4
Add UUID to stack monitoring *beat configuration.
naemono Aug 15, 2022
268944b
Ensure metricbeat sidecar for beat has cluster_uuid.
naemono Aug 16, 2022
d5bb9b8
Add beats stackmon tests
naemono Aug 16, 2022
b57902e
Add newlines to yaml files
naemono Aug 16, 2022
c86c85e
Update beats stackmon validation to use same common validation
naemono Aug 16, 2022
025a30f
Update public docs.
naemono Aug 16, 2022
97125e5
Update api docs
naemono Aug 16, 2022
9bf98fe
Fix naming of static error.
naemono Aug 16, 2022
710575d
fixing ghost changes causing ci failures
naemono Aug 16, 2022
6840e41
Add changed yaml files vi automation thatss breaking ci
naemono Aug 16, 2022
734130d
adjust all-crds in opeartor chart
naemono Aug 16, 2022
1da13e0
Merge branch 'main' into 5563-enable-beats-monitoring
naemono Aug 17, 2022
0b5581f
Apply https://github.com/elastic/cloud-on-k8s/pull/5955 to try and ge…
naemono Aug 17, 2022
aea3d88
Add missing context to beat buildPodTemplate test.
naemono Aug 17, 2022
c710cd5
Revert change to deployer/runner/gke.go as it was merged.
naemono Aug 17, 2022
3a85f16
Merge branch 'main' into 5563-enable-beats-monitoring
naemono Aug 17, 2022
45fc97a
Move Monitoring objects to commonv1.
naemono Aug 22, 2022
23ae156
Add Keyed field to struct to pass linter.
naemono Aug 23, 2022
c646180
Merge branch 'main' into 5563-enable-beats-monitoring
naemono Aug 23, 2022
2975684
Fix moved struct from esv1 -> commonv1 in stackmon tests
naemono Aug 23, 2022
2182d40
Update api docs
naemono Aug 23, 2022
ca26311
Update test for new ssl verification mode for beats
naemono Aug 23, 2022
0e37cbb
wip
naemono Aug 27, 2022
fe83cee
Use unix sockets for beat metrics gather when stack monitoring is ena…
naemono Sep 1, 2022
d4ebba2
Merge branch 'main' into 5563-enable-beats-monitoring
naemono Sep 1, 2022
44a3d1e
Regenerate api docs
naemono Sep 1, 2022
8864c1c
Docs and CRD wording updates
naemono Sep 6, 2022
42535c0
Attempt remove specific path in filebeat.
naemono Sep 12, 2022
53e7dd1
Always uppercase stack monitoring in docs.
naemono Sep 12, 2022
a14eb20
running make docs, and go vet
naemono Sep 12, 2022
e3f965d
Do not log when ES cluster UUID isn't yet available, just emit event.
naemono Sep 14, 2022
ed04231
add shared volume for filebeat logs consumption
naemono Sep 15, 2022
0f99ccb
correct spelling in beat/*/stackmon.go
naemono Sep 15, 2022
f3bf6af
Ensure Beat doesn't contain '-e' startup option when log stack monito…
naemono Sep 15, 2022
bebdf6e
Fix beat configuration test for stackmon being enabled.
naemono Sep 19, 2022
52cd829
Ensure "-e" argument to *beat is removed if provided in podtemplate c…
naemono Sep 21, 2022
0cd89e2
Add documentation for the '-e' filebeat option being automatically re…
naemono Sep 21, 2022
0658e37
uppercase Filebeat
naemono Sep 22, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions cmd/manager/main.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -824,6 +824,7 @@ func registerControllers(mgr manager.Manager, params operator.Parameters, access
{name: "EMS-ES", registerFunc: associationctl.AddMapsES},
{name: "ES-MONITORING", registerFunc: associationctl.AddEsMonitoring},
{name: "KB-MONITORING", registerFunc: associationctl.AddKbMonitoring},
{name: "BEAT-MONITORING", registerFunc: associationctl.AddBeatMonitoring},
}

for _, c := range assocControllers {
Expand Down
118 changes: 114 additions & 4 deletions config/crds/v1/all-crds.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2356,6 +2356,107 @@ spec:
is used.
type: string
type: object
monitoring:
description: Monitoring enables you to collect and ship logs and metrics
for this Beat. Metricbeat and/or Filebeat sidecars are configured
and send monitoring data to an Elasticsearch monitoring cluster
running in the same Kubernetes cluster.
properties:
logs:
description: Logs holds references to Elasticsearch clusters which
receive log data from an associated resource.
properties:
elasticsearchRefs:
description: ElasticsearchRefs is a reference to a list of
monitoring Elasticsearch clusters running in the same Kubernetes
cluster. Due to existing limitations, only a single Elasticsearch
cluster is currently supported.
items:
description: ObjectSelector defines a reference to a Kubernetes
object which can be an Elastic resource managed by the
operator or a Secret describing an external Elastic resource
not managed by the operator.
properties:
name:
description: Name of an existing Kubernetes object corresponding
to an Elastic resource managed by ECK.
type: string
namespace:
description: Namespace of the Kubernetes object. If
empty, defaults to the current namespace.
type: string
secretName:
description: 'SecretName is the name of an existing
Kubernetes secret that contains connection information
for associating an Elastic resource not managed by
the operator. The referenced secret must contain the
following: - `url`: the URL to reach the Elastic resource
- `username`: the username of the user to be authenticated
to the Elastic resource - `password`: the password
of the user to be authenticated to the Elastic resource
- `ca.crt`: the CA certificate in PEM format (optional).
This field cannot be used in combination with the
other fields name, namespace or serviceName.'
type: string
serviceName:
description: ServiceName is the name of an existing
Kubernetes service which is used to make requests
to the referenced object. It has to be in the same
namespace as the referenced resource. If left empty,
the default HTTP service of the referenced resource
is used.
type: string
type: object
type: array
type: object
metrics:
description: Metrics holds references to Elasticsearch clusters
which receive monitoring data from this resource.
properties:
elasticsearchRefs:
description: ElasticsearchRefs is a reference to a list of
monitoring Elasticsearch clusters running in the same Kubernetes
cluster. Due to existing limitations, only a single Elasticsearch
cluster is currently supported.
items:
description: ObjectSelector defines a reference to a Kubernetes
object which can be an Elastic resource managed by the
operator or a Secret describing an external Elastic resource
not managed by the operator.
properties:
name:
description: Name of an existing Kubernetes object corresponding
to an Elastic resource managed by ECK.
type: string
namespace:
description: Namespace of the Kubernetes object. If
empty, defaults to the current namespace.
type: string
secretName:
description: 'SecretName is the name of an existing
Kubernetes secret that contains connection information
for associating an Elastic resource not managed by
the operator. The referenced secret must contain the
following: - `url`: the URL to reach the Elastic resource
- `username`: the username of the user to be authenticated
to the Elastic resource - `password`: the password
of the user to be authenticated to the Elastic resource
- `ca.crt`: the CA certificate in PEM format (optional).
This field cannot be used in combination with the
other fields name, namespace or serviceName.'
type: string
serviceName:
description: ServiceName is the name of an existing
Kubernetes service which is used to make requests
to the referenced object. It has to be in the same
namespace as the referenced resource. If left empty,
the default HTTP service of the referenced resource
is used.
type: string
type: object
type: array
type: object
type: object
revisionHistoryLimit:
description: RevisionHistoryLimit is the number of revisions to retain
to allow rollback in the underlying DaemonSet or Deployment.
Expand Down Expand Up @@ -2437,6 +2538,15 @@ spec:
kibanaAssociationStatus:
description: AssociationStatus is the status of an association resource.
type: string
monitoringAssociationStatus:
additionalProperties:
description: AssociationStatus is the status of an association resource.
type: string
description: AssociationStatusMap is the map of association's namespaced
name string to its AssociationStatus. For resources that have a
single Association of a given type (for ex. single ES reference),
this map contains a single entry.
type: object
observedGeneration:
description: ObservedGeneration represents the .metadata.generation
that the status is based upon. It corresponds to the metadata generation,
Expand Down Expand Up @@ -3560,7 +3670,7 @@ spec:
properties:
logs:
description: Logs holds references to Elasticsearch clusters which
receive log data from this Elasticsearch cluster.
receive log data from an associated resource.
properties:
elasticsearchRefs:
description: ElasticsearchRefs is a reference to a list of
Expand Down Expand Up @@ -3607,7 +3717,7 @@ spec:
type: object
metrics:
description: Metrics holds references to Elasticsearch clusters
which receive monitoring data from this Elasticsearch cluster.
which receive monitoring data from this resource.
properties:
elasticsearchRefs:
description: ElasticsearchRefs is a reference to a list of
Expand Down Expand Up @@ -7563,7 +7673,7 @@ spec:
properties:
logs:
description: Logs holds references to Elasticsearch clusters which
will receive log data from this Kibana.
receive log data from an associated resource.
properties:
elasticsearchRefs:
description: ElasticsearchRefs is a reference to a list of
Expand Down Expand Up @@ -7610,7 +7720,7 @@ spec:
type: object
metrics:
description: Metrics holds references to Elasticsearch clusters
which will receive monitoring data from this Kibana.
which receive monitoring data from this resource.
properties:
elasticsearchRefs:
description: ElasticsearchRefs is a reference to a list of
Expand Down
110 changes: 110 additions & 0 deletions config/crds/v1/bases/beat.k8s.elastic.co_beats.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15316,6 +15316,107 @@ spec:
is used.
type: string
type: object
monitoring:
description: Monitoring enables you to collect and ship logs and metrics
for this Beat. Metricbeat and/or Filebeat sidecars are configured
and send monitoring data to an Elasticsearch monitoring cluster
running in the same Kubernetes cluster.
properties:
logs:
description: Logs holds references to Elasticsearch clusters which
receive log data from an associated resource.
properties:
elasticsearchRefs:
description: ElasticsearchRefs is a reference to a list of
monitoring Elasticsearch clusters running in the same Kubernetes
cluster. Due to existing limitations, only a single Elasticsearch
cluster is currently supported.
items:
description: ObjectSelector defines a reference to a Kubernetes
object which can be an Elastic resource managed by the
operator or a Secret describing an external Elastic resource
not managed by the operator.
properties:
name:
description: Name of an existing Kubernetes object corresponding
to an Elastic resource managed by ECK.
type: string
namespace:
description: Namespace of the Kubernetes object. If
empty, defaults to the current namespace.
type: string
secretName:
description: 'SecretName is the name of an existing
Kubernetes secret that contains connection information
for associating an Elastic resource not managed by
the operator. The referenced secret must contain the
following: - `url`: the URL to reach the Elastic resource
- `username`: the username of the user to be authenticated
to the Elastic resource - `password`: the password
of the user to be authenticated to the Elastic resource
- `ca.crt`: the CA certificate in PEM format (optional).
This field cannot be used in combination with the
other fields name, namespace or serviceName.'
type: string
serviceName:
description: ServiceName is the name of an existing
Kubernetes service which is used to make requests
to the referenced object. It has to be in the same
namespace as the referenced resource. If left empty,
the default HTTP service of the referenced resource
is used.
type: string
type: object
type: array
type: object
metrics:
description: Metrics holds references to Elasticsearch clusters
which receive monitoring data from this resource.
properties:
elasticsearchRefs:
description: ElasticsearchRefs is a reference to a list of
monitoring Elasticsearch clusters running in the same Kubernetes
cluster. Due to existing limitations, only a single Elasticsearch
cluster is currently supported.
items:
description: ObjectSelector defines a reference to a Kubernetes
object which can be an Elastic resource managed by the
operator or a Secret describing an external Elastic resource
not managed by the operator.
properties:
name:
description: Name of an existing Kubernetes object corresponding
to an Elastic resource managed by ECK.
type: string
namespace:
description: Namespace of the Kubernetes object. If
empty, defaults to the current namespace.
type: string
secretName:
description: 'SecretName is the name of an existing
Kubernetes secret that contains connection information
for associating an Elastic resource not managed by
the operator. The referenced secret must contain the
following: - `url`: the URL to reach the Elastic resource
- `username`: the username of the user to be authenticated
to the Elastic resource - `password`: the password
of the user to be authenticated to the Elastic resource
- `ca.crt`: the CA certificate in PEM format (optional).
This field cannot be used in combination with the
other fields name, namespace or serviceName.'
type: string
serviceName:
description: ServiceName is the name of an existing
Kubernetes service which is used to make requests
to the referenced object. It has to be in the same
namespace as the referenced resource. If left empty,
the default HTTP service of the referenced resource
is used.
type: string
type: object
type: array
type: object
type: object
revisionHistoryLimit:
description: RevisionHistoryLimit is the number of revisions to retain
to allow rollback in the underlying DaemonSet or Deployment.
Expand Down Expand Up @@ -15397,6 +15498,15 @@ spec:
kibanaAssociationStatus:
description: AssociationStatus is the status of an association resource.
type: string
monitoringAssociationStatus:
additionalProperties:
description: AssociationStatus is the status of an association resource.
type: string
description: AssociationStatusMap is the map of association's namespaced
name string to its AssociationStatus. For resources that have a
single Association of a given type (for ex. single ES reference),
this map contains a single entry.
type: object
observedGeneration:
description: ObservedGeneration represents the .metadata.generation
that the status is based upon. It corresponds to the metadata generation,
Expand Down
4 changes: 2 additions & 2 deletions config/crds/v1/bases/elasticsearch.k8s.elastic.co_elasticsearches.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -509,7 +509,7 @@ spec:
properties:
logs:
description: Logs holds references to Elasticsearch clusters which
receive log data from this Elasticsearch cluster.
receive log data from an associated resource.
properties:
elasticsearchRefs:
description: ElasticsearchRefs is a reference to a list of
Expand Down Expand Up @@ -556,7 +556,7 @@ spec:
type: object
metrics:
description: Metrics holds references to Elasticsearch clusters
which receive monitoring data from this Elasticsearch cluster.
which receive monitoring data from this resource.
properties:
elasticsearchRefs:
description: ElasticsearchRefs is a reference to a list of
Expand Down
4 changes: 2 additions & 2 deletions config/crds/v1/bases/kibana.k8s.elastic.co_kibanas.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -550,7 +550,7 @@ spec:
properties:
logs:
description: Logs holds references to Elasticsearch clusters which
will receive log data from this Kibana.
receive log data from an associated resource.
properties:
elasticsearchRefs:
description: ElasticsearchRefs is a reference to a list of
Expand Down Expand Up @@ -597,7 +597,7 @@ spec:
type: object
metrics:
description: Metrics holds references to Elasticsearch clusters
which will receive monitoring data from this Kibana.
which receive monitoring data from this resource.
properties:
elasticsearchRefs:
description: ElasticsearchRefs is a reference to a list of
Expand Down
Loading