Logstash add ElasticsearchRefs

cmd/manager/main.go

@@ -876,6 +876,7 @@ func registerControllers(mgr manager.Manager, params operator.Parameters, access
 		{name: "AGENT-KB", registerFunc: associationctl.AddAgentKibana},
 		{name: "AGENT-FS", registerFunc: associationctl.AddAgentFleetServer},
 		{name: "EMS-ES", registerFunc: associationctl.AddMapsES},
+		{name: "LOGSTASH-ES", registerFunc: associationctl.AddLogstashES},
 		{name: "ES-MONITORING", registerFunc: associationctl.AddEsMonitoring},
 		{name: "KB-MONITORING", registerFunc: associationctl.AddKbMonitoring},
 		{name: "BEAT-MONITORING", registerFunc: associationctl.AddBeatMonitoring},

config/crds/v1/all-crds.yaml

@@ -9119,6 +9119,44 @@ spec:
               count:
                 format: int32
                 type: integer
+              elasticsearchRefs:
+                description: ElasticsearchRefs are references to Elasticsearch clusters
+                  running in the same Kubernetes cluster.
+                items:
+                  description: ElasticsearchCluster definition of type to define an
+                    Elasticsearch Cluster to be referred to in Logstash pipeline
+                  properties:
+                    clusterName:
+                      type: string
+                    name:
+                      description: Name of an existing Kubernetes object corresponding
+                        to an Elastic resource managed by ECK.
+                      type: string
+                    namespace:
+                      description: Namespace of the Kubernetes object. If empty, defaults
+                        to the current namespace.
+                      type: string
+                    secretName:
+                      description: 'SecretName is the name of an existing Kubernetes
+                        secret that contains connection information for associating
+                        an Elastic resource not managed by the operator. The referenced
+                        secret must contain the following: - `url`: the URL to reach
+                        the Elastic resource - `username`: the username of the user
+                        to be authenticated to the Elastic resource - `password`:
+                        the password of the user to be authenticated to the Elastic
+                        resource - `ca.crt`: the CA certificate in PEM format (optional).
+                        This field cannot be used in combination with the other fields
+                        name, namespace or serviceName.'
+                      type: string
+                    serviceName:
+                      description: ServiceName is the name of an existing Kubernetes
+                        service which is used to make requests to the referenced object.
+                        It has to be in the same namespace as the referenced resource.
+                        If left empty, the default HTTP service of the referenced
+                        resource is used.
+                      type: string
+                  type: object
+                type: array
               image:
                 description: Image is the Logstash Docker image to deploy. Version
                   and Type have to match the Logstash in the image.
@@ -9747,6 +9785,13 @@ spec:
               availableNodes:
                 format: int32
                 type: integer
+              elasticsearchAssociationsStatus:
+                additionalProperties:
+                  description: AssociationStatus is the status of an association resource.
+                  type: string
+                description: ElasticsearchAssociationStatus is the status of any auto-linking
+                  to Elasticsearch clusters.
+                type: object
               expectedNodes:
                 format: int32
                 type: integer

config/crds/v1/bases/logstash.k8s.elastic.co_logstashes.yaml

@@ -73,6 +73,44 @@ spec:
               count:
                 format: int32
                 type: integer
+              elasticsearchRefs:
+                description: ElasticsearchRefs are references to Elasticsearch clusters
+                  running in the same Kubernetes cluster.
+                items:
+                  description: ElasticsearchCluster definition of type to define an
+                    Elasticsearch Cluster to be referred to in Logstash pipeline
+                  properties:
+                    clusterName:
+                      type: string
+                    name:
+                      description: Name of an existing Kubernetes object corresponding
+                        to an Elastic resource managed by ECK.
+                      type: string
+                    namespace:
+                      description: Namespace of the Kubernetes object. If empty, defaults
+                        to the current namespace.
+                      type: string
+                    secretName:
+                      description: 'SecretName is the name of an existing Kubernetes
+                        secret that contains connection information for associating
+                        an Elastic resource not managed by the operator. The referenced
+                        secret must contain the following: - `url`: the URL to reach
+                        the Elastic resource - `username`: the username of the user
+                        to be authenticated to the Elastic resource - `password`:
+                        the password of the user to be authenticated to the Elastic
+                        resource - `ca.crt`: the CA certificate in PEM format (optional).
+                        This field cannot be used in combination with the other fields
+                        name, namespace or serviceName.'
+                      type: string
+                    serviceName:
+                      description: ServiceName is the name of an existing Kubernetes
+                        service which is used to make requests to the referenced object.
+                        It has to be in the same namespace as the referenced resource.
+                        If left empty, the default HTTP service of the referenced
+                        resource is used.
+                      type: string
+                  type: object
+                type: array
               image:
                 description: Image is the Logstash Docker image to deploy. Version
                   and Type have to match the Logstash in the image.
@@ -8112,6 +8150,13 @@ spec:
               availableNodes:
                 format: int32
                 type: integer
+              elasticsearchAssociationsStatus:
+                additionalProperties:
+                  description: AssociationStatus is the status of an association resource.
+                  type: string
+                description: ElasticsearchAssociationStatus is the status of any auto-linking
+                  to Elasticsearch clusters.
+                type: object
               expectedNodes:
                 format: int32
                 type: integer

config/samples/logstash/logstash_es.yaml

@@ -0,0 +1,37 @@
+apiVersion: elasticsearch.k8s.elastic.co/v1
+kind: Elasticsearch
+metadata:
+  name: elasticsearch-sample
+spec:
+  version: 8.7.0
+  nodeSets:
+    - name: default
+      count: 2
+      config:
+        node.store.allow_mmap: false
+---
+apiVersion: logstash.k8s.elastic.co/v1alpha1
+kind: Logstash
+metadata:
+  name: logstash-sample
+spec:
+  count: 1
+  version: 8.7.0
+  elasticsearchRefs:
+    - clusterName: production
+      name: elasticsearch-sample
+#      secretName: external-cloud-es-ref
+  pipelines:
+    - pipeline.id: main
+      config.string: |
+        input { exec { command => 'uptime' interval => 10 } } 
+        output { 
+          elasticsearch {
+            hosts => [ "${PRODUCTION_ES_HOSTS}" ]
+            ssl => true
+            cacert => "${PRODUCTION_ES_SSL_CERTIFICATE_AUTHORITY}"
+            user => "${PRODUCTION_ES_USER}"
+            password => "${PRODUCTION_ES_PASSWORD}"
+          } 
+        }
+---

deploy/eck-operator/charts/eck-operator-crds/templates/all-crds.yaml

@@ -9173,6 +9173,44 @@ spec:
               count:
                 format: int32
                 type: integer
+              elasticsearchRefs:
+                description: ElasticsearchRefs are references to Elasticsearch clusters
+                  running in the same Kubernetes cluster.
+                items:
+                  description: ElasticsearchCluster definition of type to define an
+                    Elasticsearch Cluster to be referred to in Logstash pipeline
+                  properties:
+                    clusterName:
+                      type: string
+                    name:
+                      description: Name of an existing Kubernetes object corresponding
+                        to an Elastic resource managed by ECK.
+                      type: string
+                    namespace:
+                      description: Namespace of the Kubernetes object. If empty, defaults
+                        to the current namespace.
+                      type: string
+                    secretName:
+                      description: 'SecretName is the name of an existing Kubernetes
+                        secret that contains connection information for associating
+                        an Elastic resource not managed by the operator. The referenced
+                        secret must contain the following: - `url`: the URL to reach
+                        the Elastic resource - `username`: the username of the user
+                        to be authenticated to the Elastic resource - `password`:
+                        the password of the user to be authenticated to the Elastic
+                        resource - `ca.crt`: the CA certificate in PEM format (optional).
+                        This field cannot be used in combination with the other fields
+                        name, namespace or serviceName.'
+                      type: string
+                    serviceName:
+                      description: ServiceName is the name of an existing Kubernetes
+                        service which is used to make requests to the referenced object.
+                        It has to be in the same namespace as the referenced resource.
+                        If left empty, the default HTTP service of the referenced
+                        resource is used.
+                      type: string
+                  type: object
+                type: array
               image:
                 description: Image is the Logstash Docker image to deploy. Version
                   and Type have to match the Logstash in the image.
@@ -9801,6 +9839,13 @@ spec:
               availableNodes:
                 format: int32
                 type: integer
+              elasticsearchAssociationsStatus:
+                additionalProperties:
+                  description: AssociationStatus is the status of an association resource.
+                  type: string
+                description: ElasticsearchAssociationStatus is the status of any auto-linking
+                  to Elasticsearch clusters.
+                type: object
               expectedNodes:
                 format: int32
                 type: integer

docs/reference/api-docs.asciidoc

@@ -611,6 +611,7 @@ ObjectSelector defines a reference to a Kubernetes object which can be an Elasti
 - xref:{anchor_prefix}-github-com-elastic-cloud-on-k8s-v2-pkg-apis-agent-v1alpha1-agentspec[$$AgentSpec$$]
 - xref:{anchor_prefix}-github-com-elastic-cloud-on-k8s-v2-pkg-apis-apm-v1-apmserverspec[$$ApmServerSpec$$]
 - xref:{anchor_prefix}-github-com-elastic-cloud-on-k8s-v2-pkg-apis-beat-v1beta1-beatspec[$$BeatSpec$$]
+- xref:{anchor_prefix}-github-com-elastic-cloud-on-k8s-v2-pkg-apis-logstash-v1alpha1-elasticsearchcluster[$$ElasticsearchCluster$$]
 - xref:{anchor_prefix}-github-com-elastic-cloud-on-k8s-v2-pkg-apis-enterprisesearch-v1-enterprisesearchspec[$$EnterpriseSearchSpec$$]
 - xref:{anchor_prefix}-github-com-elastic-cloud-on-k8s-v2-pkg-apis-enterprisesearch-v1beta1-enterprisesearchspec[$$EnterpriseSearchSpec$$]
 - xref:{anchor_prefix}-github-com-elastic-cloud-on-k8s-v2-pkg-apis-kibana-v1-kibanaspec[$$KibanaSpec$$]
@@ -1841,6 +1842,24 @@ Package v1alpha1 contains API Schema definitions for the logstash v1alpha1 API g
 
 
 
+[id="{anchor_prefix}-github-com-elastic-cloud-on-k8s-v2-pkg-apis-logstash-v1alpha1-elasticsearchcluster"]
+=== ElasticsearchCluster 
+
+ElasticsearchCluster definition of type to define an Elasticsearch Cluster to be referred to in Logstash pipeline
+
+.Appears In:
+****
+- xref:{anchor_prefix}-github-com-elastic-cloud-on-k8s-v2-pkg-apis-logstash-v1alpha1-logstashspec[$$LogstashSpec$$]
+****
+
+[cols="25a,75a", options="header"]
+|===
+| Field | Description
+| *`ObjectSelector`* __xref:{anchor_prefix}-github-com-elastic-cloud-on-k8s-v2-pkg-apis-common-v1-objectselector[$$ObjectSelector$$]__ | 
+| *`clusterName`* __string__ | 
+|===
+
+
 [id="{anchor_prefix}-github-com-elastic-cloud-on-k8s-v2-pkg-apis-logstash-v1alpha1-logstash"]
 === Logstash 
 
@@ -1895,6 +1914,7 @@ LogstashSpec defines the desired state of Logstash
 | *`version`* __string__ | Version of the Logstash.
 | *`count`* __integer__ | 
 | *`image`* __string__ | Image is the Logstash Docker image to deploy. Version and Type have to match the Logstash in the image.
+| *`elasticsearchRefs`* __xref:{anchor_prefix}-github-com-elastic-cloud-on-k8s-v2-pkg-apis-logstash-v1alpha1-elasticsearchcluster[$$ElasticsearchCluster$$] array__ | ElasticsearchRefs are references to Elasticsearch clusters running in the same Kubernetes cluster.
 | *`config`* __xref:{anchor_prefix}-github-com-elastic-cloud-on-k8s-v2-pkg-apis-common-v1-config[$$Config$$]__ | Config holds the Logstash configuration. At most one of [`Config`, `ConfigRef`] can be specified.
 | *`configRef`* __xref:{anchor_prefix}-github-com-elastic-cloud-on-k8s-v2-pkg-apis-common-v1-configsource[$$ConfigSource$$]__ | ConfigRef contains a reference to an existing Kubernetes Secret holding the Logstash configuration. Logstash settings must be specified as yaml, under a single "logstash.yml" entry. At most one of [`Config`, `ConfigRef`] can be specified.
 | *`pipelines`* __xref:{anchor_prefix}-github-com-elastic-cloud-on-k8s-v2-pkg-apis-common-v1-config[$$Config$$] array__ | Pipelines holds the Logstash Pipelines. At most one of [`Pipelines`, `PipelinesRef`] can be specified.