From 0d9f03e1d62691622a6345981d34c891b561072a Mon Sep 17 00:00:00 2001 From: Pier-Hugues Pellerin Date: Mon, 10 Feb 2020 10:00:55 -0500 Subject: [PATCH] [Agent] Allow CA cert pinning on the Elasticsearch output or any code that user tlscommon.TLSConfig builder. (#16019) * Add a sha256 pin for the CA Certificate When multiples CA are presents on the system we cannot ensure that a specific one was used to validates the chains exposer by the server. This PRs adds a `ca_sha256` option to the `tlscommon.TLSConfig` that is used by all the code that has to create a TCP client with TLS support. When the option is set, it will hook a new callback in the validation chains that will inspect the verified and validated chains by Go to ensure that a lets a certificate in the chains match the provided sha256. Usage example for the Elasticsearch output. ``` output.elasticsearch: hosts: [127.0.0.1:9200] ssl.ca_sha256: ``` You can generate the pin using the **openssl** binary with the following command: ``` openssl x509 -in ca.crt -pubkey -noout | openssl pkey -pubin -outform der | openssl dgst -sha256 -binary | openssl enc -base64 ``` OpenSSL's [documentation](https://www.openssl.org/docs/manmaster/man1/dgst.html) You will need to start Elasticsearch with the following options ```yaml xpack.security.enabled: true indices.id_field_data.enabled: true xpack.license.self_generated.type: trial xpack.security.http.ssl.enabled: true xpack.security.http.ssl.key: /etc/pki/localhost/localhost.key" xpack.security.http.ssl.certificate: /etc/pki/localhost/localhost.crt" xpack.security.http.ssl.certificate_authorities: /etc/pki/ca/ca.crt" ``` This pull request also include a new service in the docker-compose.yml that will start a new Elasticsearch server with TLS and security configured. --- CHANGELOG.next.asciidoc | 1 + auditbeat/auditbeat.reference.yml | 12 + filebeat/filebeat.reference.yml | 12 + heartbeat/heartbeat.reference.yml | 12 + journalbeat/journalbeat.reference.yml | 12 + libbeat/_meta/config.reference.yml.tmpl | 12 + .../common/transport/tlscommon/ca_pinning.go | 73 ++++ .../transport/tlscommon/ca_pinning_test.go | 396 ++++++++++++++++++ libbeat/common/transport/tlscommon/config.go | 2 + .../common/transport/tlscommon/tls_config.go | 32 +- libbeat/docker-compose.yml | 33 ++ libbeat/docs/shared-ssl-config.asciidoc | 14 + libbeat/scripts/Makefile | 4 + libbeat/tests/system/beat/beat.py | 10 + libbeat/tests/system/test_ca_pinning.py | 81 ++++ metricbeat/metricbeat.reference.yml | 12 + packetbeat/packetbeat.reference.yml | 12 + .../docker/elasticsearch/pki/ca/ca.crt | 20 + .../docker/elasticsearch/pki/ca/ca.key | 27 ++ .../pki/elasticsearchssl/elasticsearchssl.crt | 19 + .../pki/elasticsearchssl/elasticsearchssl.key | 27 ++ .../docker/elasticsearch/pki/generate_pki.sh | 4 + winlogbeat/winlogbeat.reference.yml | 12 + x-pack/auditbeat/auditbeat.reference.yml | 12 + x-pack/filebeat/filebeat.reference.yml | 12 + .../functionbeat/functionbeat.reference.yml | 12 + x-pack/metricbeat/metricbeat.reference.yml | 12 + x-pack/winlogbeat/winlogbeat.reference.yml | 12 + 28 files changed, 889 insertions(+), 10 deletions(-) create mode 100644 libbeat/common/transport/tlscommon/ca_pinning.go create mode 100644 libbeat/common/transport/tlscommon/ca_pinning_test.go create mode 100644 libbeat/tests/system/test_ca_pinning.py create mode 100644 testing/environments/docker/elasticsearch/pki/ca/ca.crt create mode 100644 testing/environments/docker/elasticsearch/pki/ca/ca.key create mode 100644 testing/environments/docker/elasticsearch/pki/elasticsearchssl/elasticsearchssl.crt create mode 100644 testing/environments/docker/elasticsearch/pki/elasticsearchssl/elasticsearchssl.key create mode 100755 testing/environments/docker/elasticsearch/pki/generate_pki.sh diff --git a/CHANGELOG.next.asciidoc b/CHANGELOG.next.asciidoc index d74a98e25c2..a4034b65e6e 100644 --- a/CHANGELOG.next.asciidoc +++ b/CHANGELOG.next.asciidoc @@ -55,6 +55,7 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d - Fix issue where default go logger is not discarded when either * or stdout is selected. {issue}10251[10251] {pull}15708[15708] - Upgrade go-ucfg to latest v0.8.1. {pull}15937{15937} - Fix index names for indexing not always guaranteed to be lower case. {pull}16081[16081] +- Add `ssl.ca_sha256` option to the supported TLS option, this allow to check that a specific certificate is used as part of the verified chain. {issue}15717[15717] *Auditbeat* diff --git a/auditbeat/auditbeat.reference.yml b/auditbeat/auditbeat.reference.yml index b285e5a2ed7..2fe13ec5e5d 100644 --- a/auditbeat/auditbeat.reference.yml +++ b/auditbeat/auditbeat.reference.yml @@ -511,6 +511,12 @@ output.elasticsearch: # never, once, and freely. Default is never. #ssl.renegotiation: never + # Configure a pin that can be used to do extra validation of the verified certificate chain, + # this allow you to ensure that a specific certificate is used to validate the chain of trust. + # + # The pin is a base64 encoded string of the SHA-256 fingerprint. + #ssl.ca_sha256 + #----------------------------- Logstash output --------------------------------- #output.logstash: # Boolean flag to enable or disable the output module. @@ -604,6 +610,12 @@ output.elasticsearch: # never, once, and freely. Default is never. #ssl.renegotiation: never + # Configure a pin that can be used to do extra validation of the verified certificate chain, + # this allow you to ensure that a specific certificate is used to validate the chain of trust. + # + # The pin is a base64 encoded string of the SHA-256 fingerprint. + #ssl.ca_sha256 + # The number of times to retry publishing an event after a publishing failure. # After the specified number of retries, the events are typically dropped. # Some Beats, such as Filebeat and Winlogbeat, ignore the max_retries setting diff --git a/filebeat/filebeat.reference.yml b/filebeat/filebeat.reference.yml index e653f5de622..50961c98a80 100644 --- a/filebeat/filebeat.reference.yml +++ b/filebeat/filebeat.reference.yml @@ -1208,6 +1208,12 @@ output.elasticsearch: # never, once, and freely. Default is never. #ssl.renegotiation: never + # Configure a pin that can be used to do extra validation of the verified certificate chain, + # this allow you to ensure that a specific certificate is used to validate the chain of trust. + # + # The pin is a base64 encoded string of the SHA-256 fingerprint. + #ssl.ca_sha256 + #----------------------------- Logstash output --------------------------------- #output.logstash: # Boolean flag to enable or disable the output module. @@ -1301,6 +1307,12 @@ output.elasticsearch: # never, once, and freely. Default is never. #ssl.renegotiation: never + # Configure a pin that can be used to do extra validation of the verified certificate chain, + # this allow you to ensure that a specific certificate is used to validate the chain of trust. + # + # The pin is a base64 encoded string of the SHA-256 fingerprint. + #ssl.ca_sha256 + # The number of times to retry publishing an event after a publishing failure. # After the specified number of retries, the events are typically dropped. # Some Beats, such as Filebeat and Winlogbeat, ignore the max_retries setting diff --git a/heartbeat/heartbeat.reference.yml b/heartbeat/heartbeat.reference.yml index af10819b81a..60941544931 100644 --- a/heartbeat/heartbeat.reference.yml +++ b/heartbeat/heartbeat.reference.yml @@ -655,6 +655,12 @@ output.elasticsearch: # never, once, and freely. Default is never. #ssl.renegotiation: never + # Configure a pin that can be used to do extra validation of the verified certificate chain, + # this allow you to ensure that a specific certificate is used to validate the chain of trust. + # + # The pin is a base64 encoded string of the SHA-256 fingerprint. + #ssl.ca_sha256 + #----------------------------- Logstash output --------------------------------- #output.logstash: # Boolean flag to enable or disable the output module. @@ -748,6 +754,12 @@ output.elasticsearch: # never, once, and freely. Default is never. #ssl.renegotiation: never + # Configure a pin that can be used to do extra validation of the verified certificate chain, + # this allow you to ensure that a specific certificate is used to validate the chain of trust. + # + # The pin is a base64 encoded string of the SHA-256 fingerprint. + #ssl.ca_sha256 + # The number of times to retry publishing an event after a publishing failure. # After the specified number of retries, the events are typically dropped. # Some Beats, such as Filebeat and Winlogbeat, ignore the max_retries setting diff --git a/journalbeat/journalbeat.reference.yml b/journalbeat/journalbeat.reference.yml index 9270a862925..949df01a29a 100644 --- a/journalbeat/journalbeat.reference.yml +++ b/journalbeat/journalbeat.reference.yml @@ -449,6 +449,12 @@ output.elasticsearch: # never, once, and freely. Default is never. #ssl.renegotiation: never + # Configure a pin that can be used to do extra validation of the verified certificate chain, + # this allow you to ensure that a specific certificate is used to validate the chain of trust. + # + # The pin is a base64 encoded string of the SHA-256 fingerprint. + #ssl.ca_sha256 + #----------------------------- Logstash output --------------------------------- #output.logstash: # Boolean flag to enable or disable the output module. @@ -542,6 +548,12 @@ output.elasticsearch: # never, once, and freely. Default is never. #ssl.renegotiation: never + # Configure a pin that can be used to do extra validation of the verified certificate chain, + # this allow you to ensure that a specific certificate is used to validate the chain of trust. + # + # The pin is a base64 encoded string of the SHA-256 fingerprint. + #ssl.ca_sha256 + # The number of times to retry publishing an event after a publishing failure. # After the specified number of retries, the events are typically dropped. # Some Beats, such as Filebeat and Winlogbeat, ignore the max_retries setting diff --git a/libbeat/_meta/config.reference.yml.tmpl b/libbeat/_meta/config.reference.yml.tmpl index ede5934fd24..a949cd43327 100644 --- a/libbeat/_meta/config.reference.yml.tmpl +++ b/libbeat/_meta/config.reference.yml.tmpl @@ -391,6 +391,12 @@ output.elasticsearch: # Configure what types of renegotiation are supported. Valid options are # never, once, and freely. Default is never. #ssl.renegotiation: never + + # Configure a pin that can be used to do extra validation of the verified certificate chain, + # this allow you to ensure that a specific certificate is used to validate the chain of trust. + # + # The pin is a base64 encoded string of the SHA-256 fingerprint. + #ssl.ca_sha256 {{if not .ExcludeLogstash}} #----------------------------- Logstash output --------------------------------- #output.logstash: @@ -485,6 +491,12 @@ output.elasticsearch: # never, once, and freely. Default is never. #ssl.renegotiation: never + # Configure a pin that can be used to do extra validation of the verified certificate chain, + # this allow you to ensure that a specific certificate is used to validate the chain of trust. + # + # The pin is a base64 encoded string of the SHA-256 fingerprint. + #ssl.ca_sha256 + # The number of times to retry publishing an event after a publishing failure. # After the specified number of retries, the events are typically dropped. # Some Beats, such as Filebeat and Winlogbeat, ignore the max_retries setting diff --git a/libbeat/common/transport/tlscommon/ca_pinning.go b/libbeat/common/transport/tlscommon/ca_pinning.go new file mode 100644 index 00000000000..d83bf533d13 --- /dev/null +++ b/libbeat/common/transport/tlscommon/ca_pinning.go @@ -0,0 +1,73 @@ +// Licensed to Elasticsearch B.V. under one or more contributor +// license agreements. See the NOTICE file distributed with +// this work for additional information regarding copyright +// ownership. Elasticsearch B.V. licenses this file to you under +// the Apache License, Version 2.0 (the "License"); you may +// not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package tlscommon + +import ( + "crypto/sha256" + "crypto/x509" + "encoding/base64" + + "github.com/pkg/errors" +) + +// ErrCAPinMissmatch is returned when no pin is matched in the verified chain. +var ErrCAPinMissmatch = errors.New("provided CA certificate pins doesn't match any of the certificate authorities used to validate the certificate") + +type pins []string + +func (p pins) Matches(candidate string) bool { + for _, pin := range p { + if pin == candidate { + return true + } + } + return false +} + +// verifyPeerCertFunc is a callback defined on the tls.Config struct that will called when a +// TLS connection is used. +type verifyPeerCertFunc func([][]byte, [][]*x509.Certificate) error + +// MakeCAPinCallback loops through the verified chains and will try to match the certificates pin. +// +// NOTE: Defining a PIN to check certificates is not a replacement for the normal TLS validations it's +// an additional validation. In fact if you set `InsecureSkipVerify` to true and a PIN, the +// verifiedChains variable will be empty and the added validation will fail. +func MakeCAPinCallback(hashes pins) func([][]byte, [][]*x509.Certificate) error { + return func(_ [][]byte, verifiedChains [][]*x509.Certificate) error { + // The chain of trust has been already established before the call to the VerifyPeerCertificate + // function, after we go through the chain to make sure we have at least a certificate certificate + // that match the provided pin. + for _, chain := range verifiedChains { + for _, certificate := range chain { + h := Fingerprint(certificate) + if hashes.Matches(h) { + return nil + } + } + } + + return ErrCAPinMissmatch + } +} + +// Fingerprint takes a certificate and create a hash of the DER encoded public key. +func Fingerprint(certificate *x509.Certificate) string { + hash := sha256.Sum256(certificate.RawSubjectPublicKeyInfo) + return base64.StdEncoding.EncodeToString(hash[:]) +} diff --git a/libbeat/common/transport/tlscommon/ca_pinning_test.go b/libbeat/common/transport/tlscommon/ca_pinning_test.go new file mode 100644 index 00000000000..a9f95e5c6ac --- /dev/null +++ b/libbeat/common/transport/tlscommon/ca_pinning_test.go @@ -0,0 +1,396 @@ +// Licensed to Elasticsearch B.V. under one or more contributor +// license agreements. See the NOTICE file distributed with +// this work for additional information regarding copyright +// ownership. Elasticsearch B.V. licenses this file to you under +// the Apache License, Version 2.0 (the "License"); you may +// not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package tlscommon + +import ( + "bytes" + "crypto/rand" + "crypto/rsa" + "crypto/tls" + "crypto/x509" + "crypto/x509/pkix" + "io/ioutil" + "math/big" + "net" + "net/http" + "strings" + "testing" + "time" + + "github.com/pkg/errors" + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" + + "github.com/elastic/beats/libbeat/common" +) + +var ser int64 = 1 + +func TestCAPinning(t *testing.T) { + host := "127.0.0.1" + + t.Run("when the ca_sha256 field is not defined we use normal certificate validation", + func(t *testing.T) { + cfg := common.MustNewConfigFrom(map[string]interface{}{ + "certificate_authorities": []string{"ca_test.pem"}, + }) + + config := &Config{} + err := cfg.Unpack(config) + require.NoError(t, err) + + tlsCfg, err := LoadTLSConfig(config) + require.NoError(t, err) + + tls := tlsCfg.BuildModuleConfig(host) + require.Nil(t, tls.VerifyPeerCertificate) + }) + + t.Run("when the ca_sha256 field is defined we use CA cert pinning", func(t *testing.T) { + cfg := common.MustNewConfigFrom(map[string]interface{}{ + "ca_sha256": "hello", + }) + + config := &Config{} + err := cfg.Unpack(config) + require.NoError(t, err) + + tlsCfg, err := LoadTLSConfig(config) + require.NoError(t, err) + + tls := tlsCfg.BuildModuleConfig(host) + require.NotNil(t, tls.VerifyPeerCertificate) + }) + + t.Run("CA Root -> Certificate and we have the CA root pin", func(t *testing.T) { + msg := []byte("OK received message") + + ca, err := genCA() + require.NoError(t, err) + + serverCert, err := genSignedCert(ca, x509.KeyUsageDigitalSignature, false) + require.NoError(t, err) + + mux := http.NewServeMux() + mux.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) { + w.WriteHeader(http.StatusOK) + w.Write(msg) + }) + + // Select a random available port from the OS. + addr := "localhost:0" + + l, err := net.Listen("tcp", addr) + + server := &http.Server{ + Handler: mux, + TLSConfig: &tls.Config{ + Certificates: []tls.Certificate{ + serverCert, + }, + }, + } + + // Start server and shut it down when the tests are over. + go server.ServeTLS(l, "", "") + defer l.Close() + + // Root CA Pool + require.NoError(t, err) + rootCAs := x509.NewCertPool() + rootCAs.AddCert(ca.Leaf) + + // Get the pin of the RootCA. + pin := Fingerprint(ca.Leaf) + + tlsC := &TLSConfig{ + RootCAs: rootCAs, + CASha256: []string{pin}, + } + + config := tlsC.BuildModuleConfig("localhost") + hostToConnect := l.Addr().String() + + transport := &http.Transport{ + TLSClientConfig: config, + } + + client := &http.Client{Transport: transport} + + port := strings.TrimPrefix(hostToConnect, "127.0.0.1:") + + req, err := http.NewRequest("GET", "https://localhost:"+port, nil) + require.NoError(t, err) + resp, err := client.Do(req) + require.NoError(t, err) + content, err := ioutil.ReadAll(resp.Body) + require.NoError(t, err) + + assert.True(t, bytes.Equal(msg, content)) + + // 1. create key-pair + // 2. create pin + // 3. start server + // 4. Connect + // 5. Check wrong key do not work + // 6. Check good key work + // 7. check plain text fails to work. + }) + + t.Run("CA Root -> Intermediate -> Certificate and we receive the CA Root Pin", func(t *testing.T) { + msg := []byte("OK received message") + + ca, err := genCA() + require.NoError(t, err) + + intermediate, err := genSignedCert(ca, x509.KeyUsageDigitalSignature|x509.KeyUsageCertSign, true) + require.NoError(t, err) + + serverCert, err := genSignedCert(intermediate, x509.KeyUsageDigitalSignature, false) + require.NoError(t, err) + + mux := http.NewServeMux() + mux.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) { + w.WriteHeader(http.StatusOK) + w.Write(msg) + }) + + // Select a random available port from the OS. + addr := "localhost:0" + + l, err := net.Listen("tcp", addr) + require.NoError(t, err) + + // Server needs to provides the chain of trust, so server certificate + intermediate. + // RootCAs will trust the intermediate, intermediate will trust the server. + serverCert.Certificate = append(serverCert.Certificate, intermediate.Certificate...) + + server := &http.Server{ + Handler: mux, + TLSConfig: &tls.Config{ + Certificates: []tls.Certificate{ + serverCert, + }, + }, + } + + // Start server and shut it down when the tests are over. + go server.ServeTLS(l, "", "") + defer l.Close() + + // Root CA Pool + rootCAs := x509.NewCertPool() + rootCAs.AddCert(ca.Leaf) + + // Get the pin of the RootCA. + pin := Fingerprint(ca.Leaf) + + tlsC := &TLSConfig{ + RootCAs: rootCAs, + CASha256: []string{pin}, + } + + config := tlsC.BuildModuleConfig("localhost") + hostToConnect := l.Addr().String() + + transport := &http.Transport{ + TLSClientConfig: config, + } + + client := &http.Client{Transport: transport} + + port := strings.TrimPrefix(hostToConnect, "127.0.0.1:") + + req, err := http.NewRequest("GET", "https://localhost:"+port, nil) + require.NoError(t, err) + resp, err := client.Do(req) + require.NoError(t, err) + content, err := ioutil.ReadAll(resp.Body) + require.NoError(t, err) + + assert.True(t, bytes.Equal(msg, content)) + }) + + t.Run("When we have the wrong pin we refuse to connect", func(t *testing.T) { + msg := []byte("OK received message") + + ca, err := genCA() + require.NoError(t, err) + + intermediate, err := genSignedCert(ca, x509.KeyUsageDigitalSignature|x509.KeyUsageCertSign, true) + require.NoError(t, err) + + serverCert, err := genSignedCert(intermediate, x509.KeyUsageDigitalSignature, false) + require.NoError(t, err) + + mux := http.NewServeMux() + mux.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) { + w.WriteHeader(http.StatusOK) + w.Write(msg) + }) + + // Select a random available port from the OS. + addr := "localhost:0" + + l, err := net.Listen("tcp", addr) + require.NoError(t, err) + + // Server needs to provides the chain of trust, so server certificate + intermediate. + // RootCAs will trust the intermediate, intermediate will trust the server. + serverCert.Certificate = append(serverCert.Certificate, intermediate.Certificate...) + + server := &http.Server{ + Handler: mux, + TLSConfig: &tls.Config{ + Certificates: []tls.Certificate{ + serverCert, + }, + }, + } + + // Start server and shut it down when the tests are over. + go server.ServeTLS(l, "", "") + defer l.Close() + + // Root CA Pool + rootCAs := x509.NewCertPool() + rootCAs.AddCert(ca.Leaf) + + // Get the pin of the RootCA. + pin := "wrong-pin" + + tlsC := &TLSConfig{ + RootCAs: rootCAs, + CASha256: []string{pin}, + } + + config := tlsC.BuildModuleConfig("localhost") + hostToConnect := l.Addr().String() + + transport := &http.Transport{ + TLSClientConfig: config, + } + + client := &http.Client{Transport: transport} + + port := strings.TrimPrefix(hostToConnect, "127.0.0.1:") + + req, err := http.NewRequest("GET", "https://localhost:"+port, nil) + require.NoError(t, err) + _, err = client.Do(req) + require.Error(t, err) + }) +} + +func genCA() (tls.Certificate, error) { + ca := &x509.Certificate{ + SerialNumber: serial(), + Subject: pkix.Name{ + CommonName: "localhost", + Organization: []string{"TESTING"}, + Country: []string{"CANADA"}, + Province: []string{"QUEBEC"}, + Locality: []string{"MONTREAL"}, + StreetAddress: []string{"testing road"}, + PostalCode: []string{"HOH OHO"}, + }, + NotBefore: time.Now(), + NotAfter: time.Now().Add(1 * time.Hour), + IsCA: true, + ExtKeyUsage: []x509.ExtKeyUsage{x509.ExtKeyUsageClientAuth, x509.ExtKeyUsageServerAuth}, + KeyUsage: x509.KeyUsageDigitalSignature | x509.KeyUsageCertSign, + BasicConstraintsValid: true, + } + + caKey, err := rsa.GenerateKey(rand.Reader, 2048) // less secure key for quicker testing. + if err != nil { + return tls.Certificate{}, errors.Wrap(err, "fail to generate RSA key") + } + + caBytes, err := x509.CreateCertificate(rand.Reader, ca, ca, &caKey.PublicKey, caKey) + if err != nil { + return tls.Certificate{}, errors.Wrap(err, "fail to create certificate") + } + + leaf, err := x509.ParseCertificate(caBytes) + if err != nil { + return tls.Certificate{}, errors.Wrap(err, "fail to parse certificate") + } + + return tls.Certificate{ + Certificate: [][]byte{caBytes}, + PrivateKey: caKey, + Leaf: leaf, + }, nil +} + +// genSignedCert generates a CA and KeyPair and remove the need to depends on code of agent. +func genSignedCert(ca tls.Certificate, keyUsage x509.KeyUsage, isCA bool) (tls.Certificate, error) { + // Create another Cert/key + cert := &x509.Certificate{ + SerialNumber: big.NewInt(2000), + Subject: pkix.Name{ + CommonName: "localhost", + Organization: []string{"TESTING"}, + Country: []string{"CANADA"}, + Province: []string{"QUEBEC"}, + Locality: []string{"MONTREAL"}, + StreetAddress: []string{"testing road"}, + PostalCode: []string{"HOH OHO"}, + }, + NotBefore: time.Now(), + NotAfter: time.Now().Add(1 * time.Hour), + IsCA: isCA, + ExtKeyUsage: []x509.ExtKeyUsage{x509.ExtKeyUsageClientAuth, x509.ExtKeyUsageServerAuth}, + KeyUsage: keyUsage, + BasicConstraintsValid: true, + } + + certKey, err := rsa.GenerateKey(rand.Reader, 2048) + if err != nil { + return tls.Certificate{}, errors.Wrap(err, "fail to generate RSA key") + } + + certBytes, err := x509.CreateCertificate( + rand.Reader, + cert, + ca.Leaf, + &certKey.PublicKey, + ca.PrivateKey, + ) + + if err != nil { + return tls.Certificate{}, errors.Wrap(err, "fail to create signed certificate") + } + + leaf, err := x509.ParseCertificate(certBytes) + if err != nil { + return tls.Certificate{}, errors.Wrap(err, "fail to parse the certificate") + } + + return tls.Certificate{ + Certificate: [][]byte{certBytes}, + PrivateKey: certKey, + Leaf: leaf, + }, nil +} + +func serial() *big.Int { + ser = ser + 1 + return big.NewInt(ser) +} diff --git a/libbeat/common/transport/tlscommon/config.go b/libbeat/common/transport/tlscommon/config.go index 38f6500e183..3fdaeced560 100644 --- a/libbeat/common/transport/tlscommon/config.go +++ b/libbeat/common/transport/tlscommon/config.go @@ -33,6 +33,7 @@ type Config struct { Certificate CertificateConfig `config:",inline" yaml:",inline"` CurveTypes []tlsCurveType `config:"curve_types" yaml:"curve_types,omitempty"` Renegotiation tlsRenegotiationSupport `config:"renegotiation" yaml:"renegotiation"` + CASha256 pins `config:"ca_sha256" yaml:"ca_sha256,omitempty"` } // LoadTLSConfig will load a certificate from config with all TLS based keys @@ -88,6 +89,7 @@ func LoadTLSConfig(config *Config) (*TLSConfig, error) { CipherSuites: cipherSuites, CurvePreferences: curves, Renegotiation: tls.RenegotiationSupport(config.Renegotiation), + CASha256: config.CASha256, }, nil } diff --git a/libbeat/common/transport/tlscommon/tls_config.go b/libbeat/common/transport/tlscommon/tls_config.go index a7e69ea3ab4..41c574bc078 100644 --- a/libbeat/common/transport/tlscommon/tls_config.go +++ b/libbeat/common/transport/tlscommon/tls_config.go @@ -64,6 +64,10 @@ type TLSConfig struct { // ClientAuth controls how we want to verify certificate from a client, `none`, `optional` and // `required`, default to required. Do not affect TCP client. ClientAuth tls.ClientAuthType + + // CASha256 is the CA certificate pin, this is used to validate the CA that will be used to trust + // the server certificate. + CASha256 pins } // ToConfig generates a tls.Config object. Note, you must use BuildModuleConfig to generate a config with @@ -79,17 +83,25 @@ func (c *TLSConfig) ToConfig() *tls.Config { logp.Warn("SSL/TLS verifications disabled.") } + // When we are usign the CAsha256 pin to validate the CA used to validate the chain + // we add a custom callback. + var verifyPeerCertFn verifyPeerCertFunc + if len(c.CASha256) > 0 { + verifyPeerCertFn = MakeCAPinCallback(c.CASha256) + } + return &tls.Config{ - MinVersion: minVersion, - MaxVersion: maxVersion, - Certificates: c.Certificates, - RootCAs: c.RootCAs, - ClientCAs: c.ClientCAs, - InsecureSkipVerify: insecure, - CipherSuites: c.CipherSuites, - CurvePreferences: c.CurvePreferences, - Renegotiation: c.Renegotiation, - ClientAuth: c.ClientAuth, + MinVersion: minVersion, + MaxVersion: maxVersion, + Certificates: c.Certificates, + RootCAs: c.RootCAs, + ClientCAs: c.ClientCAs, + InsecureSkipVerify: insecure, + CipherSuites: c.CipherSuites, + CurvePreferences: c.CurvePreferences, + Renegotiation: c.Renegotiation, + ClientAuth: c.ClientAuth, + VerifyPeerCertificate: verifyPeerCertFn, } } diff --git a/libbeat/docker-compose.yml b/libbeat/docker-compose.yml index e0200a1b89b..267cc441a15 100644 --- a/libbeat/docker-compose.yml +++ b/libbeat/docker-compose.yml @@ -20,6 +20,8 @@ services: - KIBANA_PORT=5601 - ES_MONITORING_HOST=elasticsearch_monitoring - ES_MONITORING_PORT=9200 + - ES_HOST_SSL=elasticsearchssl + - ES_PORT_SSL=9200 env_file: - ${PWD}/build/test.env volumes: @@ -36,6 +38,7 @@ services: depends_on: elasticsearch: { condition: service_healthy } elasticsearch_monitoring: { condition: service_healthy } + elasticsearchssl: { condition: service_healthy } logstash: { condition: service_healthy } kafka: { condition: service_healthy } redis: { condition: service_healthy } @@ -57,6 +60,36 @@ services: healthcheck: test: ["CMD", "curl", "-f", "http://localhost:9200"] + elasticsearchssl: + extends: + file: ${ES_BEATS}/testing/environments/${TESTING_ENVIRONMENT}.yml + service: elasticsearch + healthcheck: + test: ["CMD", "curl", "-u", "admin:changeme", "-f", "https://localhost:9200", "--insecure"] + retries: 1200 + interval: 5s + start_period: 60s + environment: + - "ES_JAVA_OPTS=-Xms512m -Xmx512m" + - "network.host=" + - "transport.host=127.0.0.1" + - "http.host=0.0.0.0" + - "xpack.security.enabled=true" + - "indices.id_field_data.enabled=true" + - "xpack.license.self_generated.type=trial" + - "xpack.security.http.ssl.enabled=true" + - "xpack.security.http.ssl.key=/usr/share/elasticsearch/config/pki/elasticsearchssl/elasticsearchssl.key" + - "xpack.security.http.ssl.certificate=/usr/share/elasticsearch/config/pki/elasticsearchssl/elasticsearchssl.crt" + - "xpack.security.http.ssl.certificate_authorities=/usr/share/elasticsearch/config/pki/ca/ca.crt" + # Do no used indices as the storage for credentials, using file based allow Elasticsearch + # to be online and green much quicker. + - "xpack.security.authc.realms.file.file1.order=0" + volumes: + - ${ES_BEATS}/testing/environments/docker/elasticsearch/pki:/usr/share/elasticsearch/config/pki:ro + ports: + - 9200 + command: bash -c "bin/elasticsearch-users useradd admin -r superuser -p changeme | /usr/local/bin/docker-entrypoint.sh eswrapper" + # This host name is static because of the certificate. logstash: extends: diff --git a/libbeat/docs/shared-ssl-config.asciidoc b/libbeat/docs/shared-ssl-config.asciidoc index e0ac6b85c49..899d0dcd681 100644 --- a/libbeat/docs/shared-ssl-config.asciidoc +++ b/libbeat/docs/shared-ssl-config.asciidoc @@ -229,6 +229,20 @@ are `never`, `once`, and `freely`. The default value is never. * `once` - Allows a remote server to request renegotiation once per connection. * `freely` - Allows a remote server to repeatedly request renegotiation. + +[float] +==== `ca_sha256` + +This configure a certificate pin can that ca be used to ensure that a specific certificate is used +to as part of the verified chain. + +The pin is a base64 encoded string of the SHA-256 of the certificate. + +NOTE: This check is not a replacement for the normal SSL validation but it add additional validation. +If this option is used with `verification_mode` set to `none`, the check will always fail because +it will not receive any verified chains. + + ifeval::["{beatname_lc}" == "filebeat"] [float] ==== `client_authentication` diff --git a/libbeat/scripts/Makefile b/libbeat/scripts/Makefile index db8236cdeef..69a9d95e7f4 100755 --- a/libbeat/scripts/Makefile +++ b/libbeat/scripts/Makefile @@ -40,6 +40,8 @@ ES_HOST?=elasticsearch ES_PORT?=9200 ES_USER?=beats ES_PASS?=testing +ES_HOST_SSL?=elasticsearchssl +ES_PORT_SSL?=9200 KIBANA_HOST?=kibana KIBANA_PORT?=5601 # Kibana's Elaticsearch user @@ -398,6 +400,8 @@ write-environment: # set ENV variables for beat echo "ES_HOST=${ES_HOST}" >> ${BUILD_DIR}/test.env echo "ES_PORT=${ES_PORT}" >> ${BUILD_DIR}/test.env + echo "ES_HOST_SSL=${ES_HOST_SSL}" >> ${BUILD_DIR}/test.env + echo "ES_PORT_SSL=${ES_PORT_SSL}" >> ${BUILD_DIR}/test.env echo "ES_USER=${ES_USER}" >> ${BUILD_DIR}/test.env echo "ES_PASS=${ES_PASS}" >> ${BUILD_DIR}/test.env echo "ES_SUPERUSER_USER=${ES_SUPERUSER_USER}" >> ${BUILD_DIR}/test.env diff --git a/libbeat/tests/system/beat/beat.py b/libbeat/tests/system/beat/beat.py index d2f7f5b83fe..4869b3fe9ef 100644 --- a/libbeat/tests/system/beat/beat.py +++ b/libbeat/tests/system/beat/beat.py @@ -644,6 +644,16 @@ def get_elasticsearch_url(self): port=os.getenv("ES_PORT", "9200"), ) + def get_elasticsearch_url_ssl(self): + """ + Returns an elasticsearch.Elasticsearch instance built from the + env variables like the integration tests. + """ + return "https://{host}:{port}".format( + host=os.getenv("ES_HOST_SSL", "localhost"), + port=os.getenv("ES_PORT_SSL", "9205"), + ) + def get_kibana_url(self): """ Returns kibana host URL diff --git a/libbeat/tests/system/test_ca_pinning.py b/libbeat/tests/system/test_ca_pinning.py new file mode 100644 index 00000000000..1e0dd6f6a43 --- /dev/null +++ b/libbeat/tests/system/test_ca_pinning.py @@ -0,0 +1,81 @@ +from base import BaseTest +from idxmgmt import IdxMgmt +import os +from nose.plugins.attrib import attr +import unittest +import logging +from nose.tools import raises +from elasticsearch import RequestError + +INTEGRATION_TESTS = os.environ.get('INTEGRATION_TESTS', False) + + +class TestCAPinning(BaseTest): + """ + Test beat CA pinning for elasticsearch + """ + + @unittest.skipUnless(INTEGRATION_TESTS, "integration test") + @attr('integration') + def test_sending_events_with_a_good_sha256(self): + """ + Test Sending events while using ca pinning with a good sha256 + """ + + ca = os.path.join(self.beat_path, + "..", + "testing", + "environments", + "docker", + "elasticsearch", + "pki", + "ca", + "ca.crt") + + self.render_config_template( + elasticsearch={ + "hosts": self.get_elasticsearch_url_ssl(), + "username": "admin", + "password": "changeme", + "ssl.certificate_authorities": [ca], + "ssl.ca_sha256": "8hZS8gpciuzlu+7Xi0sdv8T7RKRRxG1TWKumUQsDam0=", + }, + ) + + proc = self.start_beat() + self.wait_until(lambda: self.log_contains("mockbeat start running.")) + self.wait_until(lambda: self.log_contains("PublishEvents: 1 events have been published")) + proc.check_kill_and_wait() + + @unittest.skipUnless(INTEGRATION_TESTS, "integration test") + @attr('integration') + def test_sending_events_with_a_bad_sha256(self): + """ + Test Sending events while using ca pinning with a bad sha256 + """ + + ca = os.path.join(self.beat_path, + "..", + "testing", + "environments", + "docker", + "elasticsearch", + "pki", + "ca", + "ca.crt") + + self.render_config_template( + elasticsearch={ + "hosts": self.get_elasticsearch_url_ssl(), + "username": "admin", + "password": "changeme", + "ssl.certificate_authorities": [ca], + "ssl.ca_sha256": "not-good-sha", + }, + ) + + proc = self.start_beat() + self.wait_until(lambda: self.log_contains("mockbeat start running.")) + self.wait_until(lambda: self.log_contains( + "provided CA certificate pins doesn't match any of the certificate authorities used to validate the certificate")) + proc.check_kill_and_wait() diff --git a/metricbeat/metricbeat.reference.yml b/metricbeat/metricbeat.reference.yml index 390a5408e58..8b561b433a2 100644 --- a/metricbeat/metricbeat.reference.yml +++ b/metricbeat/metricbeat.reference.yml @@ -1200,6 +1200,12 @@ output.elasticsearch: # never, once, and freely. Default is never. #ssl.renegotiation: never + # Configure a pin that can be used to do extra validation of the verified certificate chain, + # this allow you to ensure that a specific certificate is used to validate the chain of trust. + # + # The pin is a base64 encoded string of the SHA-256 fingerprint. + #ssl.ca_sha256 + #----------------------------- Logstash output --------------------------------- #output.logstash: # Boolean flag to enable or disable the output module. @@ -1293,6 +1299,12 @@ output.elasticsearch: # never, once, and freely. Default is never. #ssl.renegotiation: never + # Configure a pin that can be used to do extra validation of the verified certificate chain, + # this allow you to ensure that a specific certificate is used to validate the chain of trust. + # + # The pin is a base64 encoded string of the SHA-256 fingerprint. + #ssl.ca_sha256 + # The number of times to retry publishing an event after a publishing failure. # After the specified number of retries, the events are typically dropped. # Some Beats, such as Filebeat and Winlogbeat, ignore the max_retries setting diff --git a/packetbeat/packetbeat.reference.yml b/packetbeat/packetbeat.reference.yml index aa8c9399aa5..ad66e8fa18b 100644 --- a/packetbeat/packetbeat.reference.yml +++ b/packetbeat/packetbeat.reference.yml @@ -938,6 +938,12 @@ output.elasticsearch: # never, once, and freely. Default is never. #ssl.renegotiation: never + # Configure a pin that can be used to do extra validation of the verified certificate chain, + # this allow you to ensure that a specific certificate is used to validate the chain of trust. + # + # The pin is a base64 encoded string of the SHA-256 fingerprint. + #ssl.ca_sha256 + #----------------------------- Logstash output --------------------------------- #output.logstash: # Boolean flag to enable or disable the output module. @@ -1031,6 +1037,12 @@ output.elasticsearch: # never, once, and freely. Default is never. #ssl.renegotiation: never + # Configure a pin that can be used to do extra validation of the verified certificate chain, + # this allow you to ensure that a specific certificate is used to validate the chain of trust. + # + # The pin is a base64 encoded string of the SHA-256 fingerprint. + #ssl.ca_sha256 + # The number of times to retry publishing an event after a publishing failure. # After the specified number of retries, the events are typically dropped. # Some Beats, such as Filebeat and Winlogbeat, ignore the max_retries setting diff --git a/testing/environments/docker/elasticsearch/pki/ca/ca.crt b/testing/environments/docker/elasticsearch/pki/ca/ca.crt new file mode 100644 index 00000000000..a18a84fd7b6 --- /dev/null +++ b/testing/environments/docker/elasticsearch/pki/ca/ca.crt @@ -0,0 +1,20 @@ +-----BEGIN CERTIFICATE----- +MIIDSjCCAjKgAwIBAgIVAOshUH7Va8Kh1QeA4KgLw8dI29M4MA0GCSqGSIb3DQEB +CwUAMDQxMjAwBgNVBAMTKUVsYXN0aWMgQ2VydGlmaWNhdGUgVG9vbCBBdXRvZ2Vu +ZXJhdGVkIENBMB4XDTIwMDIwNzE2MzUzMFoXDTIzMDIwNjE2MzUzMFowNDEyMDAG +A1UEAxMpRWxhc3RpYyBDZXJ0aWZpY2F0ZSBUb29sIEF1dG9nZW5lcmF0ZWQgQ0Ew +ggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCv7Oq3uE0kO5Ij41U9M7ee +xOtprdA3joR68B32/SQ1FW9Igk3f/DTn8MqlAFexwzGAONPNIcj44W9KhVeT9aFA +qCKfS2no+V9aKds6wyEHY3sAmYICEHBMDor9KhPnIc8m/gl3TcGMKmouoHqFPNKE +irilBDUO7rs5w46lcbxJrHTlEA6xyQLT7+sJ4DswO/xeoemPTBa7vzkoVUyZ50/D +VSUulY4XtmQvmbe4Aa0p8sgLNzFAJRl3XqZMECwO2iJ9jFwKCUT4EbFW4aTQtylI +CBax+Cn79vKpp3gO1WVu1cdcQW3+ciAJyUydTsCA2zjGYZyzL84z7eCHW946WQWD +AgMBAAGjUzBRMB0GA1UdDgQWBBQZKfuW0o2yabRo9tosWldK43GDbjAfBgNVHSME +GDAWgBQZKfuW0o2yabRo9tosWldK43GDbjAPBgNVHRMBAf8EBTADAQH/MA0GCSqG +SIb3DQEBCwUAA4IBAQAHeIJPwxvHTismlbFJKcCM3kr/ZblXguiFTmhqylqa8wFI +ke1xpTUrdfTAkD0ohmtPAUMPBkHeyHKzvxK7Blh230/lxybJNVSpfp7FQvj1EsmW +7FbIsKoj9MwJ2Lg5h6rnFA4t0bL3q74HV+vqpMoJDe92uX0GaSH/iYb+BfZ2El8m +QfANac0O+TE70i0++v/BzUAkqhJB3pG/3ziPzdFWlXf4iUG0YhMG4Ig5P/SvGz/V +MNc+uq3bh9xsNrtcm2S/pVdt/gdsujg9MTaoOr+maJPB/+LBrkZWtZcbUe++1+Z7 +32exp0eKNA0i90cc/Ayr79MOFDxdgI7baBnLPPa8 +-----END CERTIFICATE----- diff --git a/testing/environments/docker/elasticsearch/pki/ca/ca.key b/testing/environments/docker/elasticsearch/pki/ca/ca.key new file mode 100644 index 00000000000..48982cea36e --- /dev/null +++ b/testing/environments/docker/elasticsearch/pki/ca/ca.key @@ -0,0 +1,27 @@ +-----BEGIN RSA PRIVATE KEY----- +MIIEpQIBAAKCAQEAr+zqt7hNJDuSI+NVPTO3nsTraa3QN46EevAd9v0kNRVvSIJN +3/w05/DKpQBXscMxgDjTzSHI+OFvSoVXk/WhQKgin0tp6PlfWinbOsMhB2N7AJmC +AhBwTA6K/SoT5yHPJv4Jd03BjCpqLqB6hTzShIq4pQQ1Du67OcOOpXG8Sax05RAO +sckC0+/rCeA7MDv8XqHpj0wWu785KFVMmedPw1UlLpWOF7ZkL5m3uAGtKfLICzcx +QCUZd16mTBAsDtoifYxcCglE+BGxVuGk0LcpSAgWsfgp+/byqad4DtVlbtXHXEFt +/nIgCclMnU7AgNs4xmGcsy/OM+3gh1veOlkFgwIDAQABAoIBAALZzfvoKqfZp0aZ +mnoBaopSGpZ90I/16UOsvG+SLpIFpOYB5o0ooxrXFhGSbdldlmHDifsa/wy5anpE +quSk6FYJ43W9XRv/XoIxh3HuU4yxGf8qfabW6VryKWJs2iG2tIqnNzQNuIMy9MGI +rDOYhrjLHq7d4JY7XCFVf+xCaZCwCb3yvZwVnrAqmPoeg2FrXmCVzqr1IpmwzJ0B +OfGWzi5THLm4/aGVUBfkvGURxsmwo3jGn0myr9oUkKczOKGEqvnlVuT9+ShURZp2 +tDU8zVRF0ksUNogUSfSNgWwpCYNBIqPOdxr7nT0+NEJ7b4R7/3LXEh/tRcuRNX+d +mjUMwbECgYEA/1MWpTdB9+9M7wyQasd29m1b0mkw+Jebdw+CuIK3XdPPGOfD17WO +sKZek3uK24DFGzRQf8tzHqzGvHncykAhb3oePVbfuhE5jt9bfgAOX8Naz6AK6Dmj +6+pJgXFTTNGL8JDojsIlabq4QH7oB02HoQ87GTr8IF4CjlJCHcyVB98CgYEAsGQO +uz0A1HdeuzbOP0+E86Ip03gcq66mVibXpy2qdMwEluxARW52XPKc8LKKI0QS4Qxk +giHHTQwPTLXJW9gM8v9/SQupQ/Vx8Zi3KjQ2ZAQoj6bGyDJ1P278GePJC4b0h/vG +F0sSUsmoEUGrLtq8Ofv3hDF6Ik247MQFi7i+Bt0CgYEAgP0kAqGw9SXzEw2g5CET +C5nh+qHj+KL3CqZOXxLCumcoSCfGe/KgPMRAIXgXhZ8/dOfwBy/sX8HfwRY7of3W +JnBmWIzMCD9tea2DlltG58BU33G2MO31z1iUfA2ZjMSMUyOSKZURu6F2Njcm15Gm +hIqiS7PN7jgwSGBsQIu7ercCgYEAh/nKJWrkbeVLgLTCD6okSpAzABLyvyJWlclB +q12Xrovr6dBbx2pdEk/wzdhEhuUeTKB6Bps1gV6PmMn2XLfTW6u8GrpDlODsIptg +b8dqOnW+MucVDBVhrzHGY8rmG93AOefMD/7ONEXCKvNdnDQAsA5eA2kExtb1fIer +4sbarn0CgYEAl1av+NOVduN1KrJXuZnNeN4KeNoYqJOS4s14Wk37GIujsrcE/m18 +BhZk0Al/oKZIDSuya5GGRhT+ndD8OWc4DEMWk2JnJdWKh20FfeM6UXVI46Sq3Won +vPDB6GYr45uRgtLvYeitLpXE5s9GmH8GyIV9om3TvDiceMXd/LbCodY= +-----END RSA PRIVATE KEY----- diff --git a/testing/environments/docker/elasticsearch/pki/elasticsearchssl/elasticsearchssl.crt b/testing/environments/docker/elasticsearch/pki/elasticsearchssl/elasticsearchssl.crt new file mode 100644 index 00000000000..4b373ea66a0 --- /dev/null +++ b/testing/environments/docker/elasticsearch/pki/elasticsearchssl/elasticsearchssl.crt @@ -0,0 +1,19 @@ +-----BEGIN CERTIFICATE----- +MIIDKjCCAhKgAwIBAgIUZWu3nanhrFaNe6kMhtsPM4neUCYwDQYJKoZIhvcNAQEL +BQAwNDEyMDAGA1UEAxMpRWxhc3RpYyBDZXJ0aWZpY2F0ZSBUb29sIEF1dG9nZW5l +cmF0ZWQgQ0EwHhcNMjAwMjA3MTYzNjMxWhcNMjMwMjA2MTYzNjMxWjAbMRkwFwYD +VQQDExBlbGFzdGljc2VhcmNoc3NsMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIB +CgKCAQEAl09yaI0HI3I8DmJ0UyRDedFNBFOfsFh7sYGGElj0h4H1kt8oEA3uYIH/ +oPUQ9Mkn30m+qccdQC6/pz/ZgkCfOckXtX1PVLEAK9MEqEwj6UU4uMgSIUTjXN22 +m/YedSJFtwGiQqFbCD1LijRLjlDCvHZ1W5M6XYzWoUN1y4MDZSD755TuluAp277j +6yuJIEw5SsQ/Nw4Coaqexy1ha7G/y3L+3t4SFrXaBqe+nM1xPDR0Is/p8iTdcdlu +kEFmbIqDGAPx2jvTRWYikL3MmR4u58AoIk0WqeGmLefxzV6jC6zsQGRnpmtz3jye +XHRfodf3crMZm+mw6FNPk4PJzZSsXwIDAQABo00wSzAdBgNVHQ4EFgQUmcNplxkS ++zHt5LWVM67Tzws8fBEwHwYDVR0jBBgwFoAUGSn7ltKNsmm0aPbaLFpXSuNxg24w +CQYDVR0TBAIwADANBgkqhkiG9w0BAQsFAAOCAQEAPpFdJpuwBXqBZezfQTTQFkKm +EHz3UDYKA3nHt2tcmFqAEXYx4cXaor5GG9YLThGWUp2iBXIyIzUZnpkM2wl/pIlz +8fMFxvtS6hQ2VwFDHAo2ht8ay7/vTrKcVvNL5NtPHjRlHhT94XiwYNpneiB6EMGP ++lTxWXSLpSnl0AnFdpLzPpS6DiaMHAPChAbDGK9i76D13sQBJZ/lgQiMmntEWsTr +0NNsjBk2xjMQAYs/eJXfENkAxvuzJTbQdJ1kMOvybONT4Lw8UIhoRpRY7EspwlI3 +encLBhcxYJjpzSPqdDQQRVXd4zUNFe4595LKEsm14mXaTy682HAe/HvN+yO7qw== +-----END CERTIFICATE----- diff --git a/testing/environments/docker/elasticsearch/pki/elasticsearchssl/elasticsearchssl.key b/testing/environments/docker/elasticsearch/pki/elasticsearchssl/elasticsearchssl.key new file mode 100644 index 00000000000..f374f10fa44 --- /dev/null +++ b/testing/environments/docker/elasticsearch/pki/elasticsearchssl/elasticsearchssl.key @@ -0,0 +1,27 @@ +-----BEGIN RSA PRIVATE KEY----- +MIIEpAIBAAKCAQEAl09yaI0HI3I8DmJ0UyRDedFNBFOfsFh7sYGGElj0h4H1kt8o +EA3uYIH/oPUQ9Mkn30m+qccdQC6/pz/ZgkCfOckXtX1PVLEAK9MEqEwj6UU4uMgS +IUTjXN22m/YedSJFtwGiQqFbCD1LijRLjlDCvHZ1W5M6XYzWoUN1y4MDZSD755Tu +luAp277j6yuJIEw5SsQ/Nw4Coaqexy1ha7G/y3L+3t4SFrXaBqe+nM1xPDR0Is/p +8iTdcdlukEFmbIqDGAPx2jvTRWYikL3MmR4u58AoIk0WqeGmLefxzV6jC6zsQGRn +pmtz3jyeXHRfodf3crMZm+mw6FNPk4PJzZSsXwIDAQABAoIBAB1rHu1g7hBgN3j8 +f21i0ZOvs++xaozYx0Pd0PlkPjbSd7KUnK9yZfRxkgfzXdaZ/ZyWM/HCetdtv2l/ +KoT+l3aeuHNa57+pokTjBDbMhvbltH+Itq2tPR9jJAvysD1J6pAIS0n1IUPa1wMJ +497JqPMHfQ3O9DwYE+rKuO5WjKRulUrL8K3OgHndLiHPZuUfIveSd6qux7wAebmD +OpWukVvYoC2k//Bgopdyg9VxVZtTg1SZlyFZ8wteDrbgF+eDMp9uIRddrvMUCwH4 ++GJOzkXxgkeOANjr5obMRjrr5hwoCE+RObCXAT3lx+nfCvYY5Lb72WWPQPEJ5ltP +xuxYY/ECgYEA71+DxCSUpxaK6THpJ10Z4FlTV0YAFfnMx9Jecn4CaJpQrWYFjLB7 +zkhlJWWyzPMc56+5olfcMEXHO9dT1/w3lFlJmRaS4yu/ZdPf2E6Pi6eXpeRYshj9 +NIq/pMCB1XxNogGzQA0AFBc+vw6Tx7LG+Bz/Yafi4SQN89I9v2SaeiMCgYEAodIE +epMZmVhlmrVzjPKcYtqWu464Sb3sHBwgnxvKcU1NUAUjTuzI9DwrJYgrA9NBcgHq +ckwbqiHNcej4MGFk7nN98U47eb+p6PAPNde7q42iNz2q7pKlNVml+Eg/wC2lhNah +N6K6S4wvTM6ujNIZGQ3DyKQC0tCMu+LnPxYYcpUCgYEAi9E2nfLgAVjheqR0k1GG +M8z5KRjyI+PtASqXkDiaH49DYIUe6LaNGkifC+EDN0MptwqlW3YGXwvi+8kiaB4i +OLyOiKTu11JOUaQYM7hvkBssMPHX/O8rtuz0U78+FvysO9zSXq85RILvW5mgKBz8 +qyAE632sv+TXYXuEJa8VrBECgYEAmAmh6aSh7aDPPc90NJ6R7pMgAjKy1Z4a48JN +qBBNYazWkfNx3Cq/GDIb+9R3Tc3D9KD2LNNzPqMpyvevkI1BboSWdQ0i9l3s/w3l +zJnYGvQk0DAhlKu1i22icac4NpDsreWWbZZ34Jliq5CZEXgo2pBDPhVTDc2iHLmw +uWZCLA0CgYAG99zukAD9iq2MyGiyuJ8dYU0dDvyaKcSkM3pfsgA4TpSlZljDKnAH +1VVPGB3pOHUYbcxsD2/1AJBlplvq8OVcrURuSXYl9PtwczJBgfSNNtSkHvMirWzo +q7eEeYCCs/VZUr9mY0nuzysq3ltiBW6tsdCn6d89ogs2WvseTlHZLg== +-----END RSA PRIVATE KEY----- diff --git a/testing/environments/docker/elasticsearch/pki/generate_pki.sh b/testing/environments/docker/elasticsearch/pki/generate_pki.sh new file mode 100755 index 00000000000..beb43d294ea --- /dev/null +++ b/testing/environments/docker/elasticsearch/pki/generate_pki.sh @@ -0,0 +1,4 @@ +#!/bin/sh +# Take the certificates and create a DER format and create a sha256 of it and encode it to base 64 +# https://www.openssl.org/docs/manmaster/man1/dgst.html +openssl x509 -in ca/ca.crt -pubkey -noout | openssl pkey -pubin -outform der | openssl dgst -sha256 -binary | openssl enc -base64 diff --git a/winlogbeat/winlogbeat.reference.yml b/winlogbeat/winlogbeat.reference.yml index fbf716b157d..51aaa89ec97 100644 --- a/winlogbeat/winlogbeat.reference.yml +++ b/winlogbeat/winlogbeat.reference.yml @@ -434,6 +434,12 @@ output.elasticsearch: # never, once, and freely. Default is never. #ssl.renegotiation: never + # Configure a pin that can be used to do extra validation of the verified certificate chain, + # this allow you to ensure that a specific certificate is used to validate the chain of trust. + # + # The pin is a base64 encoded string of the SHA-256 fingerprint. + #ssl.ca_sha256 + #----------------------------- Logstash output --------------------------------- #output.logstash: # Boolean flag to enable or disable the output module. @@ -527,6 +533,12 @@ output.elasticsearch: # never, once, and freely. Default is never. #ssl.renegotiation: never + # Configure a pin that can be used to do extra validation of the verified certificate chain, + # this allow you to ensure that a specific certificate is used to validate the chain of trust. + # + # The pin is a base64 encoded string of the SHA-256 fingerprint. + #ssl.ca_sha256 + # The number of times to retry publishing an event after a publishing failure. # After the specified number of retries, the events are typically dropped. # Some Beats, such as Filebeat and Winlogbeat, ignore the max_retries setting diff --git a/x-pack/auditbeat/auditbeat.reference.yml b/x-pack/auditbeat/auditbeat.reference.yml index 70c7d197a53..8f70d82821e 100644 --- a/x-pack/auditbeat/auditbeat.reference.yml +++ b/x-pack/auditbeat/auditbeat.reference.yml @@ -567,6 +567,12 @@ output.elasticsearch: # never, once, and freely. Default is never. #ssl.renegotiation: never + # Configure a pin that can be used to do extra validation of the verified certificate chain, + # this allow you to ensure that a specific certificate is used to validate the chain of trust. + # + # The pin is a base64 encoded string of the SHA-256 fingerprint. + #ssl.ca_sha256 + #----------------------------- Logstash output --------------------------------- #output.logstash: # Boolean flag to enable or disable the output module. @@ -660,6 +666,12 @@ output.elasticsearch: # never, once, and freely. Default is never. #ssl.renegotiation: never + # Configure a pin that can be used to do extra validation of the verified certificate chain, + # this allow you to ensure that a specific certificate is used to validate the chain of trust. + # + # The pin is a base64 encoded string of the SHA-256 fingerprint. + #ssl.ca_sha256 + # The number of times to retry publishing an event after a publishing failure. # After the specified number of retries, the events are typically dropped. # Some Beats, such as Filebeat and Winlogbeat, ignore the max_retries setting diff --git a/x-pack/filebeat/filebeat.reference.yml b/x-pack/filebeat/filebeat.reference.yml index 06da5fe0465..03739cb0122 100644 --- a/x-pack/filebeat/filebeat.reference.yml +++ b/x-pack/filebeat/filebeat.reference.yml @@ -1736,6 +1736,12 @@ output.elasticsearch: # never, once, and freely. Default is never. #ssl.renegotiation: never + # Configure a pin that can be used to do extra validation of the verified certificate chain, + # this allow you to ensure that a specific certificate is used to validate the chain of trust. + # + # The pin is a base64 encoded string of the SHA-256 fingerprint. + #ssl.ca_sha256 + #----------------------------- Logstash output --------------------------------- #output.logstash: # Boolean flag to enable or disable the output module. @@ -1829,6 +1835,12 @@ output.elasticsearch: # never, once, and freely. Default is never. #ssl.renegotiation: never + # Configure a pin that can be used to do extra validation of the verified certificate chain, + # this allow you to ensure that a specific certificate is used to validate the chain of trust. + # + # The pin is a base64 encoded string of the SHA-256 fingerprint. + #ssl.ca_sha256 + # The number of times to retry publishing an event after a publishing failure. # After the specified number of retries, the events are typically dropped. # Some Beats, such as Filebeat and Winlogbeat, ignore the max_retries setting diff --git a/x-pack/functionbeat/functionbeat.reference.yml b/x-pack/functionbeat/functionbeat.reference.yml index ab6a5e5bd44..e6052c2822c 100644 --- a/x-pack/functionbeat/functionbeat.reference.yml +++ b/x-pack/functionbeat/functionbeat.reference.yml @@ -777,6 +777,12 @@ output.elasticsearch: # never, once, and freely. Default is never. #ssl.renegotiation: never + # Configure a pin that can be used to do extra validation of the verified certificate chain, + # this allow you to ensure that a specific certificate is used to validate the chain of trust. + # + # The pin is a base64 encoded string of the SHA-256 fingerprint. + #ssl.ca_sha256 + #----------------------------- Logstash output --------------------------------- #output.logstash: # Boolean flag to enable or disable the output module. @@ -870,6 +876,12 @@ output.elasticsearch: # never, once, and freely. Default is never. #ssl.renegotiation: never + # Configure a pin that can be used to do extra validation of the verified certificate chain, + # this allow you to ensure that a specific certificate is used to validate the chain of trust. + # + # The pin is a base64 encoded string of the SHA-256 fingerprint. + #ssl.ca_sha256 + # The number of times to retry publishing an event after a publishing failure. # After the specified number of retries, the events are typically dropped. # Some Beats, such as Filebeat and Winlogbeat, ignore the max_retries setting diff --git a/x-pack/metricbeat/metricbeat.reference.yml b/x-pack/metricbeat/metricbeat.reference.yml index d54b06a4f2e..266ec4cebc2 100644 --- a/x-pack/metricbeat/metricbeat.reference.yml +++ b/x-pack/metricbeat/metricbeat.reference.yml @@ -1433,6 +1433,12 @@ output.elasticsearch: # never, once, and freely. Default is never. #ssl.renegotiation: never + # Configure a pin that can be used to do extra validation of the verified certificate chain, + # this allow you to ensure that a specific certificate is used to validate the chain of trust. + # + # The pin is a base64 encoded string of the SHA-256 fingerprint. + #ssl.ca_sha256 + #----------------------------- Logstash output --------------------------------- #output.logstash: # Boolean flag to enable or disable the output module. @@ -1526,6 +1532,12 @@ output.elasticsearch: # never, once, and freely. Default is never. #ssl.renegotiation: never + # Configure a pin that can be used to do extra validation of the verified certificate chain, + # this allow you to ensure that a specific certificate is used to validate the chain of trust. + # + # The pin is a base64 encoded string of the SHA-256 fingerprint. + #ssl.ca_sha256 + # The number of times to retry publishing an event after a publishing failure. # After the specified number of retries, the events are typically dropped. # Some Beats, such as Filebeat and Winlogbeat, ignore the max_retries setting diff --git a/x-pack/winlogbeat/winlogbeat.reference.yml b/x-pack/winlogbeat/winlogbeat.reference.yml index 1745ebdb730..bcdbf155a04 100644 --- a/x-pack/winlogbeat/winlogbeat.reference.yml +++ b/x-pack/winlogbeat/winlogbeat.reference.yml @@ -437,6 +437,12 @@ output.elasticsearch: # never, once, and freely. Default is never. #ssl.renegotiation: never + # Configure a pin that can be used to do extra validation of the verified certificate chain, + # this allow you to ensure that a specific certificate is used to validate the chain of trust. + # + # The pin is a base64 encoded string of the SHA-256 fingerprint. + #ssl.ca_sha256 + #----------------------------- Logstash output --------------------------------- #output.logstash: # Boolean flag to enable or disable the output module. @@ -530,6 +536,12 @@ output.elasticsearch: # never, once, and freely. Default is never. #ssl.renegotiation: never + # Configure a pin that can be used to do extra validation of the verified certificate chain, + # this allow you to ensure that a specific certificate is used to validate the chain of trust. + # + # The pin is a base64 encoded string of the SHA-256 fingerprint. + #ssl.ca_sha256 + # The number of times to retry publishing an event after a publishing failure. # After the specified number of retries, the events are typically dropped. # Some Beats, such as Filebeat and Winlogbeat, ignore the max_retries setting