-
Notifications
You must be signed in to change notification settings - Fork 4.9k
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Add Google Cloud Platform support (#13598)
This PR introduces the support for Google Cloud Platform to Functionbeat. This branch is located in the `elastic/beats` repository, so anyone on our team has access to it. ### Manager #### Authentication To use the API to deploy, remove and update functions, users need to set the environment variable `GOOGLE_APPLICATION_CREDENTIALS`. This variable should point to a JSON file which contains all the relevant information for Google to authenticate. (About authentication for GCP libs: https://cloud.google.com/docs/authentication/getting-started) #### Required roles * Cloud Functions Developer * Cloud Functions Service Agent * Service Account User * Storage Admin * Storage Object Admin Note: Cloud Functions Developer role is in beta. We should not make GCP support GA, until it becomes stable. #### Configuration ```yaml # Configure functions to run on Google Cloud Platform, currently, we assume that the credentials # are present in the environment to correctly create the function when using the CLI. # # Configure which region your project is located in. functionbeat.provider.gcp.location_id: "europe-west1" # Configure which Google Cloud project to deploy your functions. functionbeat.provider.gcp.project_id: "my-project-123456" # Configure the Google Cloud Storage we should upload the function artifact. functionbeat.provider.gcp.storage_name: "functionbeat-deploy" functionbeat.provider.gcp.functions: ``` #### Export Function templates can be exported into YAML. With this YAML configuration, users can deploy the function using the [Google Cloud Deployment Manager](https://cloud.google.com/deployment-manager/). ### New functions #### Google Pub/Sub A function under the folder `pkg/pubsub` is available to get events from Google Pub/Sub. ##### Configuration ```yaml # Define the list of function availables, each function required to have a unique name. # Create a function that accepts events coming from Google Pub/Sub. - name: pubsub enabled: false type: pubsub # Description of the method to help identify them when you run multiples functions. description: "Google Cloud Function for Pub/Sub" # The maximum memory allocated for this function, the configured size must be a factor of 64. # Default is 256MiB. #memory_size: 256MiB # Execution timeout in seconds. If the function does not finish in time, # it is considered failed and terminated. Default is 60s. #timeout: 60s # Email of the service account of the function. Defaults to {projectid}@appspot.gserviceaccount.com #service_account_email: {projectid}@appspot.gserviceaccount.com # Labels of the function. #labels: # mylabel: label # VPC Connector this function can connect to. # Format: projects/*/locations/*/connectors/* or fully-qualified URI #vpc_connector: "" # Number of maximum instances running at the same time. Default is unlimited. #maximum_instances: 0 trigger: event_type: "providers/cloud.pubsub/eventTypes/topic.publish" resource: "projects/_/pubsub/myPubSub" #service: "pubsub.googleapis.com" # Optional fields that you can specify to add additional information to the # output. Fields can be scalar values, arrays, dictionaries, or any nested # combination of these. #fields: # env: staging # Define custom processors for this function. #processors: # - dissect: # tokenizer: "%{key1} %{key2}" ``` #### Google Cloud Storage A function under the folder pkg/storage is available to get events from Google Cloud Storage. ##### Configuration ```yaml # Create a function that accepts events coming from Google Cloud Storage. - name: storage enabled: false type: storage # Description of the method to help identify them when you run multiples functions. description: "Google Cloud Function for Cloud Storage" # The maximum memory allocated for this function, the configured size must be a factor of 64. # Default is 256MiB. #memory_size: 256MiB # Execution timeout in seconds. If the function does not finish in time, # it is considered failed and terminated. Default is 60s. #timeout: 60s # Email of the service account of the function. Defaults to {projectid}@appspot.gserviceaccount.com #service_account_email: {projectid}@appspot.gserviceaccount.com # Labels of the function. #labels: # mylabel: label # VPC Connector this function can connect to. # Format: projects/*/locations/*/connectors/* or fully-qualified URI #vpc_connector: "" # Number of maximum instances running at the same time. Default is unlimited. #maximum_instances: 0 # Optional fields that you can specify to add additional information to the # output. Fields can be scalar values, arrays, dictionaries, or any nested # combination of these. #fields: # env: staging # Define custom processors for this function. #processors: # - dissect: # tokenizer: "%{key1} %{key2}" ``` ### Vendor * `cloud.google.com/go/functions/metadata` * `cloud.google.com/go/storage`
- Loading branch information
Showing
88 changed files
with
61,581 additions
and
230 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,41 @@ | ||
| // Licensed to Elasticsearch B.V. under one or more contributor | ||
| // license agreements. See the NOTICE file distributed with | ||
| // this work for additional information regarding copyright | ||
| // ownership. Elasticsearch B.V. licenses this file to you under | ||
| // the Apache License, Version 2.0 (the "License"); you may | ||
| // not use this file except in compliance with the License. | ||
| // You may obtain a copy of the License at | ||
| // | ||
| // http://www.apache.org/licenses/LICENSE-2.0 | ||
| // | ||
| // Unless required by applicable law or agreed to in writing, | ||
| // software distributed under the License is distributed on an | ||
| // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY | ||
| // KIND, either express or implied. See the License for the | ||
| // specific language governing permissions and limitations | ||
| // under the License. | ||
|
|
||
| package tlscommon | ||
|
|
||
| import "fmt" | ||
|
|
||
| // TLSVersion type for TLS version. | ||
| type TLSVersion uint16 | ||
|
|
||
| func (v TLSVersion) String() string { | ||
| if s, ok := tlsProtocolVersionsInverse[v]; ok { | ||
| return s | ||
| } | ||
| return "unknown" | ||
| } | ||
|
|
||
| //Unpack transforms the string into a constant. | ||
| func (v *TLSVersion) Unpack(s string) error { | ||
| version, found := tlsProtocolVersions[s] | ||
| if !found { | ||
| return fmt.Errorf("invalid tls version '%v'", s) | ||
| } | ||
|
|
||
| *v = version | ||
| return nil | ||
| } |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,70 @@ | ||
| // Licensed to Elasticsearch B.V. under one or more contributor | ||
| // license agreements. See the NOTICE file distributed with | ||
| // this work for additional information regarding copyright | ||
| // ownership. Elasticsearch B.V. licenses this file to you under | ||
| // the Apache License, Version 2.0 (the "License"); you may | ||
| // not use this file except in compliance with the License. | ||
| // You may obtain a copy of the License at | ||
| // | ||
| // http://www.apache.org/licenses/LICENSE-2.0 | ||
| // | ||
| // Unless required by applicable law or agreed to in writing, | ||
| // software distributed under the License is distributed on an | ||
| // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY | ||
| // KIND, either express or implied. See the License for the | ||
| // specific language governing permissions and limitations | ||
| // under the License. | ||
|
|
||
| // +build go1.13 | ||
|
|
||
| package tlscommon | ||
|
|
||
| import "crypto/tls" | ||
|
|
||
| // Define all the possible TLS version. | ||
| const ( | ||
| TLSVersionSSL30 TLSVersion = tls.VersionSSL30 | ||
| TLSVersion10 TLSVersion = tls.VersionTLS10 | ||
| TLSVersion11 TLSVersion = tls.VersionTLS11 | ||
| TLSVersion12 TLSVersion = tls.VersionTLS12 | ||
| TLSVersion13 TLSVersion = tls.VersionTLS13 | ||
|
|
||
| // TLSVersionMin is the min TLS version supported. | ||
| TLSVersionMin = TLSVersionSSL30 | ||
|
|
||
| // TLSVersionMax is the max TLS version supported. | ||
| TLSVersionMax = TLSVersion13 | ||
|
|
||
| // TLSVersionDefaultMin is the minimal default TLS version that is | ||
| // enabled by default. TLSVersionDefaultMin is >= TLSVersionMin | ||
| TLSVersionDefaultMin = TLSVersion11 | ||
|
|
||
| // TLSVersionDefaultMax is the max default TLS version that | ||
| // is enabled by default. | ||
| TLSVersionDefaultMax = TLSVersionMax | ||
| ) | ||
|
|
||
| // TLSDefaultVersions list of versions of TLS we should support. | ||
| var TLSDefaultVersions = []TLSVersion{ | ||
| TLSVersion11, | ||
| TLSVersion12, | ||
| TLSVersion13, | ||
| } | ||
|
|
||
| var tlsProtocolVersions = map[string]TLSVersion{ | ||
| "SSLv3": TLSVersionSSL30, | ||
| "SSLv3.0": TLSVersionSSL30, | ||
| "TLSv1": TLSVersion10, | ||
| "TLSv1.0": TLSVersion10, | ||
| "TLSv1.1": TLSVersion11, | ||
| "TLSv1.2": TLSVersion12, | ||
| "TLSv1.3": TLSVersion13, | ||
| } | ||
|
|
||
| var tlsProtocolVersionsInverse = map[TLSVersion]string{ | ||
| TLSVersionSSL30: "SSLv3", | ||
| TLSVersion10: "TLSv1.0", | ||
| TLSVersion11: "TLSv1.1", | ||
| TLSVersion12: "TLSv1.2", | ||
| TLSVersion13: "TLSv1.3", | ||
| } |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,66 @@ | ||
| // Licensed to Elasticsearch B.V. under one or more contributor | ||
| // license agreements. See the NOTICE file distributed with | ||
| // this work for additional information regarding copyright | ||
| // ownership. Elasticsearch B.V. licenses this file to you under | ||
| // the Apache License, Version 2.0 (the "License"); you may | ||
| // not use this file except in compliance with the License. | ||
| // You may obtain a copy of the License at | ||
| // | ||
| // http://www.apache.org/licenses/LICENSE-2.0 | ||
| // | ||
| // Unless required by applicable law or agreed to in writing, | ||
| // software distributed under the License is distributed on an | ||
| // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY | ||
| // KIND, either express or implied. See the License for the | ||
| // specific language governing permissions and limitations | ||
| // under the License. | ||
|
|
||
| // +build !go1.13 | ||
|
|
||
| package tlscommon | ||
|
|
||
| import "crypto/tls" | ||
|
|
||
| const ( | ||
| TLSVersionSSL30 TLSVersion = tls.VersionSSL30 | ||
| TLSVersion10 TLSVersion = tls.VersionTLS10 | ||
| TLSVersion11 TLSVersion = tls.VersionTLS11 | ||
| TLSVersion12 TLSVersion = tls.VersionTLS12 | ||
|
|
||
| // TLSVersionMin is the min TLS version supported. | ||
| TLSVersionMin = TLSVersionSSL30 | ||
|
|
||
| // TLSVersionMax is the max TLS version supported. | ||
| TLSVersionMax = TLSVersion12 | ||
|
|
||
| // TLSVersionDefaultMin is the minimal default TLS version that is | ||
| // enabled by default. TLSVersionDefaultMin is >= TLSVersionMin | ||
| TLSVersionDefaultMin = TLSVersion10 | ||
|
|
||
| // TLSVersionDefaultMax is the max default TLS version that | ||
| // is enabled by default. | ||
| TLSVersionDefaultMax = TLSVersionMax | ||
| ) | ||
|
|
||
| // TLSDefaultVersions list of versions of TLS we should support. | ||
| var TLSDefaultVersions = []TLSVersion{ | ||
| TLSVersion10, | ||
| TLSVersion11, | ||
| TLSVersion12, | ||
| } | ||
|
|
||
| var tlsProtocolVersions = map[string]TLSVersion{ | ||
| "SSLv3": TLSVersionSSL30, | ||
| "SSLv3.0": TLSVersionSSL30, | ||
| "TLSv1": TLSVersion10, | ||
| "TLSv1.0": TLSVersion10, | ||
| "TLSv1.1": TLSVersion11, | ||
| "TLSv1.2": TLSVersion12, | ||
| } | ||
|
|
||
| var tlsProtocolVersionsInverse = map[TLSVersion]string{ | ||
| TLSVersionSSL30: "SSLv3", | ||
| TLSVersion10: "TLSv1.0", | ||
| TLSVersion11: "TLSv1.1", | ||
| TLSVersion12: "TLSv1.2", | ||
| } |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.
Oops, something went wrong.
Oops, something went wrong.