Add metrics datastream for Azure Functions

packages/azure_functions/_dev/build/docs/README.md

@@ -6,19 +6,22 @@ Use this integration to build web APIs, respond to database changes, process IoT
 
 
 ## Data streams
+The Azure Functions integration contains two data streams: [Function App Logs](#logs) and [Metrics](#metrics)
+
+### Logs
+
 Supported log categories:
 
 | Log Category                 | Description                                                                                                                          |
 |:----------------------------:|:------------------------------------------------------------------------------------------------------------------------------------:|
 | Functionapplogs | Function app logs.        |
 
 
-## Requirements and setup
+#### Requirements and setup
 
 Refer to the [Azure Logs](https://docs.elastic.co/integrations/azure) page for more information about setting up and using this integration.
 
-## Settings
-
+#### Configuration options
 `eventhub` :
   _string_
 An Event Hub is a fully managed, real-time data ingestion service. Elastic recommends using only letters, numbers, and the hyphen (-) character for Event Hub names to maximize compatibility. You can use existing Event Hubs having underscores (_) in the Event Hub name; in this case, the integration will replace underscores with hyphens (-) when it uses the Event Hub name to create dependent Azure resources behind the scenes (e.g., the storage account container to store Event Hub consumer offsets). Elastic also recommends using a separate event hub for each log type as the field mappings of each log type differ.
@@ -67,8 +70,133 @@ https://management.azure.com/
 https://management.usgovcloudapi.net/
 ```
 
-## Logs
-
 {{event "functionapplogs"}}
 
-{{fields "functionapplogs"}}
+{{fields "functionapplogs"}}
+
+### Metrics
+**Metrics** give you insight into the performance of your Azure Function Apps. The integration includes an out-of-the-box dashboard for visualising the monitoring data generated by apps hosted in Azure Functions.
+
+#### Requirements
+
+To use this integration you will need:
+
+* **Azure App Registration**: You need to set up an Azure App Registration to allow the Agent to access the Azure APIs. The App Registration requires the Monitoring Reader role to access to be able to collect metrics from Function Apps. See more details in the Setup section.
+* **Elasticsearch and Kibana**: You need Elasticsearch to store and search your data and Kibana to visualize and manage it. You can use our hosted Elasticsearch Service on Elastic Cloud, which is recommended, the [Native Azure Integration](https://azuremarketplace.microsoft.com/en-us/marketplace/apps/elastic.elasticsearch?tab=Overview), or self-manage the Elastic Stack on your hardware.
+
+#### Setup
+
+
+```text
+         ┌────────────────────┐       ┌─────────┐       ┌─-─────────────────────┐
+         │                    │       │         │       │    azure.functions    │
+         │     Azure APIs     │──────▶│  Agent  │──────▶│    <<data stream>>    │
+         │                    │       │         │       │                       │
+         └────────────────────┘       └─────────┘       └───-───────────────────┘                                              
+```
+
+Elastic Agent needs an App Registration to access Azure on your behalf to collect data using the Azure REST APIs. App Registrations are required to access Azure APIs programmatically.
+
+To start collecting data with this integration, you need to:
+
+* Set up a new Azure app registration by registering an app, adding credentials, and assigning an appropriate role.
+* Specify integration [settings](#main-options) in Kibana, which will determine how the integration will access the Azure APIs.
+
+#### Register a new app
+
+To create a new app registration:
+
+1. Sign in to the [Azure Portal](https://portal.azure.com/).
+2. Search for and select **Azure Active Directory**.
+3. Under **Manage**, select **App registrations** > **New registration**.
+4. Enter a display _Name_ for your application (for example, "elastic-agent").
+5. Specify who can use the application.
+6. Don't enter anything for _Redirect URI_. This is optional and the agent doesn't use it.
+7. Select **Register** to complete the initial app registration.
+
+Take note of the **Application (client) ID**, which you will use later when specifying the **Client ID** in the integration settings.
+
+#### Add credentials
+
+Credentials allow your application to access Azure APIs and authenticate itself, requiring no interaction from a user at runtime.
+
+This integration uses Client Secrets to prove its identity.
+
+1. In the [Azure Portal](https://portal.azure.com/), select the application you created in the previous section.
+2. Select **Certificates & secrets** > **Client secrets** > **New client secret**.
+3. Add a description (for example, "Elastic Agent client secrets").
+4. Select an expiration for the secret or specify a custom lifetime.
+5. Select **Add**.
+
+Take note of the content in the **Value** column in the **Client secrets** table, which you will use later when specifying a **Client Secret** in the integration settings. **This secret value is never displayed again after you leave this page.** Record the secret's value in a safe place.
+
+#### Assign role
+
+1. In the [Azure Portal](https://portal.azure.com/), search for and select **Subscriptions**.
+2. Select the subscription to assign the application.
+3. Select **Access control (IAM)**.
+4. Select **Add** > **Add role assignment** to open the _Add role assignment page_.
+5. In the **Role** tab, search and select the role **Monitoring Reader**.
+6. Select the **Next** button to move to the **Members** tab.
+7. Select **Assign access to** > **User, group, or service principal**, and select **Select members**. This page does not display Azure AD applications in the available options by default.
+8. To find your application, search by name (for example, "elastic-agent") and select it from the list.
+9. Click the **Select** button.
+10. Then click the **Review + assign** button.
+
+Take note of the following values, which you will use later when specifying settings.
+
+* `Subscription ID`: use the content of the "Subscription ID" you selected.
+* `Tenant ID`: use the "Tenant ID" from the Azure Active Directory you use.
+
+Your App Registration is now ready to be used with the Elastic Agent.
+
+#### Additional Resources
+
+If you want to learn more about this process, you can read these two general guides from Microsoft:
+
+* [Quickstart: Register an application with the Microsoft identity platform](https://docs.microsoft.com/en-us/azure/active-directory/develop/quickstart-register-app) 
+* [Use the portal to create an Azure AD application and service principal that can access resources](https://docs.microsoft.com/en-us/azure/active-directory/develop/howto-create-service-principal-portal)
+
+#### Main options
+
+The settings' main section contains all the options needed to access the Azure APIs and collect the Azure Functions metrics data. You will now use all the values from [App registration](#register-a-new-app) including:
+
+`Client ID` _string_
+: The unique identifier of the App Registration (sometimes referred to as Application ID).
+
+`Client Secret` _string_
+: The client secret for authentication.
+
+`Subscription ID` _string_
+: The unique identifier for the Azure subscription. You can provide just one subscription ID. The Agent uses this ID to access Azure APIs. 
+
+`Tenant ID` _string_
+: The unique identifier of the Azure Active Directory's Tenant ID.
+
+#### Advanced options
+
+There are two additional advanced options:
+
+`Resource Manager Endpoint` _string_
+: Optional. By default, the integration uses the Azure public environment. To override, users can provide a specific resource manager endpoint to use a different Azure environment.
+
+Examples:
+
+* `https://management.chinacloudapi.cn` for Azure ChinaCloud
+* `https://management.microsoftazure.de` for Azure GermanCloud
+* `https://management.azure.com` for Azure PublicCloud
+* `https://management.usgovcloudapi.net` for Azure USGovernmentCloud
+
+`Active Directory Endpoint`  _string_
+: Optional. By default, the integration uses the associated Active Directory Endpoint. To override, users can provide a specific active directory endpoint to use a different Azure environment.
+
+Examples:
+
+* `https://login.chinacloudapi.cn` for Azure ChinaCloud
+* `https://login.microsoftonline.de` for Azure GermanCloud
+* `https://login.microsoftonline.com` for Azure PublicCloud
+* `https://login.microsoftonline.us` for Azure USGovernmentCloud
+
+#### Metrics Reference
+
+{{fields "metrics"}}

packages/azure_functions/changelog.yml

@@ -1,4 +1,9 @@
 # newer versions go on top
+- version: "0.1.1"
+  changes:
+    - description: Add Azure Functions metrics data stream and a Kibana dashboard
+      type: enhancement
+      link: https://github.com/elastic/integrations/pull/7130
 - version: "0.0.1"
   changes:
     - description: Add Azure Functions logs data stream

packages/azure_functions/data_stream/functionapplogs/manifest.yml

@@ -7,6 +7,45 @@ streams:
     title: Azure Functions logs
     description: Collect Azure Functions logs using azure-eventhub input
     vars:
+      - name: eventhub
+        type: text
+        title: Event Hub
+        multi: false
+        required: true
+        show_user: true
+        description: >-
+          Elastic recommends using one event hub for each integration. Visit [Create an event hub](https://docs.elastic.co/integrations/azure#create-an-event-hub) to learn more. Use event hub names up to 30 characters long to avoid compatibility issues.
+      - name: consumer_group
+        type: text
+        title: Consumer Group
+        multi: false
+        required: true
+        show_user: true
+        default: $Default
+      - name: connection_string
+        type: password
+        title: Connection String
+        multi: false
+        required: true
+        show_user: true
+        description: >-
+          The connection string required to communicate with Event Hubs. See [Get an Event Hubs connection string](https://docs.microsoft.com/en-us/azure/event-hubs/event-hubs-get-connection-string) to learn more.
+      - name: storage_account
+        type: text
+        title: Storage Account
+        multi: false
+        required: true
+        show_user: true
+        description: >-
+          The name of the storage account where the consumer group's state/offsets will be stored and updated.
+      - name: storage_account_key
+        type: password
+        title: Storage Account Key
+        multi: false
+        required: true
+        show_user: true
+        description: >-
+          The storage account key, this key will be used to authorize access to data in your storage account.
       - name: preserve_original_event
         required: true
         show_user: true

packages/azure_functions/data_stream/metrics/_dev/test/pipeline/test-common-config.yml

@@ -0,0 +1,2 @@
+dynamic_fields:
+  event.ingested: ".*"

packages/azure_functions/data_stream/metrics/agent/stream/stream.yml.hbs

@@ -0,0 +1,70 @@
+metricsets: ["monitor"]
+default_resource_type: "Microsoft.Web/sites"
+add_cloud_metadata: true
+period: {{period}}
+{{#if client_id}}
+client_id: {{client_id}}
+{{/if}}
+{{#if client_secret}}
+client_secret: {{client_secret}}
+{{/if}}
+{{#if tenant_id}}
+tenant_id: {{tenant_id}}
+{{/if}}
+{{#if subscription_id}}
+subscription_id: {{subscription_id}}
+{{/if}}
+{{#if refresh_list_interval}}
+refresh_list_interval: {{refresh_list_interval}}
+{{/if}}
+{{#if resource_manager_endpoint}}
+resource_manager_endpoint: {{resource_manager_endpoint}}
+{{/if}}
+{{#if active_directory_endpoint}}
+active_directory_endpoint: {{active_directory_endpoint}}
+{{/if}}
+
+resources:
+{{#if resource_groups}}
+{{#each resource_groups}}
+    - resource_group: "{{this}}"
+      resource_type: "Microsoft.Web/sites"
+      metrics:
+        - name: ["FunctionExecutionCount", "FunctionExecutionUnits", "AppConnections", "AverageMemoryWorkingSet", "BytesReceived", "BytesSent", "CurrentAssemblies", "Gen0Collections", "Gen1Collections", "Gen2Collections", "Handles", "HealthCheckStatus", "Http2xx", "Http3xx", "Http4xx", "Http5xx", "HttpResponseTime", "IoOtherBytesPerSecond", "IoOtherOperationsPerSecond", "IoReadBytesPerSecond", "IoReadOperationsPerSecond", "IoWriteBytesPerSecond", "IoWriteOperationsPerSecond", "MemoryWorkingSet", "Requests", "RequestsInApplicationQueue", "Threads", "TotalAppDomains", "TotalAppDomainsUnloaded"]
+          namespace: "Microsoft.Web/sites"
+          timegrain: "PT5M"
+          ignore_unsupported: true
+        - name: ["FileSystemUsage"]
+          namespace: "Microsoft.Web/sites"
+          timegrain: "PT6H"
+          ignore_unsupported: true
+{{/each}}
+{{/if}}
+{{#if resource_ids}}
+{{#each resource_ids}}
+    - resource_id: "{{this}}"
+      metrics:
+        - name: ["FunctionExecutionCount", "FunctionExecutionUnits", "AppConnections", "AverageMemoryWorkingSet", "BytesReceived", "BytesSent", "CurrentAssemblies", "Gen0Collections", "Gen1Collections", "Gen2Collections", "Handles", "HealthCheckStatus", "Http2xx", "Http3xx", "Http4xx", "Http5xx", "HttpResponseTime", "IoOtherBytesPerSecond", "IoOtherOperationsPerSecond", "IoReadBytesPerSecond", "IoReadOperationsPerSecond", "IoWriteBytesPerSecond", "IoWriteOperationsPerSecond", "MemoryWorkingSet", "Requests", "RequestsInApplicationQueue", "Threads", "TotalAppDomains", "TotalAppDomainsUnloaded"]
+          namespace: "Microsoft.Web/sites"
+          timegrain: "PT5M"
+          ignore_unsupported: true
+        - name: ["FileSystemUsage"]
+          namespace: "Microsoft.Web/sites"
+          timegrain: "PT6H"
+          ignore_unsupported: true
+{{/each}}
+{{/if}}
+{{#unless resource_ids }}
+    {{#unless resource_groups }}
+    - resource_query: "resourceType eq 'Microsoft.Web/sites'"
+      metrics:
+        - name: ["FunctionExecutionCount", "FunctionExecutionUnits", "AppConnections", "AverageMemoryWorkingSet", "BytesReceived", "BytesSent", "CurrentAssemblies", "Gen0Collections", "Gen1Collections", "Gen2Collections", "Handles", "HealthCheckStatus", "Http2xx", "Http3xx", "Http4xx", "Http5xx", "HttpResponseTime", "IoOtherBytesPerSecond", "IoOtherOperationsPerSecond", "IoReadBytesPerSecond", "IoReadOperationsPerSecond", "IoWriteBytesPerSecond", "IoWriteOperationsPerSecond", "MemoryWorkingSet", "Requests", "RequestsInApplicationQueue", "Threads", "TotalAppDomains", "TotalAppDomainsUnloaded"]
+          namespace: "Microsoft.Web/sites"
+          timegrain: "PT5M"
+          ignore_unsupported: true
+        - name: ["FileSystemUsage"]
+          namespace: "Microsoft.Web/sites"
+          timegrain: "PT6H"
+          ignore_unsupported: true
+    {{/unless}}
+{{/unless}}

packages/azure_functions/data_stream/metrics/elasticsearch/ingest_pipeline/default.yml

@@ -0,0 +1,17 @@
+---
+description: Pipeline for parsing Azure Functions metrics.
+processors:
+  - set:
+      field: ecs.version
+      value: "8.0.0"
+  - rename:
+      field: azure.monitor
+      target_field: azure.functions
+      ignore_missing: true
+on_failure:
+  - set:
+      field: event.kind
+      value: pipeline_error
+  - append:
+      field: error.message
+      value: '{{ _ingest.on_failure_message }}'