Skip to content

docs(azure_blob): add managed identity and workload identity auth #2381

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
temporaer wants to merge 1 commit into
base: master
Choose a base branch
from
Open

docs(azure_blob): add managed identity and workload identity auth #2381

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
185 changes: 184 additions & 1 deletion pipeline/outputs/azure_blob.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,9 +19,12 @@
| Key | Description | Default |
| :--------------------------- | :--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | :---------------------------- |
| `account_name` | Azure Storage account name. | _none_ |
| `auth_type` | Specify the type to authenticate against the service. Supported values: `key`, `sas`. | `key` |
| `auth_type` | Specify the type to authenticate against the service. Supported values: `key`, `sas`, `managed_identity`, `workload_identity`. | `key` |
| `shared_key` | Specify the Azure Storage Shared Key to authenticate against the service. This configuration property is mandatory when `auth_type` is `key`. | _none_ |
| `sas_token` | Specify the Azure Storage shared access signatures to authenticate against the service. This configuration property is mandatory when `auth_type` is `sas`. | _none_ |
| `client_id` | Azure client ID for managed identity or workload identity authentication. For system-assigned managed identity, set to `system`. Required when `auth_type` is `managed_identity` or `workload_identity`. | _none_ |
| `tenant_id` | Azure tenant ID. Required when `auth_type` is `workload_identity`. | _none_ |
| `workload_identity_token_file` | Path to the projected service account token file for workload identity authentication. Only used when `auth_type` is `workload_identity`. | `/var/run/secrets/azure/tokens/azure-identity-token` |
| `container_name` | Name of the container that will contain the blobs. | _none_ |
| `blob_type` | Specify the desired blob type. Supported values: `appendblob`, `blockblob`. | `appendblob` |
| `auto_create_container` | If `container_name` doesn't exist in the remote service, enabling this option handles the exception and auto-creates the container. | `on` |
Expand Down Expand Up @@ -111,6 +114,186 @@

![Azure Blob](../../.gitbook/assets/azure_blob.png)

### Configuration for Managed Identity

Check warning on line 117 in pipeline/outputs/azure_blob.md

View workflow job for this annotation

GitHub Actions / runner / vale 

[vale] reported by reviewdog 🐶
[FluentBit.Headings] 'Configuration for Managed Identity' should use sentence-style capitalization.

Raw Output:
{"message": "[FluentBit.Headings] 'Configuration for Managed Identity' should use sentence-style capitalization.", "location": {"path": "pipeline/outputs/azure_blob.md", "range": {"start": {"line": 117, "column": 5}}}, "severity": "INFO"}

Azure Managed Identity lets your application authenticate to Azure Blob Storage without managing credentials. This works on Azure VMs, Azure Container Instances, Azure App Service, and other Azure compute services with managed identity support.

#### System-assigned managed identity

{% tabs %}
{% tab title="fluent-bit.yaml" %}

```yaml
service:
flush: 1
log_level: info

pipeline:
inputs:
- name: dummy
dummy: '{"name": "Fluent Bit", "year": 2024}'
samples: 1
tag: var.log.containers.app-default-96cbdef2340.log

outputs:
- name: azure_blob
match: "*"
account_name: YOUR_ACCOUNT_NAME
auth_type: managed_identity
client_id: system
container_name: logs
auto_create_container: on
tls: on
```

{% endtab %}
{% tab title="fluent-bit.conf" %}

```text
[SERVICE]
flush 1
log_level info

[INPUT]
name dummy
dummy {"name": "Fluent Bit", "year": 2024}
samples 1
tag var.log.containers.app-default-96cbdef2340.log

[OUTPUT]
name azure_blob
match *
account_name YOUR_ACCOUNT_NAME
auth_type managed_identity
client_id system
container_name logs
auto_create_container on
tls on
```

{% endtab %}
{% endtabs %}

#### User-assigned managed identity

For user-assigned managed identities, set `client_id` to the client ID (UUID) of the managed identity:

{% tabs %}
{% tab title="fluent-bit.yaml" %}

```yaml
service:
flush: 1
log_level: info

pipeline:
inputs:
- name: dummy
dummy: '{"name": "Fluent Bit", "year": 2024}'
samples: 1
tag: var.log.containers.app-default-96cbdef2340.log

outputs:
- name: azure_blob
match: "*"
account_name: YOUR_ACCOUNT_NAME
auth_type: managed_identity
client_id: YOUR_MANAGED_IDENTITY_CLIENT_ID
container_name: logs
auto_create_container: on
tls: on
```

{% endtab %}
{% tab title="fluent-bit.conf" %}

```text
[SERVICE]
flush 1
log_level info

[INPUT]
name dummy
dummy {"name": "Fluent Bit", "year": 2024}
samples 1
tag var.log.containers.app-default-96cbdef2340.log

[OUTPUT]
name azure_blob
match *
account_name YOUR_ACCOUNT_NAME
auth_type managed_identity
client_id YOUR_MANAGED_IDENTITY_CLIENT_ID
container_name logs
auto_create_container on
tls on
```

{% endtab %}
{% endtabs %}

### Configuration for Workload Identity

Check warning on line 235 in pipeline/outputs/azure_blob.md

View workflow job for this annotation

GitHub Actions / runner / vale 

[vale] reported by reviewdog 🐶
[FluentBit.Headings] 'Configuration for Workload Identity' should use sentence-style capitalization.

Raw Output:
{"message": "[FluentBit.Headings] 'Configuration for Workload Identity' should use sentence-style capitalization.", "location": {"path": "pipeline/outputs/azure_blob.md", "range": {"start": {"line": 235, "column": 5}}}, "severity": "INFO"}

Azure Workload Identity lets pods in Azure Kubernetes Service (AKS) authenticate to Azure Blob Storage using a Kubernetes service account federated with Azure AD.

{% tabs %}
{% tab title="fluent-bit.yaml" %}

```yaml
service:
flush: 1
log_level: info

pipeline:
inputs:
- name: dummy
dummy: '{"name": "Fluent Bit", "year": 2024}'
samples: 1
tag: var.log.containers.app-default-96cbdef2340.log

outputs:
- name: azure_blob
match: "*"
account_name: YOUR_ACCOUNT_NAME
auth_type: workload_identity
client_id: YOUR_CLIENT_ID
tenant_id: YOUR_TENANT_ID
container_name: logs
auto_create_container: on
tls: on
```

{% endtab %}
{% tab title="fluent-bit.conf" %}

```text
[SERVICE]
flush 1
log_level info

[INPUT]
name dummy
dummy {"name": "Fluent Bit", "year": 2024}
samples 1
tag var.log.containers.app-default-96cbdef2340.log

[OUTPUT]
name azure_blob
match *
account_name YOUR_ACCOUNT_NAME
auth_type workload_identity
client_id YOUR_CLIENT_ID
tenant_id YOUR_TENANT_ID
container_name logs
auto_create_container on
tls on
```

{% endtab %}
{% endtabs %}

The `workload_identity_token_file` option can be set to override the default token file path if your AKS cluster mounts the projected service account token at a non-standard location.

Check warning on line 295 in pipeline/outputs/azure_blob.md

View workflow job for this annotation

GitHub Actions / runner / vale 

[vale] reported by reviewdog 🐶
[FluentBit.WordList] Use 'path' instead of 'file path'.

Raw Output:
{"message": "[FluentBit.WordList] Use 'path' instead of 'file path'.", "location": {"path": "pipeline/outputs/azure_blob.md", "range": {"start": {"line": 295, "column": 84}}}, "severity": "INFO"}

### Configuring and using Azure Emulator: Azurite

#### Install and run Azurite
Expand Down
Loading