| fluent-plugin-azurestorage | fluentd | ruby |
|---|---|---|
| >= 0.2.0 | >= v0.14.0 | >= 2.4 |
| < 0.2.0 | >= v0.12.0 | >= 2.1 |
Azure Storate output plugin buffers logs in local file and upload them to Azure Storage periodically. This plugin is porting from fluent-plugin-s3 for AzureStorage.
Install from RubyGems:
$ gem install fluent-plugin-azurestorage
<match pattern>
type azurestorage
azure_storage_account <your azure storage account>
azure_storage_access_key <your azure storage access key>
azure_instance_msi <your azure instance MSI id>
azure_oauth_refresh_interval <your azure MSI token refresh interval in seconds>
azure_container <your azure storage container>
azure_storage_type blob
store_as gzip
auto_create_container true
path logs/
azure_object_key_format %{path}%{time_slice}_%{index}.%{file_extension}
time_slice_format %Y%m%d-%H
# if you want to use ${tag} or %Y/%m/%d/ like syntax in path / s3_object_key_format,
# need to specify tag for ${tag} and time for %Y/%m/%d in <buffer> argument.
<buffer tag,time>
@type file
path /var/log/fluent/azurestorage
timekey 3600 # 1 hour partition
timekey_wait 10m
timekey_use_utc true # use utc
</buffer>
</match>
For <buffer>, you can use any record field in path / azure_object_key_format.
path logs/${tag}/${foo}
<buffer tag,foo>
# parameters...
</buffer>
See official article for more detail: Buffer section configurations
Note that this configuration doesn't work with fluentd v0.12.
<match pattern>
type azurestorage
azure_storage_account <your azure storage account>
azure_storage_access_key <your azure storage access key>
azure_instance_msi <your azure instance MSI id>
azure_oauth_refresh_interval <your azure MSI token refresh interval in seconds>
azure_container <your azure storage container>
azure_storage_type blob
store_as gzip
auto_create_container true
path logs/
azure_object_key_format %{path}%{time_slice}_%{index}.%{file_extension}
buffer_path /var/log/fluent/azurestorage
time_slice_format %Y%m%d-%H
time_slice_wait 10m
utc
</match>
Your Azure Storage Account Name. This can be got from Azure Management potal. This parameter is required when environment variable 'AZURE_STORAGE_ACCOUNT' is not set.
Your Azure Storage Access Key(Primary or Secondary). This also can be got from Azure Management potal. Storage access key authentication is used when this parameter is provided or environment variable 'AZURE_STORAGE_ACCESS_KEY' is set.
Your Azure Managed Service Identity ID. When storage key authentication is not used, the plugin uses OAuth2 to authenticate as given MSI. This authentication method only works on Azure VM. If the VM has only one MSI assigned, this parameter becomes optional and the only MSI will be used. Otherwise this parameter is required.
OAuth2 access token refreshment interval in second. Only applies when MSI authentication is used.
Azure Storage Container name
This plugin create container if not exist when you set 'auto_create_container' to true.
Azure Storage type. Now supports only 'blob'(default). 'tables' and 'queues' are not implemented.
The format of Azure Storage object keys. You can use several built-in variables:
- %{path}
- %{time_slice}
- %{index}
- %{file_extension}
to decide keys dynamically.
%{path} is exactly the value of path configured in the configuration file. E.g., "logs/" in the example configuration above. %{time_slice} is the time-slice in text that are formatted with time_slice_format. %{index} is the sequential number starts from 0, increments when multiple files are uploaded to Azure Storage in the same time slice. %{file_extention} is always "gz" for now.
The default format is "%{path}%{time_slice}_%{index}.%{file_extension}".
For instance, using the example configuration above, actual object keys on Azure Storage will be something like:
"logs/20130111-22_0.gz"
"logs/20130111-23_0.gz"
"logs/20130111-23_1.gz"
"logs/20130112-00_0.gz"
With the configuration:
azure_object_key_format %{path}/events/ts=%{time_slice}/events_%{index}.%{file_extension}
path log
time_slice_format %Y%m%d-%H
You get:
"log/events/ts=20130111-22/events_0.gz"
"log/events/ts=20130111-23/events_0.gz"
"log/events/ts=20130111-23/events_1.gz"
"log/events/ts=20130112-00/events_0.gz"
The fluent-mixin-config-placeholders mixin is also incorporated, so additional variables such as %{hostname}, %{uuid}, etc. can be used in the azure_object_key_format. This could prove useful in preventing filename conflicts when writing from multiple servers.
azure_object_key_format %{path}/events/ts=%{time_slice}/events_%{index}-%{hostname}.%{file_extension}
Archive format on Azure Storage. You can use following types:
- gzip (default)
- json
- text
- lzo (Need lzop command)
- lzma2 (Need xz command)
Change one line format in the Azure Storage object. Supported formats are 'out_file', 'json', 'ltsv' and 'single_value'.
- out_file (default)
time\ttag\t{..json1..}
time\ttag\t{..json2..}
...
- json
{..json1..}
{..json2..}
...
At this format, "time" and "tag" are omitted. But you can set these information to the record by setting "include_tag_key" / "tag_key" and "include_time_key" / "time_key" option. If you set following configuration in AzureStorage output:
format json
include_time_key true
time_key log_time # default is time
then the record has log_time field.
{"log_time":"time string",...}
- ltsv
key1:value1\tkey2:value2
key1:value1\tkey2:value2
...
"ltsv" format also accepts "include_xxx" related options. See "json" section.
- single_value
Use specified value instead of entire recode. If you get '{"message":"my log"}', then contents are
my log1
my log2
...
You can change key name by "message_key" option.
path prefix of the files on Azure Storage. Default is "" (no prefix).
path prefix of the files to buffer logs.
Format of the time used as the file name. Default is '%Y%m%d'. Use '%Y%m%d%H' to split files hourly.
The time to wait old logs. Default is 10 minutes. Specify larger value if old logs may reache.
Use UTC instead of local time.
Azure Storage output plugin is licensed according to the terms of the Apache License, Version 2.0.
The full version of this lisence can be found at http://www.apache.org/licenses/LICENSE-2.0 TXT or HTML
- Fork it
- Create your feature branch (
git checkout -b my-new-feature) - Commit your changes (
git commit -am 'Add some feature') - Push to the branch (
git push origin my-new-feature) - Create new Pull Request