Configure Data Sources in a Log Analytics Workspace

[josh-barker]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Description

I'd like to be able to configure Data Sources on a Log Analytics Workspace

New or Affected Resource(s)

• azurerm_log_analytics_workspace

and/or:

• azurerm_log_analytics_workspace_iis_logs
• azurerm_log_analytics_workspace_linux_performance_collection
• azurerm_log_analytics_workspace_linux_performance_object
• azurerm_log_analytics_workspace_linux_syslog
• azurerm_log_analytics_workspace_linux_syslogCollection
• azurerm_log_analytics_workspace_windows_event
• azurerm_log_analytics_workspace_windows_performance_counter

Note: There may be other data sources and properties that can be configured.

Potential Terraform Configuration

Inline example of configuring "datasource" properties in a Log Analytics workspace

resource "azurerm_log_analytics_workspace" "ws" {
  name                = "Demo"
  location            = "US East"
  resource_group_name = "Some-Resource-Group"
  sku                 = "Standard"

  windows_performance_counter { 
    object_name = "LogicalDisk"
    instance_name = "*"
    interval_seconds = 10
    counter_name = "Disk Writes/sec"
  }

  windows_performance_counter { 
    object_name = "LogicalDisk"
    instance_name = "*"
    interval_seconds = 10
    counter_name = "Free Megabytes"
  }

  linux_performance_collection {
    state = "Enabled"
  }

  linux_performance_object {
    performance_counters = ["% Processor Time", "% Privileged Time"]
    object_name = "Processor"
    instance_name = "*"
    interval_seconds = 10
  }
}

Configure "datasource" using a separate resource for a Log Analytics workspace.

resource "azurerm_log_analytics_workspace_iis_logs" "iis_logs" {
  name                = "iis_logs"
  resource_group_name = "Some-Resource-Group"

  state = "OnPremiseEnabled"
}

resource "azurerm_log_analytics_workspace_linux_performance_collection" "lin_perf_collection" {
  name                = "lin_perf_collection"
  resource_group_name = "Some-Resource-Group"

  state = "Enabled"
}

resource "azurerm_log_analytics_workspace_linux_performance_object" "lin_processor_performance" {
  name                = "lin_processor_performance"
  resource_group_name = "Some-Resource-Group"

  performance_counters = ["% Processor Time", "% Privileged Time"]
  object_name = "Processor"
  instance_name = "*"
  interval_seconds = 10
}

resource "azurerm_log_analytics_workspace_linux_syslog_collection" "lin_syslog_collection" {
  name                = "lin_syslog_collection"
  resource_group_name = "Some-Resource-Group"

  state = "Enabled"
}

resource "azurerm_log_analytics_workspace_linux_syslog_collection" "lin_syslog_collection" {
  name                = "lin_syslog_collection"
  resource_group_name = "Some-Resource-Group"

  syslog_name = "kern"
  severities = ["emerg", "alert", "crit", "err", "warning"]
}

resource "azurerm_log_analytics_workspace_windows_event" "win_event" {
  name                = "win_event"
  resource_group_name = "Some-Resource-Group"
  
  eventLogName = "Application"
  eventTypes = ["Error", "Warning"]
}

resource "azurerm_log_analytics_workspace_windows_performance_counter" "win_disk_writes" {
  name                = "win_disk_writes"
  resource_group_name = "Some-Resource-Group"

  object_name = "LogicalDisk"
  instance_name = "*"
  interval_seconds = 10
  counter_name = "Disk Writes/sec"
}

[laughtonsm]

This would be an awesome feature. Currently looking to do this and surprised to find no support already

[russjury]

I'd also like to see azurerm_log_analytics_workspace_custom_logs and azurerm_log_analytics_workspace_custom_fields be supported in this feature too - I think the mechanism is similar enough to be included. I can help with examples/how you'd do it via the API. An example potential implementation (there's more to it, but skipped for brevity):

resource "azurerm_log_analytics_workspace_custom_logs" "customlog01_CL" {
  name = "customlog01_CL"
  resource_group_name = "Some-Resource-Group"
  workspace_id = "00000000-0000-0000-0000-000000000000"
  description = "Description of custom log"
  extractions = []
  inputs = []
}

[tombuildsstuff]

hi @GCole64

Taking a look at the comments posted above, since they're coming in via email unfortunately we believe that you may be responding to the wrong email? This issue tracker is intended to track/discuss bugs and enhancements in the Terraform Provider for Microsoft Azure - as such we try to limit off-topic conversations.

Thanks!

[diegoitaliait]

Hi,
are there any new features for this new feature?

[ghost]

looking forward for this feature

[ssrirama-aka]

My requirement is to use Terraform to link an activity log as data source into a given log analytics workspace. I started looking at issue https://github.com/terraform-providers/terraform-provider-azurerm/issues/4446 and followed it here. What I don't see captured yet in this issue is the need to configure Azure's Activity Log as a data source in a workspace.

Thinking along the lines that @josh-barker outlined, we could define a resource like this:

• azurerm_log_analytics_workspace_activity_log

With properties like this:

resource "azurerm_log_analytics_workspace_activity_log" "my_activity_log" {
  name                  = "some_name"
  subscription_id   = "subscriptionId"
  workspace_id      = "workspaceId"
}

[ssrirama-aka]

@tombuildsstuff Given the votes and utility value of this, could the enhancement be targeted to a 2.x release of the AzureRM provider?

[magodo]

I'll try to implement this, hopefully.

[patrickbsf]

I don't see how this closes out this case. There are still quite a view datasource types to add - it seems like #6321 only handles windows performance counters and windows events.
syslog, custom logs, etc. all still need handled.

[katbyte]

thanks @patrickbsf, looks like this was automagically closed by that PR. reopening!

[magodo]

I'm wondering whether it make sense to just embed azurerm_log_analytics_workspace_linux_performance_collection and azurerm_log_analytics_workspace_linux_syslog_collection into azurerm_log_analytics_workspace_linux_performance_counter and azurerm_log_analytics_workspace_linux_syslog respectively. As it is rare to setup several "linux performance counter"/"linux syslog" data sources while disable them.

[magodo]

Currently, the progress of this issue is blocked by Azure/azure-rest-api-specs#9072. Once that issue has been addressed, we can move on to implementing the remaining data sources.

[PriyankaRanganath]

Currently, the progress of this issue is blocked by Azure/azure-rest-api-specs#9072. Once that issue has been addressed, we can move on to implementing the remaining data sources.

We have implemented above requested features as separate resources and raised PR for the same.
Regarding the REST API spec issue, We have referred equivalent powershell cmdlets for the log analytics data sources and implemented all properties accordingly.:

1. Linux syslog
2. Linux syslog collection enable and disable
3. Linux performance object
4. Linux performance collection
   The PR needs review and approval.

@magodo / @tombuildsstuff, could you kindly review and approve the same?

[tombuildsstuff]

This is still blocked on the Service Team: Azure/azure-rest-api-specs#9072

[premaarya]

My requirement is to use Terraform to link an activity log as data source into a given log analytics workspace. I started looking at issue https://github.com/terraform-providers/terraform-provider-azurerm/issues/4446 and followed it here. What I don't see captured yet in this issue is the need to configure Azure's Activity Log as a data source in a workspace.

Thinking along the lines that @josh-barker outlined, we could define a resource like this:

• azurerm_log_analytics_workspace_activity_log

With properties like this:

resource "azurerm_log_analytics_workspace_activity_log" "my_activity_log" {
  name                  = "some_name"
  subscription_id   = "subscriptionId"
  workspace_id      = "workspaceId"
}

I have a similar requirement. Will it be a good solution to run the ARM template('azurerm_template_deployment') in my terraform script to configure the activity log collection to log analytics?

[zioalex]

HI All, any progress here?

[dsab123]

Hey friends, thanks to the new azapi provider from microsoft I was able to terraform custom logs!

resource "azapi_resource" "VarLogMessages-Ingest" {
  provider  = azapi
  type      = "Microsoft.OperationalInsights/workspaces/dataSources@2020-08-01"
  name      = "VarLogMessages-Ingest"
  parent_id = log-analytics-workspace.id

  body = jsonencode({
    properties = {
      customLogName = "VarLogMessages_CL"
      description = "Captures the /var/log/messages of the linux machines"
      inputs = [{
        location = {
          fileSystemLocations = {
            linuxFileTypeLogPaths = ["/var/log/messages"],
          }
        },
        recordDelimiter = {
          regexDelimiter = {
            pattern = "\\n",
            matchIndex = 0,
            numberdGroup = null # lol MSFT typo
          }
        }
        }
      ],
      extractions = [
        {
          extractionName = "TimeGenerated",
          extractionType = "DateTime",
          extractionProperties = {
            dateTimeExtraction = {
              regex = null,
              joinStringRegex = null
            }
          }
        }
      ]
    }
    kind = "CustomLog"
  })
}

NOTE: this provider is still in preview. The logs don't seem to be imported into the state which is a bit of a pain. I'm going to move these resources to a separate module to segregate them out from the rest of my architecture.

ALSO NOTE: your VM will need a couple agents running to ship the logs. See this blog post by thorsten hans on that those are

The MSFT blog post announcing the provider is here; I'm working on a blog post on how I got to the above HCL which should be out in a few days.

[jakeprice-me]

Here's the blog post @dsab123's said he was working on, summarising how he got this to work:

• Linux VM Custom Logs with Terraform and AzApi | Sabbagh's Blagh

[aristosvo]

I was wondering if azurerm_monitor_data_collection_rule is an implementation of this functionality or that this is a different kind of feature

[jakeprice-me]

I think so @aristosvo, but it doesn't fully support all the possible data sources at the moment (such as custom text file logs).

Or at least it's not documented (and I haven't been able to get it to work with custom text file logs).

[danielguo72]

Hi

Is there a plan to add custom log creation to the log_analytic_workspace? Given DCE and DCR are now the favourable option to write logs into log analytic workspace, creating custom table will be essential and would love to see azurerm provider to support this natively instead of using azapi. Thanks