[Azure Module] - Support for new AD SignIn Logs

[jamiehynds]

Up until recently, Azure AD sign-in logs were limited to interactive sign ins, however Microsoft recently added support for additional SignIn types including:

• NonInteractive
• ServicePrincipal
• ManagedIdentity

Related Elastic discuss issue: https://discuss.elastic.co/t/filebeat-azure-module-additional-azure-ad-log-sources/262026

In order to reduce AzureAD sign-in blindspots, this issue is aimed to track progress on support for these new signin types within our Azure module.

Relevant lines in ingest pipelines:

https://github.com/elastic/integrations/blob/1a817c4d9003a97c4a663e123fe6582a2f5fdf57/packages/azure/data_stream/auditlogs/elasticsearch/ingest_pipeline/default.yml#L21-L22

https://github.com/elastic/integrations/blob/1a817c4d9003a97c4a663e123fe6582a2f5fdf57/packages/azure/data_stream/signinlogs/elasticsearch/ingest_pipeline/default.yml#L17-L18

[elasticmachine]

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

[matthew-lubbers]

I've attached some log examples that are from my personal Azure tenant that should help.

** Logs Redacted **

Here are the missing log sources:

• Managed Identity Sign-ins - https://docs.microsoft.com/en-us/azure/azure-monitor/reference/tables/aadmanagedidentitysigninlogs
• Non-Interactive User Sign-ins - https://docs.microsoft.com/en-us/azure/azure-monitor/reference/tables/aadnoninteractiveusersigninlogs
• Provisioning Logs - https://docs.microsoft.com/en-us/azure/azure-monitor/reference/tables/aadprovisioninglogs
• Service Principal Sign-ins - https://docs.microsoft.com/en-us/azure/azure-monitor/reference/tables/aadserviceprincipalsigninlogs

Let me know if you need any additional data in regards to this feature enhancement.

Thanks!

[andrewkroh]

@matthew-lubbers Those logs you provided are a big help. I'm curious about the sample for NonInteractiveUserSignInLogs. Does callerIpAddress literally contain this string of ipAddress? Or did you do some anonymization on this sample first and changed the value?

  "callerIpAddress": "ipAddress",
  "properties": {
    "ipAddress": "172.221.235.105",

[matthew-lubbers]

@andrewkroh - The callerIpAddress contains the literal string of the ipAddress. That ipAddress was not anonymized but it should have been. I've went ahead and removed the previous .zip file with my sample logs attached.

[andrewkroh]

I am wondering where the AAD Provisioning Logs fit into the module. Would those fit into an existing fileset? Should they be handled by a new fileset? I'm not familiar with Azure or how a user configures log exports on the Azure side. How granular should the exports be? Are we making it harder to ingest logs if we require separate eventhubs for each type?

[matthew-lubbers]

When you say an fileset, what do you mean?

Log exports are configured using Diagnostic Settings on either an Azure resource, an Azure subscription, or an Azure AD tenant (this configuration). If you configure the Diagnostic Settings to forward to an Event Hub and have the "Create in selected namespace" option, after an log arrives to the Event Hub Namespace, the Event Hub will be auto-generated. If the "Create in selected namespace" is not selected/configured, an Event Hub will need to be created per log type. This is not required per say but, that does mean Logstash/Filebeat/etc. needs to understand the various schemas and process those log types accordingly.

[andrewkroh]

When you say an fileset, what do you mean?

Fileset (also sometimes called "dataset") is one data source within a module. The current filesets of the Azure module would be activitylogs, auditlogs, platformlogs, signinlogs.

So I'm wondering if AAD Provisioning logs should have their own fileset within the Azure module or would those logs be routed to one of the existing Event Hubs already handled by one of the filesets.

[matthew-lubbers]

@andrewkroh - I believe each of these new log sources could be their own fileset under the Azure module.

[andrewkroh]

The new sign-in log types are implemented in the module (7.16) and Fleet integration.

Provisioning Logs will be addressed separately via #28550.