[Security Solution] Strange Built-In Rule Behavior

[ossie-git]

Describe the bug:

While testing some of the pre-built rules, I noticed that they weren't triggering as expected. Recreating the same rule resulted in my own custom rule triggering correctly

Kibana/Elasticsearch Stack version:

Elastic Cloud - 7.17.0

Server OS version:

Elastic Cloud - 7.17.0

Endpoint running winlogbeat is Windows 10 Enterprise

Browser and Browser OS versions:

Chrome 98

Elastic Endpoint version:

Elastic Cloud - 7.17.0

Original install method (e.g. download page, yum, from source, etc.):

chocolatey on Windows

Functional Area (e.g. Endpoint management, timelines, resolver, etc.):

Steps to reproduce:

1. Enable the Whoami Process Activity rule (https://www.elastic.co/guide/en/security/7.17/whoami-process-activity.html)
2. Run whoami on an endpoint running winlogbeat
3. Check the alerts once the rule rules. You will not find any
4. Create your own rule with the same logic
5. Your own rule triggers an alert when you run whoami on an endpoint

Current behavior:

No alerts are triggers

Expected behavior:

An alert should be triggered

Screenshots (if relevant):

Here is what the query in the default rule looks like:

I recreated the rule with the same identical query (but a faster trigger but that shouldn't make a difference):

When I then run whoami several times on the endpoint, my rule triggers but the prebuilt rule does not:

Errors in browser console (if relevant):

Provide logs and/or server output (if relevant):

Any additional context (logs, chat logs, magical formulas, etc.):

[elasticmachine]

Pinging @elastic/security-solution (Team: SecuritySolution)

[peluja1012]

Hi @ossie-git, thank you for reporting this issue. Could you please share with us the rule configuration njson content for each of those rules? You can get that by using our Rule Export feature.

[ossie-git]

Hi @peluja1012,

HYG.

Here is the ndjson file for my custom rule: rules_export.ndjson.zip. However, I couldn't export the prebuilt rule (the link above also mentions this: "You cannot export Elastic prebuilt rules."). Thanks

[peluja1012]

Hi @ossie-git, I apologize for the confusion. In order for us to take a look at the contents of your prebuilt rule, could you please execute the following query in Kibana -> Dev tools and send us the output?

GET .kibana*/_search
{
  "query": {
    "bool": {
      "must": [
        { "match": { "type": "alert" } },
        { "match": { "alert.name.keyword": "Whoami Process Activity" } }
      ]
    }
  }
}

[ossie-git]

Hi @peluja1012,

HYG.

#! this request accesses system indices: [.kibana_7.17.0_001, .kibana_security_session_1, .kibana_task_manager_7.17.0_001], but in a future major version, direct access to system indices will be prevented by default
{
  "took" : 738,
  "timed_out" : false,
  "_shards" : {
    "total" : 4,
    "successful" : 4,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 1,
      "relation" : "eq"
    },
    "max_score" : 4.54235,
    "hits" : [
      {
        "_index" : ".kibana_7.17.0_001",
        "_type" : "_doc",
        "_id" : "alert:725cedc9-89ed-11ec-bb8f-8d72bbc24e9d",
        "_score" : 4.54235,
        "_source" : {
          "alert" : {
            "name" : "Whoami Process Activity",
            "tags" : [
              "Elastic",
              "Host",
              "Windows",
              "Threat Detection",
              "Discovery",
              "__internal_rule_id:ef862985-3f13-4262-a686-5f357bbb9bc2",
              "__internal_immutable:true"
            ],
            "alertTypeId" : "siem.signals",
            "consumer" : "siem",
            "params" : {
              "author" : [
                "Elastic"
              ],
              "description" : "Identifies use of whoami.exe which displays user, group, and privileges information for the user who is currently logged on to the local system.",
              "falsePositives" : [
                "Some normal use of this program, at varying levels of frequency, may originate from scripts, automation tools and frameworks. Usage by non-engineers and ordinary users is unusual."
              ],
              "from" : "now-9m",
              "license" : "Elastic License v2",
              "outputIndex" : ".siem-signals-default",
              "maxSignals" : 100,
              "riskScore" : 21,
              "riskScoreMapping" : [ ],
              "severity" : "low",
              "severityMapping" : [ ],
              "threat" : [
                {
                  "framework" : "MITRE ATT&CK",
                  "technique" : [
                    {
                      "reference" : "https://attack.mitre.org/techniques/T1033/",
                      "name" : "System Owner/User Discovery",
                      "id" : "T1033"
                    }
                  ],
                  "tactic" : {
                    "reference" : "https://attack.mitre.org/tactics/TA0007/",
                    "name" : "Discovery",
                    "id" : "TA0007"
                  }
                }
              ],
              "timestampOverride" : "event.ingested",
              "to" : "now",
              "references" : [ ],
              "version" : 7,
              "exceptionsList" : [ ],
              "ruleId" : "ef862985-3f13-4262-a686-5f357bbb9bc2",
              "immutable" : true,
              "query" : """process where event.type in ("start", "process_started") and process.name : "whoami.exe"
""",
              "language" : "eql",
              "index" : [
                "winlogbeat-*",
                "logs-endpoint.events.*",
                "logs-windows.*"
              ],
              "type" : "eql"
            },
            "schedule" : {
              "interval" : "5m"
            },
            "enabled" : true,
            "actions" : [ ],
            "throttle" : null,
            "notifyWhen" : "onActiveAlert",
            "apiKeyOwner" : "1049828193",
            "apiKey" : "UMxJPVas+qk+GRB+ZHdT1VrP9zcAs/yItNrcRdayozLApzat+4zPYkNSynKbpgc+dlNp3Kwpk9wrQPr6ArMzxR4FbSfU9YsVuE5SbSC48H/WPte+V5ev7ERSSVZOc71VeW31JTaj4Mju3XlUjmviEI+ugGN/retsOn80FJI4eEpflZokwgu/rKri5/X1d2P/3rowV5dSeOcSuQ==",
            "legacyId" : "725cedc9-89ed-11ec-bb8f-8d72bbc24e9d",
            "createdBy" : "1049828193",
            "updatedBy" : "1049828193",
            "createdAt" : "2022-02-09T21:15:46.344Z",
            "updatedAt" : "2022-02-09T23:29:56.825Z",
            "muteAll" : false,
            "mutedInstanceIds" : [ ],
            "executionStatus" : {
              "status" : "ok",
              "lastExecutionDate" : "2022-02-11T00:13:41.849Z",
              "error" : null,
              "lastDuration" : 3085
            },
            "meta" : {
              "versionApiKeyLastmodified" : "7.17.0"
            },
            "scheduledTaskId" : "316f3ed0-8a00-11ec-bb8f-8d72bbc24e9d"
          },
          "type" : "alert",
          "references" : [ ],
          "migrationVersion" : {
            "alert" : "7.16.0"
          },
          "coreMigrationVersion" : "7.17.0",
          "updated_at" : "2022-02-11T00:13:44.934Z"
        }
      }
    ]
  }
}

[marshallmain]

Hi @ossie-git, I believe the difference you're seeing is due to the timestamp override defined on the prebuilt version of the rule (https://github.com/elastic/kibana/blob/7.17/x-pack/plugins/security_solution/server/lib/detection_engine/rules/prepackaged_rules/discovery_whoami_command_activity.json#L46) which is not included in your custom version.

Can you check if the documents coming from winlogbeat are populating event.ingested? If they are populating only @timestamp and not event.ingested that would explain why the custom rule finds the docs and the prebuilt rule does not - the custom rule uses @timestamp by default as the timestamp field when querying whereas the prebuilt rule uses event.ingested as the timestamp field.

[ossie-git]

Hi @marshallmain ,

Looking at the hits, I noticed the following:

• the winlogbeat-* records do not have event.ingested. I'm not sure why. Isn't this part of being ECS-complaint?
• the logs-* records (created by the Elastic Agent) do have the event.ingested field. However, when I look at the source _index, it gives me: .ds-logs-system.security-default-2022.02.09-000001. However, looking at the rule, it only looks at the following indices:
  • winlogbeat-*
  • logs-endpoint.events.*
  • logs-windows.*

So I'm not sure if:

• the rule itself should look at logs-*?
• the Elastic Agent is not sending to the proper index
• winlogbeat-* should be updated to add the missing field?

as it looks like it will never trigger in its current form. Thanks

[elasticmachine]

Pinging @elastic/security-detections-response (Team:Detections and Resp)

[marshallmain]

event.ingested is typically populated through an ingest pipeline. I asked the beats developers and they said

Winlogbeat 7.x does not use Ingest Node pipelines for processing data. So there are no pipelines that could add event.ingested. So none of the data from Winlogbeat 7.x will contain event.ingested unless the user has done something on their own to add it.
Winlogbeat 8.0+ uses ingest pipelines for processing so it can add event.ingested.

The 8.0 docs for winlogbeat provide a bit more information on the ingest pipeline.

Isn't this part of being ECS-complaint?

It's not a required field in ECS so the docs are still compliant. Most fields in ECS are not required.

[ossie-git]

Thanks @marshallmain. If this is the case:

• why doesn't the rule use @timestamp instead to be more generic? I think that writing a rule to look at event.ingested makes users who aren't used the suggested collection method miss out on rules. What if I'm using Winlogbeat 7.x or NXLog or Logstash and the field isn't being sent? In fact, 8.0 wasn't even released when I tested this rule
• why isn't the rule looking at the logs-* that Elastic Agent created?
• if specific rules work only on 8.x and not 7.x, shouldn't these be labeled or disabled for 7.x users instead of confusing users?

Unless I'm missing something, if this issue is generic and found in other rules as well (I only tested this simple rule), that means that a lot of users aren't getting alerts for rules they enabled.

[marshallmain]

@timestamp is not always reliable from the detection engine's viewpoint, since it is typically set by the agent according to the system clock. event.ingested is generally the most reliable field to ensure that all arriving documents are queried, even if the documents arrive long after they were generated at the source or the source clock is inaccurate. With that said, there's an open enhancement issue to support EQL rules falling back to @timestamp for documents that don't have the timestamp_override field (event.ingested in this case) set.

I've forwarded on your question about the choice of index patterns queried by the rule to get an answer.

Although Winlogbeat does not populate event.ingested by default in 7.x, the Elastic Endpoint and other data sources do and it is possible to add an ingest pipeline for the winlogbeat indices as well to populate event.ingested.

[ossie-git]

Thanks @marshallmain As I mentioned above, I did find it in the index used by Elastic Agent. However, this was .ds-logs-system.security-default-2022.02.09-000001 and therefore didn't match the indices the rule searches in. So none of the indices used by the rule actually match