Limit rules to certain time during the day

[jurgenhaas]

We have a few cardinality rules defined that are checking that our services are running OK by making sure that a certain number of events can be found in ES. Now, some of those rules only make sense within business hours and we would love to define such periods as part of the rules.

In other words, let's enable some rules to be executed only within specific periods each day.

Is that a valid feature request?

[dmaciel]

I am trying to do this with a new rule type, but I will be better if this use case is supported by the app by default

[lwhitworth]

Just came here to request the same. In my scenario I want to use a flatline rule to alert if succesful auths to our Radius server stop working (so no successful auths in x minutes). However once everyone goes home for the evening we see a few hours of no auths so the rule would be triggered throughout the night and at weekends. What I'd like is in the rule the ability to say only alert from 9am to 5pm.

Only solution I can see at the moment is to alert out to a third party that handles this part of the logic, but native support would be awesome

[jurgenhaas]

Just thought of a workaround here: we could move the relevant rule files out of the rules directory at 5pm with a cron task and move them back into that directory at 9am. As far as I understand, ElastAlert can already handle dynamic changes in the rules directory. I probably give that a try until we come to a conclusion on how to proceed with this feature request.

[Qmando]

If you do that, be sure to set old_query_limit to something small in config.yaml, otherwise, when you add the rule back, it will pick up from where it left off the previous day. It will always pick up where it last left off for any rule, given that it's within old_query_limit, which defaults to 1 week.

Other than that, that seems like it should work.

[BaeHwidong-NBT]

I reached here thinking of the exactly same feature as @lwhitworth wrote. I'll try @jurgenhaas's suggestion for now. Thank you!

[iekulyk]

You can configure you filter to ignore records within certain timespans

[icyerasor]

Certainly not the most elegant solution, but what i found out seems to work for a flatline kind of rule that should only be triggerd at certain times of the day is a construct like this:

filter:
- bool:
    should:
        - bool:
            must:
                - query_string:
                    query: "(log_message:\"My Log message\")"
                - range:
                    "@timestamp":
                        time_zone: "CET"
                        from: "now-24h/d+6h"
                        to: "now+24h/d"
        - bool:
            must:
                - range:
                    "@timestamp":
                        time_zone: "CET"
                        from: "now-24h/d"
                        to: "now-24h/d+6h"
threshold: 1

which should count every log entry until 6am as a match (2nd condition) and check if the query_string condition matches between 6am and end of day (1st condition).
The now-24h/d resolves to the start of the current day (as /d rounds to the "next" day in this scenario, as from translates to gt and to to lte which will both round up to the next day, see ref-doc

[JustinPealing]

Could the run_every option is enhanced to accept cron syntax in the same way that the 'aggregation' option does? e.g:

run_every:
    schedule: '* 9-17 * * 1-5'

[Qmando]

Yes. I'm sorry this has been outstanding for so long. I will try to finish it up soon.

[josephka333]

Hi @Qmando please do help us with this feature its very useful and would be a great addition.

[pvsms]

I landed here for the very same request to run the rule only during certain hours of the day. Let me know if there is any feature enhanced on the same.

[joeyJsonar]

@Qmando Hello, we managed to integrate python APScheduler on our fork of elastalert, thus allowing a cron field in rule yaml. I just didn't have the time to have a pull request. I'll schedule to do one this Friday or weekend (long weekend here in Canada so I should be able to find the time).

[pvsms]

Excellent! How quickly the PR can be merged and get a new JAR version, just to understand when i can integrate with it and plan my activities accordingly. Thanks again!

[pvsms]

@joeyJsonar: Were you able to create and get the PR merged? Thx.

[Qmando]

I've merged this feature into a new branch, beta, and released it as a new package version 0.2.0b1 available on pypi.

This includes a couple other changes as well, like threading support, but you can now limit rule execution to certain times of the day using limit_execution using cron syntax. For example

limit_execution: "* 7-22 * * *"

Would mean to only run the rule between 7 am and 10 pm every day.

This feature is still in beta, of course, but you're welcome to try.

[JaredRietdyk]

@Qmando, I've been running with limit_execution for a few days now and it seems to be working well. I believe there might be an issue with the frequency type, where after the cron is done it will go back and alert on the results since the last cron.

[gsagwan]

Hi,
i am facing issue where in my rule is paused until the cron specified.
limit_execution: "* 17-23 * * *"

This is what i get in logs after specifying the range.
INFO:elastalert:Pausing OTP dip Alert until next run at 2018-12-17 22:30 IST

[JaredRietdyk]

@gsagwan, I was confused at first as well. The limit execution is limiting the rule to run ONLY during that time. So if you want the rule to run and stop during that time you would need to use something like:
limit_execution: "* 0-16,24 * * *"

[gsagwan]

@gsagwan, I was confused at first as well. The limit execution is limiting the rule to run ONLY during that time. So if you want the rule to run and stop during that time you would need to use something like:
limit_execution: "* 0-16,24 * * *"

Hi @JaredRietdyk , Thanks for your comments.
I am trying to put the below cron.
limit_execution: "* 04-17 * * *"

But it continues to run post 17 as well.
Not able to figure out how this is working.
Basically i need to run my rule during 0400-1700 hours.

[Qmando]

@gsagwan

I think I may see the problem. It's treating the cron schedule as UTC. So when you add 04-17 but, for example, you're in UTC-5 (EST), it's going to actually run from 0-12,23-24.

I think I only tested this with minutes and days of the week 🤦‍♂️

I'll try to get a fix up soon that will treat the schedule as local time.

[thesm3rdo]

Hi @gsagwan , is there an update on this limit_execution property ? I cannot see it in the beta branch and am wanting to track its progress as it's a piece of functionality that will be very useful to our business.

[Qmando]

@thesm3rdo
https://github.com/Yelp/elastalert/blob/beta/elastalert/elastalert.py#L1238

It's there for use already. No update regarding using local timezones, or when this will be released from beta to a full release.

[Atem18]

@Qmando I see that you are pushing things to Master branch but not Beta.
So what is the current status of the feature ?
Do you cherry-pick commits from beta branch to master ?

[Qmando]

I created the beta release for testing some large and potentially dangerous changes, in order to get some help testing them. When I merge it into a new release, it will contain everything from the master branch too. Apologies for the very slow pace of releases.

[Atem18]

@Qmando No problem, make the best production release possible ! :)

[Atem18]

@Qmando Hi, any update on merging in to master ?

[Qmando]

I did pull the changes form master into beta. But, I'll probably wait until I've finished the py3 migration to release those changes. Sorry It's been very slow.

[Atem18]

Ok thanks a lot

[0xSeb]

I found a solution making an Enhancement so that you can "plug it" on any rule of yours :
What do you think about it ? @Qmando

https://github.com/0x-29A/elastalert_hour_range

[anuarabdullah]

Hi, as today can I know how to implement the limit_execution function in elastalert properly? I use limit_execution: "* 6-23 * * *" in my rule.yml but the rule keep running past 2300 hrs.

Also this limit_execution run on local timezone or other timezone?

[rabiashaikabdulkader]

I've merged this feature into a new branch, beta, and released it as a new package version 0.2.0b1 available on pypi.

This includes a couple other changes as well, like threading support, but you can now limit rule execution to certain times of the day using limit_execution using cron syntax. For example

limit_execution: "* 7-22 * * *"

Would mean to only run the rule between 7 am and 10 pm every day.

This feature is still in beta, of course, but you're welcome to try.

Hi @Qmando I would like to ask if this feature is available in elastalert 0.2.2 version. I am looking to check for the rules at a particular time and trigger an alert. May I know how I should achieve it? Can you please guide me by copy pasting the sample alert.yaml file. Say for example if I want to run the rule between 02:00 - 3:00 am UTC time everyday.

Say I tried to edit my alert file like this. I am not sure if this is correct, kindly help :

`type: frequency
index: prod-filebeat-k8s-*
timeframe:
minutes: 0
num_events: 1
limit_execution: "* 02-03 * * *"
filter:

• term:`

[svenkyedem]

HI @Qmando . I have tested limit_execution feature.i have given cron as below
limit_execution: "55-55 6-7 * * *"
In logs it is showing like below
INFO:elastalert:Pausing Backup Fail Alert rule until next run at 2020-11-11 06:55 UTC
When the time is 6:55 UTC it is showing like
INFO:elastalert:Pausing Backup Fail Alert rule until next run at 2020-11-11 07:55 UTC
When the time is 7:55 UTC it is showing like
INFO:elastalert:Pausing Backup Fail Alert rule until next run at 2020-11-12 06:55 UTC

Basically it is not executing anytime.

elastalert version: v0.2.0

[jgutta]

Hi! Also wanted to express interest in this feature. It doesn't seem to be in V0.2.4, is there any a plan to enable it soon?

[r1296]

Would be great if this feature could be added in one of the next releases.

[ParshantMehra]

Any update when will this feature be available?

[fberrez]

elastalert is no more maintained.
However, the limit_execution feature was added in elastalert 0.2.0 (see: jertel/elastalert2#106) but the documentation has not been updated.