Incorrect timestamp #1047

kernelpanicfbsd · 2017-04-24T09:21:46Z

Hello all, I'm using elastalert to query a winlogbeat index which does use @timestamp for the date/time.

However, the email alert I'm receiving has a timestamp value that is an hour behind the actual timestamp on the document it has matched; for example a document with the timestamp of @timestamp: 2017-04-24T10:05:42.687Z in the elastalert database has a timestamp of @timestamp: 2017-04-24T09:05:42.687Z on the elastalert email.

Can anyone help?

Qmando · 2017-04-24T17:28:37Z

Are you sure you're not looking in Kibana, where the timestamp is automatically adjusted to local time? Can you post the raw JSON of the document that matched (including _id) along with the alert (including _id)?

kernelpanicfbsd · 2017-04-24T17:34:17Z

Hello thanks for getting back to me - yes, I was looking in Kibana, however my understanding is that by default Elastalert uses local time too, so shouldn't the Kibana and Elastalert timestamps be the same value?

Qmando · 2017-04-24T17:38:41Z

Internally ElastAlert converts everything into UTC (The Z at the end of the timestamps you posted means UTC in iso8601). In a few places it will convert it into local time to display in the alert, but when it does it will add the timezone.

kernelpanicfbsd · 2017-04-24T17:43:49Z

OK thanks, so what should I do to get the two timestamp values to agree with each other?

Qmando · 2017-04-24T17:49:07Z

You can set use_local_time: true in your config. This will at least make the log lines and title of the alert have the local timezone. In Logstash you can add a second timestamp field, converted into local time, and then put that field into your alert_text

kernelpanicfbsd · 2017-04-24T17:54:23Z

Thanks again - is use_local_time: true set on config.yaml or in a rule?

Qmando · 2017-04-24T18:18:58Z

config.yaml to make it apply to all rules, or per rule

kernelpanicfbsd · 2017-04-24T18:31:20Z

Adding the use_local_time option has not made a difference, the alert @timestamp is still an hour behind so I assume I'll need to go the Logstash route?

Test ElastAlert Rule

@timestamp: 2017-04-24T18:24:21.841Z
_id: AVuhNQq9ZFqcPo9Hyrv8
_index: winlogbeat-2017.04
_type: wineventlog
event_data: {
"MemberName": "####################",
"SubjectUserName": "############",
"TargetUserName": "######"
}
event_id: 4728
message: A member was added to a security-enabled global group.

Subject:
Security ID: #####################
Account Name: #############
Account Domain: #############
Logon ID: ############

Member:
Security ID: ##########################
Account Name: ########################

Group:
Security ID: ######################
Group Name: ##########
Group Domain: ###########

Additional Information:
Privileges: -
num_hits: 1
num_matches: 1

arlindonatal · 2017-05-25T18:27:25Z

@kernelpanicfbsd any solution?

Qmando · 2017-05-25T18:32:39Z

@arlindonatal
Try an enhancement with this code:

from elastalert.util import pretty_ts
from elastalert.enhancements import BaseEnhancement

class TimeEnhancement(BaseEnhancement):
    def process(self, match):
        match['@timestamp'] = pretty_ts(match['@timestamp'])

arlindonatal · 2017-05-25T18:47:01Z

@Qmando thks!

I use in the "alert_text_args", but the timezone is incorrect, returns with +3 hours.

my rule:
....
timestamp_type: custom
timestamp_format: "%Y-%m-%dT%H:%M:%S.%fZ"
timestamp_format_expr: 'ts[:23] + ts[26:]'
use_local_time: true

alert_text_type: "alert_text_only"
alert_text: |
Period: {0} to {1}.
alert_text_args:

"@timestamp" ##<< timestamp is wrong +3 hours
"starttime" ## < how get end time?

Other problem is the period. How get endtime the rule?

Qmando · 2017-05-25T18:50:53Z

If you are using pretty_ts function, it will use the local machines timezone (/etc/timezone). Is that incorrect? Or are you still just printing out UTC?

Try adding
match['starttime'] = self.rule['starttime'] to the enhancement

Cas-pian · 2017-10-25T07:20:19Z

Seems I met this problem too.
The '' is default true, so no need to set.
And This is the email I got:

At least 1 events occurred between 2017-10-25 15:04 CST and 2017-10-25 15:05 CST

@timestamp: 2017-10-25T07:05:46Z
_id: AV9SWf-KUz6aUoZzhBUI
_index: fluentd-20171025
.....

The @timestamp field is not OK for me, but the time format in the first line is OK.
Any other settings to fix this? kibana has convert it to local timezone.

Cas-pian · 2017-10-26T03:18:49Z

After searching I fixed this problem by previous issues. Thank you very much!

UglyAnimal · 2019-02-13T15:24:22Z

@arlindonatal
Try an enhancement with this code:

from elastalert.util import pretty_ts
from elastalert.enhancements import BaseEnhancement

class TimeEnhancement(BaseEnhancement):
    def process(self, match):
        match['@timestamp'] = pretty_ts(match['@timestamp'])

I'm getting notifications with correct timestamp with this enhancement. But I'm getting an error as well :

ERROR:root:Error writing alert info to Elasticsearch: RequestError(400, u'mapper_parsing_exception', u'failed to parse field [match_time] of type [date]')
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/elastalert/elastalert.py", line 1494, in writeback
    doc_type=doc_type, body=body)
  File "/usr/lib/python2.7/site-packages/elasticsearch-6.3.1-py2.7.egg/elasticsearch/client/utils.py", line 76, in _wrapped
    return func(*args, params=params, **kwargs)
  File "/usr/lib/python2.7/site-packages/elasticsearch-6.3.1-py2.7.egg/elasticsearch/client/__init__.py", line 319, in index
    _make_path(index, doc_type, id), params=params, body=body)
  File "/usr/lib/python2.7/site-packages/elasticsearch-6.3.1-py2.7.egg/elasticsearch/transport.py", line 318, in perform_request
    status, headers_response, data = connection.perform_request(method, url, params, body, headers=headers, ignore=ignore, timeout=timeout)
  File "/usr/lib/python2.7/site-packages/elasticsearch-6.3.1-py2.7.egg/elasticsearch/connection/http_requests.py", line 90, in perform_request
    self._raise_error(response.status_code, raw_data)
  File "/usr/lib/python2.7/site-packages/elasticsearch-6.3.1-py2.7.egg/elasticsearch/connection/base.py", line 125, in _raise_error
    raise HTTP_EXCEPTIONS.get(status_code, TransportError)(status_code, error_message, additional_info)
RequestError: RequestError(400, u'mapper_parsing_exception', u'failed to parse field [match_time] of type [date]')

How can I resolve this?

ilysia · 2019-05-08T15:11:32Z

@arlindonatal
Try an enhancement with this code:

from elastalert.util import pretty_ts
from elastalert.enhancements import BaseEnhancement

class TimeEnhancement(BaseEnhancement):
    def process(self, match):
        match['@timestamp'] = pretty_ts(match['@timestamp'])

I'm getting notifications with correct timestamp with this enhancement. But I'm getting an error as well :

ERROR:root:Error writing alert info to Elasticsearch: RequestError(400, u'mapper_parsing_exception', u'failed to parse field [match_time] of type [date]')
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/elastalert/elastalert.py", line 1494, in writeback
    doc_type=doc_type, body=body)
  File "/usr/lib/python2.7/site-packages/elasticsearch-6.3.1-py2.7.egg/elasticsearch/client/utils.py", line 76, in _wrapped
    return func(*args, params=params, **kwargs)
  File "/usr/lib/python2.7/site-packages/elasticsearch-6.3.1-py2.7.egg/elasticsearch/client/__init__.py", line 319, in index
    _make_path(index, doc_type, id), params=params, body=body)
  File "/usr/lib/python2.7/site-packages/elasticsearch-6.3.1-py2.7.egg/elasticsearch/transport.py", line 318, in perform_request
    status, headers_response, data = connection.perform_request(method, url, params, body, headers=headers, ignore=ignore, timeout=timeout)
  File "/usr/lib/python2.7/site-packages/elasticsearch-6.3.1-py2.7.egg/elasticsearch/connection/http_requests.py", line 90, in perform_request
    self._raise_error(response.status_code, raw_data)
  File "/usr/lib/python2.7/site-packages/elasticsearch-6.3.1-py2.7.egg/elasticsearch/connection/base.py", line 125, in _raise_error
    raise HTTP_EXCEPTIONS.get(status_code, TransportError)(status_code, error_message, additional_info)
RequestError: RequestError(400, u'mapper_parsing_exception', u'failed to parse field [match_time] of type [date]')

How can I resolve this?

I tried the same thing. The problems are:

It didn't change the time zone, just the time
It also changes the @timestamp. This means it has problems when it's trying to write_back it to elasticsearch, because of the type change. This is the error you are getting, as far as I understand.

ilysia · 2019-05-08T15:23:45Z

The @timestamp format issue can be fixed by using something like this:

from elastalert.util import pretty_ts
from elastalert.enhancements import BaseEnhancement

class TimeEnhancement(BaseEnhancement):
    def process(self, match):
        pretty_time = pretty_ts(match['@timestamp'])
        match['local_time'] = pretty_time

And then in the rule referencing in "local_time" instead of "@timestamp":
alert_text_args: - "local_time"

nikhilgangrade · 2019-05-21T18:25:10Z

I tried the above solution as well what @Qmando provided.
But i am getting an error
File "elastalert\elastalert.py", line 22, in <module> from config import get_rule_hashes ImportError: cannot import name get_rule_hashes

If anyone has the solution for this

nikhilgangrade · 2019-05-21T19:42:36Z

Thanks for the quick response.
But now i am getting an error
from elastalert.config import get_rule_hashes ImportError: No module named config

Qmando · 2019-05-21T20:41:14Z

How are you starting elastalert? Try using python -m elastalert.elastalert

nikhilgangrade · 2019-05-21T20:46:17Z

I am starting as-
python -m elastalert.elastalert --verbose --rule my_rules/email_rule.yaml

Everything was running before i was getting email alerts but the timestamp shows sometime a day old.
Like today i received an alert as
2019-05-20T16:00:10.489Z Server WIN-3J7RM38R0LU, High Memory used 64%.

Even the time is wrong

nikhilgangrade · 2019-05-21T20:47:10Z

While testing it i am not getting any error.

elastalert-test-rule my_rules/email_rule.yaml

Qmando · 2019-05-21T20:48:21Z

Try just elastalert. Something is off with the path it's trying to load modules from. I'm not quite sure what it is yet

Qmando · 2019-05-21T20:54:58Z

Not python -m elastalert but JUST elastalert.

Though I really have zero experience with python on windows so something else may be afoot here.

nikhilgangrade · 2019-05-21T20:59:07Z

I got the output as-
Traceback (most recent call last): File "C:\Python27\Scripts\elastalert-script.py", line 11, in <module> load_entry_point('elastalert==0.1.39', 'console_scripts', 'elastalert')() File "C:\Python27\lib\site-packages\elastalert-0.1.39-py2.7.egg\elastalert\elastalert.py", line 1925, in main client = ElastAlerter(args) File "C:\Python27\lib\site-packages\elastalert-0.1.39-py2.7.egg\elastalert\elastalert.py", line 109, in __init__ self.conf = load_rules(self.args) File "C:\Python27\lib\site-packages\elastalert-0.1.39-py2.7.egg\elastalert\config.py", line 516, in load_rules raise EAException('Error loading file %s: %s' % (rule_file, e)) elastalert.util.EAException: Error loading file rules\example_spike_single_metric_agg.yaml: Invalid Rule file: rules\example_spike_single_metric_agg.yaml {'index': 'metricbeat-*', 'name': 'Metricbeat Average CPU Spike Rule', 'query_key': 'beat.hostname', 'spike_type': 'up', 'alert': ['debug'], 'spike_height': 2, 'buffer_time': {'hours': 1}, 'doc_type': 'metricsets', 'rule_file': 'rules\\example_spike_single_metric_agg.yaml', 'min_doc_count': 5, 'metric_agg_key': 'system.cpu.user.pct', 'type': 'spike_aggregation', 'filter': [{'term': {'metricset.name': 'cpu'}}], 'metric_agg_type': 'avg', 'threshold_cur': 0.9} is not valid under any of the given schemas

and then i have bunch of schema, which says Failed validating 'oneof' in schema:

Qmando · 2019-05-21T21:02:37Z

~~I think you're missing timeframe~~
I'm not sure what your missing there

nikhilgangrade · 2019-05-21T21:03:01Z

I am not sure why its checking example_spike_single_metric_agg.yaml in rules folder.

nikhilgangrade · 2019-05-21T21:06:27Z

~~I think you're missing timeframe~~
I'm not sure what your missing there

Even i am unable to understand, just to change the @timestamp i have to work on enhancements.py otherwise i am getting the alerts on email and JIRA.

I even added timeframe, bbut nothing changes

nikhilgangrade · 2019-05-22T15:53:27Z

And one the problem that i am facing is, elastalert is getting started from 5/20/2019 till today i.e. 5/22/2019. But it shows the alert notification of the 20th May.
I am not sure why it is doing it.
python -m elastalert.elastalert --verbose --rule rules\memory_threshold.yaml INFO:elastalert:Starting up INFO:elastalert:Queried rule Metricbeat Memory Spike rule from 2019-05-20 09:00 Pacific Daylight Time to 2019-05-20 09:01 Pacific Daylight Time: 0 / 0 hits INFO:elastalert:Queried rule Metricbeat Memory Spike rule from 2019-05-20 09:01 Pacific Daylight Time to 2019-05-20 09:02 Pacific Daylight Time: 0 / 0 hits
...
INFO:elastalert:Queried rule Metricbeat Memory Spike rule from 2019-05-22 08:43 Pacific Daylight Time to 2019-05-22 08:44 Pacific Daylight Time: 0 / 0 hits INFO:elastalert:Queried rule Metricbeat Memory Spike rule from 2019-05-22 08:44 Pacific Daylight Time to 2019-05-22 08:45 Pacific Daylight Time: 0 / 0 hits INFO:elastalert:Queried rule Metricbeat Memory Spike rule from 2019-05-22 08:45 Pacific Daylight Time to 2019-05-22 08:46 Pacific Daylight Time: 0 / 0 hits INFO:elastalert:Queried rule Metricbeat Memory Spike rule from 2019-05-22 08:46 Pacific Daylight Time to 2019-05-22 08:47 Pacific Daylight Time: 0 / 0 hits INFO:elastalert:Queried rule Metricbeat Memory Spike rule from 2019-05-22 08:47 Pacific Daylight Time to 2019-05-22 08:47 Pacific Daylight Time: 0 / 0 hits INFO:elastalert:Opened Jira ticket: SWCS-44 INFO:elastalert:Alert for Metricbeat Memory Spike rule at 2019-05-20T17:22:33.353Z: INFO:elastalert:2019-05-20T17:22:33.353Z Server WIN-3J7RM38R0LU, High Memory used 0.8012.

And even the alert is wrong becasue at that particular time i have no spike.
If anyone can help me with this

nikhilgangrade · 2019-05-22T18:31:42Z

The above error got resolved by using
old_query_limit:
minutes: 0

in the config.yaml file

liujunnan12311 · 2021-10-27T10:18:21Z

@Qmando Hi Qmando, this method didn't work for me, this method just modify the time of alert email conent， trigger sending email time still use UTC time.

eg：
my current timezone is UTC+8:00, I sent test data "@timestamp": "2021-10-28T17:10:46.000", can't receive e-mail, but if send "@timestamp": "2021-10-28T09:10:46.000" will receive alter email. how can i trigger alert by using local timezone.

How should I deal with this problem? Look forward to your reply.

Thanks

leandrob04 mentioned this issue Jun 25, 2017

Problem to execute enhancement #1185

Closed

Qmando closed this as completed May 22, 2019

perceptron01 mentioned this issue Oct 29, 2019

Enable to select whether to prohibit enhancement from being run on alert information to be written back to ES #2542

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Incorrect timestamp #1047

Incorrect timestamp #1047

kernelpanicfbsd commented Apr 24, 2017

Qmando commented Apr 24, 2017

kernelpanicfbsd commented Apr 24, 2017

Qmando commented Apr 24, 2017

kernelpanicfbsd commented Apr 24, 2017

Qmando commented Apr 24, 2017

kernelpanicfbsd commented Apr 24, 2017

Qmando commented Apr 24, 2017 •

edited

Loading

kernelpanicfbsd commented Apr 24, 2017

arlindonatal commented May 25, 2017

Qmando commented May 25, 2017

arlindonatal commented May 25, 2017

Qmando commented May 25, 2017

Cas-pian commented Oct 25, 2017

Cas-pian commented Oct 26, 2017

UglyAnimal commented Feb 13, 2019

ilysia commented May 8, 2019

ilysia commented May 8, 2019 •

edited

Loading

nikhilgangrade commented May 21, 2019

nikhilgangrade commented May 21, 2019

Qmando commented May 21, 2019

nikhilgangrade commented May 21, 2019

nikhilgangrade commented May 21, 2019

Qmando commented May 21, 2019 •

edited

Loading

Qmando commented May 21, 2019

nikhilgangrade commented May 21, 2019

Qmando commented May 21, 2019 •

edited

Loading

nikhilgangrade commented May 21, 2019

nikhilgangrade commented May 21, 2019 •

edited

Loading

nikhilgangrade commented May 22, 2019 •

edited

Loading

nikhilgangrade commented May 22, 2019

liujunnan12311 commented Oct 27, 2021 •

edited

Loading

Incorrect timestamp #1047

Incorrect timestamp #1047

Comments

kernelpanicfbsd commented Apr 24, 2017

Qmando commented Apr 24, 2017

kernelpanicfbsd commented Apr 24, 2017

Qmando commented Apr 24, 2017

kernelpanicfbsd commented Apr 24, 2017

Qmando commented Apr 24, 2017

kernelpanicfbsd commented Apr 24, 2017

Qmando commented Apr 24, 2017 • edited Loading

kernelpanicfbsd commented Apr 24, 2017

arlindonatal commented May 25, 2017

Qmando commented May 25, 2017

arlindonatal commented May 25, 2017

Qmando commented May 25, 2017

Cas-pian commented Oct 25, 2017

Cas-pian commented Oct 26, 2017

UglyAnimal commented Feb 13, 2019

ilysia commented May 8, 2019

ilysia commented May 8, 2019 • edited Loading

nikhilgangrade commented May 21, 2019

nikhilgangrade commented May 21, 2019

Qmando commented May 21, 2019

nikhilgangrade commented May 21, 2019

nikhilgangrade commented May 21, 2019

Qmando commented May 21, 2019 • edited Loading

Qmando commented May 21, 2019

nikhilgangrade commented May 21, 2019

Qmando commented May 21, 2019 • edited Loading

nikhilgangrade commented May 21, 2019

nikhilgangrade commented May 21, 2019 • edited Loading

nikhilgangrade commented May 22, 2019 • edited Loading

nikhilgangrade commented May 22, 2019

liujunnan12311 commented Oct 27, 2021 • edited Loading

Qmando commented Apr 24, 2017 •

edited

Loading

ilysia commented May 8, 2019 •

edited

Loading

Qmando commented May 21, 2019 •

edited

Loading

Qmando commented May 21, 2019 •

edited

Loading

nikhilgangrade commented May 21, 2019 •

edited

Loading

nikhilgangrade commented May 22, 2019 •

edited

Loading

liujunnan12311 commented Oct 27, 2021 •

edited

Loading