Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Filebeat] Add support for postgresql version 10 logs #13810

Closed
kaiyan-sheng opened this issue Sep 26, 2019 · 2 comments · Fixed by #17732
Closed

[Filebeat] Add support for postgresql version 10 logs #13810

kaiyan-sheng opened this issue Sep 26, 2019 · 2 comments · Fixed by #17732
Assignees
Labels
enhancement Filebeat Filebeat Team:Integrations Label for the Integrations team Team:Services (Deprecated) Label for the former Integrations-Services team

Comments

@kaiyan-sheng
Copy link
Contributor

Based on the documentation, we only support postgresql versions 9.5 on Ubuntu and 9.6
on Debian. On discuss forum, user is asking for version 10.x, the postgresql log doesn't work with current grok pattern: https://discuss.elastic.co/t/postgres-module-for-filebeat-is-not-matching-the-logs/200680

Example logs are given by user:

2019-09-22 06:28:24 UTC DETAIL:  parameters: $1 = '86', $2 = '575', $3 = 'Item Inserted', $4 = 'Item Updated', $5 = 'Subject Updated', $6 = 'Subject Created', $7 = '2019-01-22 00:00:00+00'
2019-09-22 06:28:24 UTC LOG:  duration: 112.337 ms  execute S_59: UPDATE qrtz_TRIGGERS SET TRIGGER_STATE = $1 WHERE SCHED_NAME = 'Scheduler_1' AND TRIGGER_NAME = $2 AND TRIGGER_GROUP = $3 AND TRIGGER_STATE = $4
2019-09-22 06:28:24 UTC DETAIL:  parameters: $1 = 'ACQUIRED', $2 = 'surveyInvitation_3Prbn85DiBWe8wHa_158802_77133_1260104', $3 = 'ExecutorsService', $4 = 'WAITING'
2019-09-22 06:28:24 UTC LOG:  duration: 2474.307 ms  execute S_30: SELECT * FROM qrtz_LOCKS WHERE SCHED_NAME = 'Scheduler_1' AND LOCK_NAME = $1 FOR UPDATE
2019-09-22 06:28:24 UTC DETAIL:  parameters: $1 = 'TRIGGER_ACCESS'
2019-09-22 06:28:24 UTC LOG:  duration: 18.327 ms  execute S_32: SELECT al.id, al.tenant_id, al.created_by_id, al.create_ip, al.audit_date, al.audit_table, al.entity_id, al.entity_name, al.reason_for_change, al.audit_log_event_type_id,
        aet.lookup_code, al.old_value, al.new_value, al.event_crf_id, al.event_crf_version_id, al.study_id, al.study_site_id, ss.rc_oid, al.subject_id, s.unique_identifier,
        al.study_event_id, sed.name AS studyEventName, al.user_id, al.value_index, al.crf_version_id, al.global_logs, cv.version_name, crf.id AS crfId, crf.name AS crfName
         FROM public.rc_audit_log_events AS al
        LEFT JOIN rc_crf_versions AS cv ON cv.id=al.crf_version_id
            LEFT JOIN rc_crfs AS crf ON crf.id=cv.crf_id
            LEFT JOIN ad_lookup_codes AS aet ON aet.id=al.audit_log_event_type_id
            LEFT JOIN rc_study_sites AS ss ON ss.id=al.study_site_id
            LEFT JOIN rc_subjects AS s ON s.id=al.subject_id
            LEFT JOIN rc_study_events AS se ON se.id=al.study_event_id
            LEFT JOIN rc_study_event_definitions AS sed ON sed.id=se.study_event_definition_id
            WHERE al.tenant_id=$1 AND al.study_id=$2  AND aet.lookup_code IN ($3, $4, $5, $6) AND al.audit_date >= $7 ORDER BY al.id DESC  limit $8
@sayden
Copy link
Contributor

sayden commented Oct 7, 2019

Probably related with this: #12078

@andresrc andresrc added Team:Services (Deprecated) Label for the former Integrations-Services team [zube]: Inbox [zube]: Investigate and removed [zube]: Investigate [zube]: Inbox labels Jan 29, 2020
@sayden
Copy link
Contributor

sayden commented Apr 15, 2020

I have been playing with this and, just to clarify, that logging is not default in postgres. You need to activate log_min_duration_statement = 0 at least to show duration of statements. Anyways I'm going to see if I can workaround it with the current pattern as it looks like a fair enhancement.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement Filebeat Filebeat Team:Integrations Label for the Integrations team Team:Services (Deprecated) Label for the former Integrations-Services team
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants