Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[FIlebeat] add strict_date_optional_time_nanos date format to PanOS module #26158

Merged
merged 5 commits into from
Jun 24, 2021
Merged

[FIlebeat] add strict_date_optional_time_nanos date format to PanOS module #26158

merged 5 commits into from
Jun 24, 2021

Conversation

legoguy1000
Copy link
Contributor

@legoguy1000 legoguy1000 commented Jun 7, 2021 •
edited
Loading

What does this PR do?

Adds strict_date_optional_time_nanos date format to the date processors

Why is it important?

See #26033

Checklist

  • My code follows the style guidelines of this project
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • I have made corresponding change to the default configuration files
  • I have added tests that prove my fix is effective or that my feature works
  • I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Author's Checklist

How to test this PR locally

cd beats/x-pack/filebeat
TESTING_FILEBEAT_MODULES=panw mage -v pythonIntegTest

Related issues

Use cases

Screenshots

Logs

@botelastic botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Jun 7, 2021
@elasticmachine
Copy link
Collaborator

elasticmachine commented Jun 7, 2021 •
edited by jenkins-beats-ci bot
Loading

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS
Pipeline View Test View Changes Artifacts preview

Expand to view the summary

Build stats

  • Build Cause: Pull request #26158 updated

  • Start Time: 2021-06-24T16:58:26.096+0000

  • Duration: 103 min 17 sec

  • Commit: 45b9cb2

Test stats 🧪

Test Results
Failed 0
Passed 7396
Skipped 1201
Total 8597

Trends 🧪

Image of Build Times

Image of Tests

💚 Flaky test report

Tests succeeded.

Expand to view the summary

Test stats 🧪

Test Results
Failed 0
Passed 7396
Skipped 1201
Total 8597

@legoguy1000
Copy link
Contributor Author

@jamiehynds Do you have sample data I can add to this to validate the changes??

@legoguy1000 legoguy1000 marked this pull request as ready for review June 8, 2021 01:33
@mergify
Copy link
Contributor

mergify bot commented Jun 8, 2021

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b 26033-panos-date upstream/26033-panos-date
git merge upstream/master
git push upstream 26033-panos-date

@mergify
Copy link
Contributor

mergify bot commented Jun 8, 2021

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b 26033-panos-date upstream/26033-panos-date
git merge upstream/master
git push upstream 26033-panos-date

@jamiehynds
Copy link

@jamiehynds Do you have sample data I can add to this to validate the changes??

@legoguy1000 working on getting sample data for you. Thanks as always for the contribution!

@elasticmachine
Copy link
Collaborator

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

@botelastic botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Jun 14, 2021
@jamiehynds
Copy link

@legoguy1000 sample log below. Adjusted the IP's to ensure they are anonymised.

1,2021-05-26T16:27:07.000000Z,no-serial,TRAFFIC,end,9.1,2021-05-26T16:26:47.000000Z,127.0.0.0,127.0.0.1,127.0.0.2,127.0.0.3,intrazone-default,,,web-browsing,vsys1,untrust,untrust,ethernet1/1,ethernet1/1,Cortex Data Lake,,688290,1,35834,443,35834,20077,0x1400070,tcp,allow,7291,1696,5595,21,2021-05-26T16:26:30.000000Z,1,medium-risk,,620386,0x8800000000000000,US,SG,,14,7,tcp-fin,22,18,0,0,,GP cloud service,from-policy,,,0,,0,1970-01-01T00:00:00.000000Z,N/A,0,0,0,0,6a2f6161-88f2-4afc-8dd5-256bc4505a64,0,0,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,

@mergify
Copy link
Contributor

mergify bot commented Jun 16, 2021

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b 26033-panos-date upstream/26033-panos-date
git merge upstream/master
git push upstream 26033-panos-date

@legoguy1000 legoguy1000 marked this pull request as draft June 24, 2021 12:06
@legoguy1000 legoguy1000 marked this pull request as ready for review June 24, 2021 12:57
@P1llus
Copy link
Member

P1llus commented Jun 24, 2021

/test

@mergify
Copy link
Contributor

mergify bot commented Jun 24, 2021

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b 26033-panos-date upstream/26033-panos-date
git merge upstream/master
git push upstream 26033-panos-date

@P1llus P1llus merged commit 7742fda into elastic:master Jun 24, 2021
@P1llus P1llus added the backport-v7.14.0 Automated backport with mergify label Jun 24, 2021
mergify bot pushed a commit that referenced this pull request Jun 24, 2021
@legoguy1000
[FIlebeat] add strict_date_optional_time_nanos date format to PanOS m…
0eb6bbc 
…odule (#26158)

* #26033: add strict_date_optional_time_nanos format

* update changelog

* added new sample log, need to check CSV parsing

* update sample data

Co-authored-by: Marius Iversen <marius.iversen@elastic.co>
(cherry picked from commit 7742fda)

# Conflicts:
#	x-pack/filebeat/module/panw/panos/ingest/pipeline.yml
#	x-pack/filebeat/module/panw/panos/test/global_protect.log-expected.json
#	x-pack/filebeat/module/panw/panos/test/hipmatch.log-expected.json
@mergify mergify bot mentioned this pull request Jun 24, 2021
Closed
mdelapenya added a commit to mdelapenya/beats that referenced this pull request Jun 28, 2021
@mdelapenya
Merge branch 'master' into archive-system-tests
b6dbec3 
* master: (32 commits)
  [Metricbeat] Change Account ID to Project ID in `gcp.billing` module (elastic#26412)
  update libbeat fields.ecs.yml file and ecsVersion to 1.10.0 (elastic#26121)
  [Filebeat] Update AWS ELB ingest pipeline (elastic#26441)
  [FIlebeat] add strict_date_optional_time_nanos date format to PanOS module (elastic#26158)
  Fix the irregular and typo on prometheus module. (elastic#25726)
  [Filebeat] Parse additonal debug data fields for Okta module (elastic#25818)
  fix: update MSSQL Server linux image's Docker registry (elastic#26440)
  Update indexing.go godocs (elastic#26408)
  Do not close filestream harvester if an unexpected error is returned when close.on_state_change.* is enabled (elastic#26411)
  Add support for copytruncate method when rotating input logs with an external tool in `filestream` input (elastic#23457)
  Allow fields with ip_range datatype (elastic#26444)
  Add Anomali ThreatStream support to threatintel module (elastic#26350)
  fix: use the right param type (elastic#26469)
  [Automation] Update elastic stack version to 8.0.0-7640093f for testing (elastic#26460)
  Set SM Filebeat modules as GA (elastic#26226)
  Fix rfc5464 date parsing in the syslog input (elastic#26419)
  Add linked account information into billing metricset (elastic#26285)
  [Filebeat] Update HA Proxy log grok patterns (elastic#25835)
  disable metricbeat logstash test_node_stats (elastic#26436)
  chore: pass BEAT_VERSION when running E2E tests (elastic#26291)
  ...
@P1llus P1llus added backport-v7.14.0 Automated backport with mergify and removed backport-v7.14.0 Automated backport with mergify labels Jun 28, 2021
@marc-gr marc-gr added the needs_integration_sync Changes in this PR need synced to elastic/integrations. label Jun 28, 2021
@legoguy1000 legoguy1000 deleted the 26033-panos-date branch June 28, 2021 13:03
@legoguy1000
Copy link
Contributor Author

@Mergifyio refresh

@mergify
Copy link
Contributor

mergify bot commented Jun 28, 2021

Command refresh: success

Pull request refreshed

@legoguy1000
Copy link
Contributor Author

@Mergifyio backport 7.x

@mergify
Copy link
Contributor

mergify bot commented Jun 28, 2021

Command backport 7.x: success

Backports have been created

@marc-gr marc-gr mentioned this pull request Jun 28, 2021
Merged
6 tasks
marc-gr pushed a commit to marc-gr/beats that referenced this pull request Jun 28, 2021
@legoguy1000 @P1llus @marc-gr
[FIlebeat] add strict_date_optional_time_nanos date format to PanOS m…
047c540 
…odule (elastic#26158)

* elastic#26033: add strict_date_optional_time_nanos format

* update changelog

* added new sample log, need to check CSV parsing

* update sample data

Co-authored-by: Marius Iversen <marius.iversen@elastic.co>
marc-gr added a commit that referenced this pull request Jun 29, 2021
@marc-gr @legoguy1000 @P1llus
[FIlebeat] add strict_date_optional_time_nanos date format to PanOS m…
9c89a29 
…odule (#26158) (#26525)

* #26033: add strict_date_optional_time_nanos format

* update changelog

* added new sample log, need to check CSV parsing

* update sample data

Co-authored-by: Marius Iversen <marius.iversen@elastic.co>

Co-authored-by: Alex Resnick <adr8292@gmail.com>
Co-authored-by: Marius Iversen <marius.iversen@elastic.co>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@P1llus P1llus P1llus approved these changes

Assignees
No one assigned
Labels
backport-v7.14.0 Automated backport with mergify enhancement Integration:Palo Alto needs_integration_sync Changes in this PR need synced to elastic/integrations. v7.14.0
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Palo Alto | include "strict_date_optional_time_nanos" format for date fields
5 participants
@legoguy1000 @elasticmachine @jamiehynds @P1llus @marc-gr