[Filebeat] Add PanOS Global Protect & User ID logs #24927

legoguy1000 · 2021-04-05T00:16:09Z

What does this PR do?

Updates the Panw PanOS module to parse the Palo Alto Global Protect and User ID logs.

Why is it important?

Currently Global Protect and User ID logs are not parsed.

Checklist

My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
I have made corresponding change to the default configuration files
I have added tests that prove my fix is effective or that my feature works
I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Author's Checklist

How to test this PR locally

Related issues

Use cases

Screenshots

Logs

elasticmachine · 2021-04-05T00:21:48Z

💔 Build Failed

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Build Cause: adriansr commented: /test
Start Time: 2021-05-12T06:49:06.511+0000
Duration: 113 min 34 sec
Commit: 657e6de

Test stats 🧪

Test	Results
Failed	0
Passed	13713
Skipped	2285
Total	15998

Trends 🧪

Steps errors

Expand to view the steps failures

`filebeat-packaging-arm-arm - mage package`

Took 9 min 43 sec . View more details on here
Description: mage package

`x-pack/filebeat-packaging-arm-arm - mage package`

Took 7 min 12 sec . View more details on here
Description: mage package

`Error signal`

Took 0 min 0 sec . View more details on here
Description: Error 'hudson.AbortException: script returned exit code 1'

Log output

Expand to view the last 100 lines of log output

[2021-05-12T08:40:45.394Z]  Version:           20.10.3
[2021-05-12T08:40:45.394Z]  API version:       1.41
[2021-05-12T08:40:45.394Z]  Go version:        go1.13.15
[2021-05-12T08:40:45.394Z]  Git commit:        48d30b5
[2021-05-12T08:40:45.394Z]  Built:             Fri Jan 29 14:33:34 2021
[2021-05-12T08:40:45.394Z]  OS/Arch:           linux/arm64
[2021-05-12T08:40:45.394Z]  Context:           default
[2021-05-12T08:40:45.394Z]  Experimental:      true
[2021-05-12T08:40:45.394Z] 
[2021-05-12T08:40:45.394Z] Server: Docker Engine - Community
[2021-05-12T08:40:45.394Z]  Engine:
[2021-05-12T08:40:45.394Z]   Version:          20.10.3
[2021-05-12T08:40:45.394Z]   API version:      1.41 (minimum version 1.12)
[2021-05-12T08:40:45.394Z]   Go version:       go1.13.15
[2021-05-12T08:40:45.394Z]   Git commit:       46229ca
[2021-05-12T08:40:45.394Z]   Built:            Fri Jan 29 14:31:49 2021
[2021-05-12T08:40:45.394Z]   OS/Arch:          linux/arm64
[2021-05-12T08:40:45.394Z]   Experimental:     false
[2021-05-12T08:40:45.394Z]  containerd:
[2021-05-12T08:40:45.394Z]   Version:          1.4.4
[2021-05-12T08:40:45.394Z]   GitCommit:        05f951a3781f4f2c1911b05e61c160e9c30eaa8e
[2021-05-12T08:40:45.394Z]  runc:
[2021-05-12T08:40:45.394Z]   Version:          1.0.0-rc93
[2021-05-12T08:40:45.394Z]   GitCommit:        12644e614e25b05da6fd08a38ffa0cfe1903fdec
[2021-05-12T08:40:45.394Z]  docker-init:
[2021-05-12T08:40:45.394Z]   Version:          0.19.0
[2021-05-12T08:40:45.394Z]   GitCommit:        de40ad0
[2021-05-12T08:40:45.394Z] Change ownership of all files inside the specific folder from root/root to current user/group
[2021-05-12T08:40:45.394Z] Unable to find image 'arm64v8/alpine:3' locally
[2021-05-12T08:40:46.460Z] 3: Pulling from arm64v8/alpine
[2021-05-12T08:40:46.460Z] 595b0fe564bb: Pulling fs layer
[2021-05-12T08:40:47.017Z] 595b0fe564bb: Verifying Checksum
[2021-05-12T08:40:47.017Z] 595b0fe564bb: Download complete
[2021-05-12T08:40:47.017Z] 595b0fe564bb: Pull complete
[2021-05-12T08:40:47.017Z] Digest: sha256:8f18fae117ec6e5777cc62ba78cbb3be10a8a38639ccfb949521abd95c8301a4
[2021-05-12T08:40:47.017Z] Status: Downloaded newer image for arm64v8/alpine:3
[2021-05-12T08:40:49.397Z] Change permissions with write access of all files inside the specific folder
[2021-05-12T08:40:50.671Z] Running in /var/lib/jenkins/workspace/PR-24927-17-972d6bef-a0f1-49f1-8900-ad5d28ab75a9/src/github.com/elastic/beats/build
[2021-05-12T08:40:51.532Z] + rm -rf ve
[2021-05-12T08:40:51.532Z] + find . -type d -name vendor -exec rm -r {} ;
[2021-05-12T08:40:52.770Z] + python .ci/scripts/pre_archive_test.py
[2021-05-12T08:40:53.280Z] Copy ./filebeat/build into build/filebeat/build
[2021-05-12T08:40:53.280Z] Copy ./filebeat/build/golang-crossbuild into build/filebeat/build/golang-crossbuild
[2021-05-12T08:40:53.280Z] Copy ./filebeat/build/package/filebeat-oss/filebeat-oss-linux-arm64.docker/docker-build into build/filebeat/build/package/filebeat-oss/filebeat-oss-linux-arm64.docker/docker-build
[2021-05-12T08:40:53.478Z] Running in /var/lib/jenkins/workspace/PR-24927-17-972d6bef-a0f1-49f1-8900-ad5d28ab75a9/src/github.com/elastic/beats/build
[2021-05-12T08:40:53.557Z] Recording test results
[2021-05-12T08:40:57.662Z] None of the test reports contained any result
[2021-05-12T08:40:57.669Z] [Checks API] No suitable checks publisher found.
[2021-05-12T08:40:58.538Z] + tar --version
[2021-05-12T08:40:59.797Z] + tar --exclude=test-build-artifacts-filebeat-packaging-arm-arm-tgz -czf test-build-artifacts-filebeat-packaging-arm-arm-tgz .
[2021-05-12T08:41:09.946Z] + gsutil --version
[2021-05-12T08:41:21.448Z] Masking supported pattern matches of $FILE_CREDENTIAL
[2021-05-12T08:41:22.670Z] + gcloud auth activate-service-account --key-file ****
[2021-05-12T08:41:24.354Z] Activated service account credentials for: [beats-ci-gcs-plugin@elastic-ci-prod.iam.gserviceaccount.com]
[2021-05-12T08:41:25.479Z] + gsutil -m -q cp -a public-read test-build-artifacts-filebeat-packaging-arm-arm-tgz gs://beats-ci-temp/Beats/beats/PR-24927-17
[2021-05-12T08:41:29.207Z] + python .ci/scripts/search_system_tests.py
[2021-05-12T08:41:29.472Z] [INFO] system-tests=''. If no empty then let's create a tarball
[2021-05-12T08:41:30.368Z] + go clean -modcache
[2021-05-12T08:41:33.874Z] Cleaning up /var/lib/jenkins/workspace/PR-24927-17-972d6bef-a0f1-49f1-8900-ad5d28ab75a9
[2021-05-12T08:41:33.874Z] Client: Docker Engine - Community
[2021-05-12T08:41:33.874Z]  Version:           20.10.3
[2021-05-12T08:41:33.874Z]  API version:       1.41
[2021-05-12T08:41:33.874Z]  Go version:        go1.13.15
[2021-05-12T08:41:33.874Z]  Git commit:        48d30b5
[2021-05-12T08:41:33.874Z]  Built:             Fri Jan 29 14:33:34 2021
[2021-05-12T08:41:33.874Z]  OS/Arch:           linux/arm64
[2021-05-12T08:41:33.874Z]  Context:           default
[2021-05-12T08:41:33.874Z]  Experimental:      true
[2021-05-12T08:41:33.874Z] 
[2021-05-12T08:41:33.874Z] Server: Docker Engine - Community
[2021-05-12T08:41:33.874Z]  Engine:
[2021-05-12T08:41:33.874Z]   Version:          20.10.3
[2021-05-12T08:41:33.874Z]   API version:      1.41 (minimum version 1.12)
[2021-05-12T08:41:33.874Z]   Go version:       go1.13.15
[2021-05-12T08:41:33.874Z]   Git commit:       46229ca
[2021-05-12T08:41:33.874Z]   Built:            Fri Jan 29 14:31:49 2021
[2021-05-12T08:41:33.874Z]   OS/Arch:          linux/arm64
[2021-05-12T08:41:33.874Z]   Experimental:     false
[2021-05-12T08:41:33.874Z]  containerd:
[2021-05-12T08:41:33.874Z]   Version:          1.4.4
[2021-05-12T08:41:33.874Z]   GitCommit:        05f951a3781f4f2c1911b05e61c160e9c30eaa8e
[2021-05-12T08:41:33.874Z]  runc:
[2021-05-12T08:41:33.874Z]   Version:          1.0.0-rc93
[2021-05-12T08:41:33.874Z]   GitCommit:        12644e614e25b05da6fd08a38ffa0cfe1903fdec
[2021-05-12T08:41:33.874Z]  docker-init:
[2021-05-12T08:41:33.874Z]   Version:          0.19.0
[2021-05-12T08:41:33.874Z]   GitCommit:        de40ad0
[2021-05-12T08:41:33.874Z] Change ownership of all files inside the specific folder from root/root to current user/group
[2021-05-12T08:41:34.941Z] Change permissions with write access of all files inside the specific folder
[2021-05-12T08:41:35.151Z] Running in /var/lib/jenkins/workspace/PR-24927-17-972d6bef-a0f1-49f1-8900-ad5d28ab75a9
[2021-05-12T08:41:39.180Z] Failed in branch filebeat-packaging-arm-arm
[2021-05-12T08:41:39.244Z] Stage "Packaging-Pipeline" skipped due to earlier failure(s)
[2021-05-12T08:41:39.295Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-24927/src/github.com/elastic/beats
[2021-05-12T08:41:39.908Z] Running on Jenkins in /var/lib/jenkins/workspace/Beats_beats_PR-24927
[2021-05-12T08:41:39.954Z] [INFO] getVaultSecret: Getting secrets
[2021-05-12T08:41:39.992Z] Masking supported pattern matches of $VAULT_ADDR or $VAULT_ROLE_ID or $VAULT_SECRET_ID
[2021-05-12T08:41:40.646Z] + chmod 755 generate-build-data.sh
[2021-05-12T08:41:40.646Z] + ./generate-build-data.sh https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-24927/ https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-24927/runs/17 FAILURE 6753872
[2021-05-12T08:41:40.646Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-24927/runs/17/steps/?limit=10000 -o steps-info.json
[2021-05-12T08:41:42.508Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-24927/runs/17/tests/?status=FAILED -o tests-errors.json

💚 Flaky test report

Tests succeeded.

Expand to view the summary

Test stats 🧪

Test	Results
Failed	0
Passed	13713
Skipped	2285
Total	15998

elasticmachine · 2021-04-09T15:15:14Z

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

mergify · 2021-04-19T22:21:34Z

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b 24722/4-palo-new-logs upstream/24722/4-palo-new-logs
git merge upstream/master
git push upstream 24722/4-palo-new-logs

mergify · 2021-04-27T19:32:43Z

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b 24722/4-palo-new-logs upstream/24722/4-palo-new-logs
git merge upstream/master
git push upstream 24722/4-palo-new-logs

mergify · 2021-04-28T15:59:47Z

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b 24722/4-palo-new-logs upstream/24722/4-palo-new-logs
git merge upstream/master
git push upstream 24722/4-palo-new-logs

mergify · 2021-05-03T14:02:32Z

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b 24722/4-palo-new-logs upstream/24722/4-palo-new-logs
git merge upstream/master
git push upstream 24722/4-palo-new-logs

mergify · 2021-05-06T08:47:25Z

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b 24722/4-palo-new-logs upstream/24722/4-palo-new-logs
git merge upstream/master
git push upstream 24722/4-palo-new-logs

mergify · 2021-05-10T16:50:51Z

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b 24722/4-palo-new-logs upstream/24722/4-palo-new-logs
git merge upstream/master
git push upstream 24722/4-palo-new-logs

adriansr

Thanks a lot for your contribution!

Other than the conflicts and the event.outcome issue, this is looking good to me.

x-pack/filebeat/module/panw/panos/_meta/fields.yml

x-pack/filebeat/module/panw/panos/ingest/pipeline.yml

legoguy1000 · 2021-05-11T23:12:53Z

@adriansr Should be GTG now.

adriansr · 2021-05-12T06:48:43Z

/test

adriansr

LGTM, ignoring failing arm build (known issue)

Updates the Panw PanOS module to parse the Palo Alto Global Protect and User ID logs. (cherry picked from commit 99ba1a2)

Updates the Panw PanOS module to parse the Palo Alto Global Protect and User ID logs. (cherry picked from commit 99ba1a2) Co-authored-by: Alex Resnick <adr8292@gmail.com>

botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Apr 5, 2021

P1llus added the Team:Security-External Integrations label Apr 7, 2021

botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Apr 7, 2021

legoguy1000 force-pushed the 24722/4-palo-new-logs branch 2 times, most recently from 5e23b5b to bbc2cd8 Compare April 9, 2021 15:14

legoguy1000 marked this pull request as ready for review April 9, 2021 15:15

jamiehynds added the enhancement label Apr 19, 2021

jamiehynds added the 7.14 Candidate label Apr 22, 2021

legoguy1000 force-pushed the 24722/4-palo-new-logs branch from bbc2cd8 to c7a9127 Compare April 22, 2021 17:09

legoguy1000 force-pushed the 24722/4-palo-new-logs branch from c7a9127 to 902a961 Compare April 27, 2021 19:45

legoguy1000 force-pushed the 24722/4-palo-new-logs branch from 902a961 to caddfe7 Compare April 28, 2021 20:41

legoguy1000 force-pushed the 24722/4-palo-new-logs branch from caddfe7 to 83ebcf6 Compare May 3, 2021 14:24

andrewkroh assigned adriansr May 3, 2021

andrewkroh added the needs_integration_sync Changes in this PR need synced to elastic/integrations. label May 3, 2021

legoguy1000 force-pushed the 24722/4-palo-new-logs branch from 83ebcf6 to 9958599 Compare May 6, 2021 23:18

adriansr suggested changes May 11, 2021

View reviewed changes

x-pack/filebeat/module/panw/panos/_meta/fields.yml Outdated Show resolved Hide resolved

x-pack/filebeat/module/panw/panos/ingest/pipeline.yml Show resolved Hide resolved

x-pack/filebeat/module/panw/panos/ingest/pipeline.yml Outdated Show resolved Hide resolved

legoguy1000 added 6 commits May 11, 2021 21:56

elastic#24724: Add Global Protect logs

f10b0ff

Add User ID (more changes needed)

01a6c7e

update userid ingest

f71ccc8

update fields

d426748

update fields

5db554a

copy source.user to user

e5eb00c

legoguy1000 added 3 commits May 11, 2021 21:56

additional logs

24cb1e7

comments from @adriansr

6c443fd

update generated data

657e6de

legoguy1000 force-pushed the 24722/4-palo-new-logs branch from 0e1c297 to 657e6de Compare May 11, 2021 22:52

adriansr approved these changes May 12, 2021

View reviewed changes

adriansr added the backport-v7.14.0 Automated backport with mergify label May 12, 2021

adriansr merged commit 99ba1a2 into elastic:master May 12, 2021

mergify bot pushed a commit that referenced this pull request May 12, 2021

[Filebeat] Add PanOS Global Protect & User ID logs (#24927)

92ff38c

Updates the Panw PanOS module to parse the Palo Alto Global Protect and User ID logs. (cherry picked from commit 99ba1a2)

mergify bot mentioned this pull request May 12, 2021

[7.x](backport #24927) [Filebeat] Add PanOS Global Protect & User ID logs #25680

Merged

legoguy1000 deleted the 24722/4-palo-new-logs branch May 12, 2021 11:03

adriansr pushed a commit that referenced this pull request Jun 24, 2021

[Filebeat] Add PanOS Global Protect & User ID logs (#24927)

8b375a8

Updates the Panw PanOS module to parse the Palo Alto Global Protect and User ID logs. (cherry picked from commit 99ba1a2)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Filebeat] Add PanOS Global Protect & User ID logs #24927

[Filebeat] Add PanOS Global Protect & User ID logs #24927

legoguy1000 commented Apr 5, 2021 •

edited

Loading

elasticmachine commented Apr 5, 2021 •

edited by jenkins-beats-ci bot

Loading

Build stats

Test stats 🧪

Trends 🧪

`filebeat-packaging-arm-arm - mage package`

`x-pack/filebeat-packaging-arm-arm - mage package`

`Error signal`

Test stats 🧪

elasticmachine commented Apr 9, 2021

mergify bot commented Apr 19, 2021

mergify bot commented Apr 27, 2021

mergify bot commented Apr 28, 2021

mergify bot commented May 3, 2021

mergify bot commented May 6, 2021

mergify bot commented May 10, 2021

adriansr left a comment

legoguy1000 commented May 11, 2021

adriansr commented May 12, 2021

adriansr left a comment •

edited

Loading

[Filebeat] Add PanOS Global Protect & User ID logs #24927

[Filebeat] Add PanOS Global Protect & User ID logs #24927

Conversation

legoguy1000 commented Apr 5, 2021 • edited Loading

What does this PR do?

Why is it important?

Checklist

Author's Checklist

How to test this PR locally

Related issues

Use cases

Screenshots

Logs

elasticmachine commented Apr 5, 2021 • edited by jenkins-beats-ci bot Loading

💔 Build Failed

Build stats

Test stats 🧪

Trends 🧪

Steps errors

filebeat-packaging-arm-arm - mage package

x-pack/filebeat-packaging-arm-arm - mage package

Error signal

Log output

💚 Flaky test report

Test stats 🧪

elasticmachine commented Apr 9, 2021

mergify bot commented Apr 19, 2021

mergify bot commented Apr 27, 2021

mergify bot commented Apr 28, 2021

mergify bot commented May 3, 2021

mergify bot commented May 6, 2021

mergify bot commented May 10, 2021

adriansr left a comment

Choose a reason for hiding this comment

legoguy1000 commented May 11, 2021

adriansr commented May 12, 2021

adriansr left a comment • edited Loading

Choose a reason for hiding this comment

legoguy1000 commented Apr 5, 2021 •

edited

Loading

elasticmachine commented Apr 5, 2021 •

edited by jenkins-beats-ci bot

Loading

`filebeat-packaging-arm-arm - mage package`

`x-pack/filebeat-packaging-arm-arm - mage package`

`Error signal`

adriansr left a comment •

edited

Loading