Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update ExtraHop Integration #4481

Merged
merged 41 commits into from
Nov 11, 2019
Merged

Update ExtraHop Integration #4481

merged 41 commits into from
Nov 11, 2019

Conversation

Dan-at-Extrahop
Copy link
Contributor

@Dan-at-Extrahop Dan-at-Extrahop commented Sep 25, 2019

Status

Ready

Related Issues

Description

New version of the ExtraHop integration (full featured). Complete content package with new powerful commands, real-time incident creation via REST, associated playbooks, enhanced system playbook, end-to-end ticket tracking through a playbook and a field trigger script. This is an ExtraHop supported integration that has been in the works as a partnership since last year, most recently we've been working with:

  • Tyler R
  • Matt C
  • Prasen S
  • Marketing team

Screenshots

image

Related PRs

Required version of Demisto

4.5

Does it break backward compatibility?

  • No
    • This is a complete rewrite of the integration which consumes all existing command functionality, but renamed commands, inputs, and outputs. Renamed to be ExtraHop v2.

Deprecate old ExtraHop integration?

  • Yes

Must have

Dependencies

Additional changes

Technical writer review

Mention and link to the files that require a technical writer review.

 - full featured integration includes new and improved commands
 - playbook framework for ExtraHop Detection incident type
 - updated test playbook
 - enhanced Endpoint Enrichment playbook to include ExtraHop
@welcome
Copy link

welcome bot commented Sep 25, 2019

Hi and welcome to the Demisto Content project! Thank you and congrats on your first pull request, we will review it soon! Until then you can check out our documentation for more details. We would be thrilled to see you get involved in our Slack DFIR community for discussions. Hope you have a great time here :)

@CLAassistant
Copy link

CLAassistant commented Sep 25, 2019

CLA assistant check
All committers have signed the CLA.

 - unquoted = in playbook yml
 - detaileddescription in integration yml
@content-bot content-bot added the Contribution Thank you! Contributions are always welcome! label Sep 25, 2019
@content-bot content-bot changed the base branch from master to Dan-at-Extrahop_master_base September 25, 2019 08:00
@content-bot
Copy link
Collaborator

Thank you for your contribution. Your generosity and caring are unrivaled! Rest assured - our content wizard @yaakovi will very shortly look over your proposed changes.

@content-bot content-bot requested a review from yaakovi September 25, 2019 08:00
 - add 'secrets' to whitelist
 - move script to folder
 - add tests to playbooks and scripts
 - add descriptions to all playbook tasks
 - update integration argument display names
 - lint issue
 - add script key to yml
 - only failure should be known/expected backwards compatibility breakage
	- updated references in playbooks and script
	- need to deprecate old ExtraHop integration
Copy link
Contributor

@yaakovi yaakovi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

very nice work, please fix the comment.

Integrations/ExtraHop_v2/CHANGELOG.md Outdated Show resolved Hide resolved
Playbooks/playbook-Endpoint_Enrichment_-_Generic_v2.1.yml Outdated Show resolved Hide resolved
Playbooks/playbook-Endpoint_Enrichment_-_Generic_v2.1.yml Outdated Show resolved Hide resolved
Integrations/ExtraHop_v2/ExtraHop_v2_documentation.html Outdated Show resolved Hide resolved
Playbooks/playbook-Endpoint_Enrichment_-_Generic_v2.1.yml Outdated Show resolved Hide resolved
Integrations/ExtraHop_v2/ExtraHop_v2.py Outdated Show resolved Hide resolved
Integrations/ExtraHop_v2/ExtraHop_v2.py Show resolved Hide resolved
Integrations/ExtraHop_v2/ExtraHop_v2.py Outdated Show resolved Hide resolved
Integrations/ExtraHop_v2/ExtraHop_v2.py Outdated Show resolved Hide resolved
Integrations/ExtraHop_v2/ExtraHop_v2.py Outdated Show resolved Hide resolved
@yaakovi yaakovi requested a review from michalgold October 6, 2019 08:36
@yaakovi
Copy link
Contributor

yaakovi commented Oct 6, 2019

@michalgold please go over the included playbooks.

@content-bot
Copy link
Collaborator

A lengthy period of time has transpired since the PR was reviewed. @Dan-at-Extrahop Please address the reviewer's comments and push your committed changes.

@yaakovi yaakovi requested review from idovandijk and removed request for michalgold October 16, 2019 13:26
@yaakovi
Copy link
Contributor

yaakovi commented Oct 16, 2019

@idovandijk please review the playbooks here

Copy link
Contributor

@idovandijk idovandijk left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

General Points

  • Well done, the playbooks look great!
  • After configuring your integration, I used the "Test" button which completed successfully. However, when running your test playbook, the very first task resulted in an error:
    image

so currently the "test" command is misleading. Please fix it to properly test whether the integration works and can be used with the entered credentials.

  • Please edit all your different playbooks to use commands that aren't tied to a specific integration. What I mean, is that by doing this:
    image

you are telling it to run extrahop-device-search only using "ExtraHop v2" integration. This is unnecessary, and is usually used when one wants to run a general command like !url using only a certain integration that supports it (out of many others). Moreover, if you release a new version of ExtraHop, these tasks won't work anymore, because they're tied to ExtraHop v2. It should look like this:
image

  • Please change the task names to meet our conventions. You can refer to these playbooks, which meet our standards and are good examples, to see how we name different tasks: IP Enrichment - Internal - Generic v2, Extract Indicators From File - Generic v2. Note capitalization of letters and the meaningfulness and conciseness of the names. Also, task names shouldn't be the same as the name of the command behind them, they should add a little more insight into what's going to happen, in a few words.

  • I can't seem to see alerts, been having this loading screen for over a day:
    image

I will be adding additional playbook-specific points in each playbook file.
Please note that I offered in one of my points to create a call regarding a certain aspect of the playbook. We would like to understand better from you how this playbook should be used, to tweak this to better fit the usage in Demisto. Ping me if you think you'd be able to make a call with @yaakovi and me :)

@Dan-at-Extrahop
Copy link
Contributor Author

@idovandijk Thank you for the review.

  1. From the loading screen you posted, I noticed that your ExtraHop Reveal(x) instance needs to be upgraded to 7.8 which is the minimum version for this integration. Here's a guide to upgrade, follow the Upgrade the firmware steps.
  2. It looks like you aren't using the latest commit from this branch, I implemented @yaakovi suggestion on removing "ExtraHop v2" version identifiers from all playbook tasks, and I've also updated error messages so that any task errors will be more verbose. From your Test playbook error screenshot I can see the old version of the errors, please pull the latest commit here and try again.

I'm unable to see the playbook-specific points that you added, please advise where to see your comments. Yes, I'll email you now to setup a time for a call, thank you.

@content-bot
Copy link
Collaborator

The CircleCI build failed again. @Dan-at-Extrahop take a look at the build details here - and try and fix the issues so that we can merge your proposed changes as soon as possible.


Failed Build Steps

  • Validate Files and Yaml
  • Build Content Descriptor

@richardbluestone
Copy link
Contributor

@yaakovi I'm going through the integration yml file and I can't review it properly as I don't understand the context output descriptions. For example, "ExtraHop.Record.Source.to" is defined as "to". ExtraHop.Record.Source.till is defined as "till". There are 100's of these. I need more information to understand.

@content-bot
Copy link
Collaborator

The CircleCI build failed again. @Dan-at-Extrahop take a look at the build details here - and try and fix the issues so that we can merge your proposed changes as soon as possible.


Failed Build Steps

  • Validate Files and Yaml
  • Build Content Descriptor

Copy link
Contributor

@yaakovi yaakovi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Dan-at-Extrahop please help @richardbluestone better understand the integration output descriptions so we can merge the PR in time for the release.

@Dan-at-Extrahop
Copy link
Contributor Author

@richardbluestone Thanks for the review, I'm happy to answer any questions to assist. These ExtraHop.Record.Source outputs refer specifically to only our extrahop-query-records command, which is a minor part of the integration used to pull transactional records out of ExtraHop. The descriptions present are the current level of detail available in our product today, since these fields are used generally to cover all 60+ record types. We do not believe this will be an issue for our joint customers, since as mentioned this is the level of detail available for records in ExtraHop and it's understandable in the context of a particular record. Beyond these context outputs, all of the commands and arguments have full descriptions, and the rest of the context outputs have detailed descriptions available. I appreciate your review to get this approved in time for the upcoming release.

@Dan-at-Extrahop
Copy link
Contributor Author

@idovandijk all playbook changes have been pushed. @yaakovi the two outstanding build issues are:

  • It does not like the mobile layout. Do I need to remove it?
  • It does not like keyword ticket in our incident field named detectionticketed. It seems like this should only complain if we tried to make an incident field that is explicitly named ticket, but that's not the case here.

Lastly just a reminder note for whatever needs to be done to also deprecate the existing ExtraHop integration when this new ExtraHop Reveal(x) integration is released.

This should be all of the changes required for us to merge in time for 11/15 release, with the exception of the review from @richardbluestone.

@content-bot
Copy link
Collaborator

The CircleCI build failed again. @Dan-at-Extrahop take a look at the build details here - and try and fix the issues so that we can merge your proposed changes as soon as possible.


Failed Build Steps

  • Validate Files and Yaml

@idovandijk
Copy link
Contributor

@yaron-libman Need your review for descriptions of playbooks and tasks, and task names :)

@content-bot
Copy link
Collaborator

The CircleCI build failed again. @Dan-at-Extrahop take a look at the build details here - and try and fix the issues so that we can merge your proposed changes as soon as possible.


Failed Build Steps

  • Validate Files and Yaml

Copy link
Contributor

@idovandijk idovandijk left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Playbooks are ready

@richardbluestone richardbluestone removed their assignment Nov 10, 2019
@yaakovi yaakovi merged commit 42867af into demisto:Dan-at-Extrahop_master_base Nov 11, 2019
@content-bot content-bot mentioned this pull request Nov 11, 2019
5 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Contribution Thank you! Contributions are always welcome! docs-approved
Projects
None yet
Development

Successfully merging this pull request may close these issues.

8 participants