Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Gem Security pack #33434

Conversation

liormgem
Copy link
Contributor

@liormgem liormgem commented Mar 19, 2024

Contributing to Cortex XSOAR Content

Make sure to register your contribution by filling the contribution registration form

The Pull Request will be reviewed only after the contribution registration form is filled.

Status

  • In Progress
  • Ready
  • In Hold - (Reason for hold)

Related Issues

fixes: link to the issue

Description

Pack Gem integrates with the Gem Security platform.
Pack includes:

  • 1 Automation
  • 3 Classifiers
  • 16 Incident Fields
  • 1 Incident Type
  • 1 Integration
  • 1 Layout
  • 3 Playbooks
  • 1 Pre-process Rule

Must have

  • Tests
  • Documentation

@CLAassistant
Copy link

CLAassistant commented Mar 19, 2024

CLA assistant check
All committers have signed the CLA.

@kgal-pan
Copy link
Contributor

@liormgem - Thanks for the contribution. When filling out the Contribution Form, use gem-security-18933 as your Partner ID.

Pack includes:

1 Automation
3 Classifiers
16 Incident Fields
1 Incident Type
1 Integration
1 Layout
3 Playbooks
1 Pre-process Rule
* Fix post commit validation issues

* Fix tests coverage
@liormgem liormgem changed the title Gem Security pack Commit Gem Security pack Mar 20, 2024
@merit-maita
Copy link
Contributor

@liormgem - Thanks for the contribution. When filling out the Contribution Form, use gem-security-18933 as your Partner ID.

@liormgem can you please address this comment, so i can go ahead and review the pr

@melamedbn
Copy link
Contributor

Hi @liormgem,

I've reviewed your contribution. Thank you for your effort and dedication. I would appreciate it if you could address the following feedback:

Classification and Mapping

  1. Classifier Functionality: It may be beneficial to review the classifier to ensure it correctly classifies alerts. If there are any improvements needed, we can work on them together.
  2. Mapping Differences: Could you help me understand if there are any differences between the mapping done in 'Gem Mapper' and 'Gem Mapper Webhook'? This would help us ensure consistency and efficiency in our mapping process.

Playbooks

Gem Handle Alert for Root Usage

  1. Task '1' Adjustment: To ensure the playbook closes automatically, let's adjust the else path to lead either to the Done section header or to an error handling path.
  2. Inputs and Outputs Clarification: I noticed that the playbook has no inputs and outputs. Is this intended? If so, could you explain the reasoning behind it?
  3. Question Marks Addition: Let's add question marks where needed, such as in Task '1', to ensure clarity and understanding for anyone reviewing the playbook.
  4. Grid Alignment: Could you please fix the arrow exiting task '4' so that it leads correctly to task '11' on the grid?
  5. Task '10' and '11' Clarification: It would be helpful to clarify the need for both tasks '10' and '11'. Additionally, let's enhance the title of Task '10' to provide more context and clarity.
  6. Task '2' Adjustment: For Task '2', let's remove the index position ".[1]" and ".[0]" and use the path instead.
  7. Task '6' Enhancement: For Task '6', let's first deduplicate and then count the number of unique values. Additionally, add a conditional task that counts the usernames and checks using 'greater or equals to'.
  8. Task '7' Adjustment: Similar to Task '1', let's ensure the else path in Task '7' leads either to the Done section header or to an error handling path.

Gem Handle ec2

  1. Task '7' Clarification: What happens if it's not an EC2 instance?
  2. Task '1' Title Enhancement: Let's put a more descriptive title for Task '1'.
  3. Task '1' Slack User Check: We need to make sure the Slack user is provided via the inputs or else the ASK task might not be relevant.

Gem Validate triggering event

  1. Grid Alignment: Please fix the arrow exiting the Playbook Triggered header directing to task '10' on the grid.
  2. Question Marks Addition: Add question marks where needed for clarity.
  3. Task '10' Adjustment: To ensure the playbook closes automatically, let's adjust the else path in Task '10' to lead either to the Done section header or to an error handling path.

Incident Fields

  1. New Incident Fields: Why were new incident fields created if most are available in our common types (OOTB)? When using OOTB fields, they will be mapped in a way that fits more OOTB content items.

Layout

  1. Section Width Adjustment: Let's adjust the width of the sections so we won't have blank spaces.

Please let me know if there are any more revisions needed or if there's anything else I can assist you with.

Best regards,
Ben

* Fix playbooks

* Format playbooks and fix Gem Alert Classifier

* Fix blank space in Gem Layout

* Fix incident fields
@liormgem
Copy link
Contributor Author

Hi @melamedbn,
Thank you for the feedback.
The new Incident Fields were changed to the relevant OOTB fields in case we found one.

All your remarks were addressed and fixed, answers to your clarification questions are below:

Classification and Mapping
2. Mapping Differences: Could you help me understand if there are any differences between the mapping done in 'Gem Mapper' and 'Gem Mapper Webhook'? This would help us ensure consistency and efficiency in our mapping process.

There are small differences in the way the webhook sends the Gem alerts and the way the fetching Endpoint serves it.
For example - the Gem Alert ID in the webhook is mapped from event.alert_id and in the regular mapper from metadata.alert_id

Playbooks
Gem Handle Alert for Root Usage
2. Inputs and Outputs Clarification: I noticed that the playbook has no inputs and outputs. Is this intended? If so, could you explain the reasoning behind it?

It is.
It gets all the information it needs from the relevant Gem Alert and is meant to be attached to one.

Gem Handle ec2

  1. Task '7' Clarification: What happens if it's not an EC2 instance?

The playbook is not relevant in this case.
Will add a Done section header.

  1. Task '1' Slack User Check: We need to make sure the Slack user is provided via the inputs or else the ASK task might not be relevant.

It’s still relevant manually and I think that’s ok

Thanks,
Lior

@merit-maita merit-maita requested review from JudahSchwartz and removed request for merit-maita, yucohen and michal-dagan March 31, 2024 09:08
@JudahSchwartz JudahSchwartz requested review from JasBeilin and removed request for JudahSchwartz March 31, 2024 09:12
@JasBeilin JasBeilin requested review from maimorag and removed request for JasBeilin March 31, 2024 11:13
@JasBeilin JasBeilin removed their assignment Mar 31, 2024
@melamedbn
Copy link
Contributor

Hi @melamedbn, Thank you for the feedback. The new Incident Fields were changed to the relevant OOTB fields in case we found one.

All your remarks were addressed and fixed, answers to your clarification questions are below:

Classification and Mapping
2. Mapping Differences: Could you help me understand if there are any differences between the mapping done in 'Gem Mapper' and 'Gem Mapper Webhook'? This would help us ensure consistency and efficiency in our mapping process.

There are small differences in the way the webhook sends the Gem alerts and the way the fetching Endpoint serves it. For example - the Gem Alert ID in the webhook is mapped from event.alert_id and in the regular mapper from metadata.alert_id

Playbooks
Gem Handle Alert for Root Usage
2. Inputs and Outputs Clarification: I noticed that the playbook has no inputs and outputs. Is this intended? If so, could you explain the reasoning behind it?

It is. It gets all the information it needs from the relevant Gem Alert and is meant to be attached to one.

Gem Handle ec2

  1. Task '7' Clarification: What happens if it's not an EC2 instance?

The playbook is not relevant in this case. Will add a Done section header.

  1. Task '1' Slack User Check: We need to make sure the Slack user is provided via the inputs or else the ASK task might not be relevant.

It’s still relevant manually and I think that’s ok

Thanks, Lior

Great. Please modify the content items and ping me back for review verification.

Ben

@kgal-pan kgal-pan reopened this May 6, 2024
@liormgem
Copy link
Contributor Author

liormgem commented May 6, 2024

Hi @melamedbn, Thank you for the feedback. The new Incident Fields were changed to the relevant OOTB fields in case we found one.
All your remarks were addressed and fixed, answers to your clarification questions are below:

Classification and Mapping
2. Mapping Differences: Could you help me understand if there are any differences between the mapping done in 'Gem Mapper' and 'Gem Mapper Webhook'? This would help us ensure consistency and efficiency in our mapping process.

There are small differences in the way the webhook sends the Gem alerts and the way the fetching Endpoint serves it. For example - the Gem Alert ID in the webhook is mapped from event.alert_id and in the regular mapper from metadata.alert_id

Playbooks
Gem Handle Alert for Root Usage
2. Inputs and Outputs Clarification: I noticed that the playbook has no inputs and outputs. Is this intended? If so, could you explain the reasoning behind it?

It is. It gets all the information it needs from the relevant Gem Alert and is meant to be attached to one.

Gem Handle ec2

  1. Task '7' Clarification: What happens if it's not an EC2 instance?

The playbook is not relevant in this case. Will add a Done section header.

  1. Task '1' Slack User Check: We need to make sure the Slack user is provided via the inputs or else the ASK task might not be relevant.

It’s still relevant manually and I think that’s ok
Thanks, Lior

Great. Please modify the content items and ping me back for review verification.

Ben

Hi @melamedbn,
Items modified.

@kgal-pan
Copy link
Contributor

kgal-pan commented May 6, 2024

@liormgem - See https://app.circleci.com/pipelines/github/demisto/content/359788/workflows/d8957b00-e011-469b-abea-dcd1ab231f03/jobs/747032?invite=true#step-116-217_45. Can you please sync your fork with upstream?

Copy link
Contributor

@maimorag maimorag left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @liormgem,
Thank you for your contribution!
Good work :)

Comment on lines 27 to 38
- display: Service Account ID
additionalinfo: The Service Account ID to use for connection
name: client_id
type: 0
section: Connect
required: true
- display: Service Account Secret
additionalinfo: The Service Account Secret to use for connection
name: client_secret
type: 4
section: Connect
required: true
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Type 4 is deprecated, please use type 9 instead.
For example:

Suggested change
- display: Service Account ID
additionalinfo: The Service Account ID to use for connection
name: client_id
type: 0
section: Connect
required: true
- display: Service Account Secret
additionalinfo: The Service Account Secret to use for connection
name: client_secret
type: 4
section: Connect
required: true
- display: Service Account ID
name: credentials
defaultvalue: ""
type: 9
required: true
section: Connect
displaypassword: Service Account Secret

Comment on lines 578 to 579
client_id=params['client_id'],
client_secret=params['client_secret']
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

After changing to type 9 since type 4 is deprecated:

Suggested change
client_id=params['client_id'],
client_secret=params['client_secret']
client_id=demisto.getParam('credentials')['identifier'],
client_secret=demisto.getParam('credentials')['password']

Comment on lines +85 to +123
def http_request(self, method: str, url_suffix='', full_url=None, headers=None, json_data=None, params=None, auth=True):
"""
Sends an HTTP request to the specified URL, adding the required headers and authentication token.

Args:
method (str): The HTTP method to use (e.g., GET, POST, PUT, DELETE).
url_suffix (str, optional): The URL suffix to append to the base URL. Defaults to ''.
full_url (str, optional): The full URL to send the request to. If provided, `url_suffix` will be ignored.
Defaults to None.
headers (dict, optional): Additional headers to include in the request. Defaults to None.
json_data (dict, optional): JSON data to include in the request body. Defaults to None.
params (dict, optional): Query parameters to include in the request URL. Defaults to None.
auth (bool, optional): Whether to include authentication headers. Defaults to True.

Returns:
dict: The response from the HTTP request.

Raises:
Exception: If the request fails.

"""
if auth:
headers = headers or {}
headers['Authorization'] = f'Bearer {self._auth_token}'
try:
response = super()._http_request(
method=method,
url_suffix=url_suffix,
full_url=full_url,
headers=headers,
json_data=json_data,
params=params,
raise_on_status=True
)
demisto.debug(f"Got response: {response}")
return response
except DemistoException as e:
demisto.error(f"Failed to execute {method} request to {url_suffix}. Error: {str(e)}")
raise Exception(f"Failed to execute {method} request to {url_suffix}. Error: {str(e)}")
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

you can use http_request from the CommonServerPython http_request and to add error handler which outputs the new formatted error message.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm using this function to add the Authorization header to the request

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@liormgem I still recommend using the 'http_request' function from CommonServerPython.
It's supported within XSOAR, ensuring compatibility, and we actively maintain it, promptly adapting it to any changes that may arise.

Additionally, I'm available to assist you in integrating it into your code if needed.

However, if you still prefer to use your own 'http_request', you can proceed with that choice and resolve this note.

Comment on lines 790 to 794
if not time_start:
raise DemistoException('Start time is a required parameter.')

if not time_end:
raise DemistoException('End time is a required parameter.')
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If start_time and end_time is required parameters then the user can't run the command without them.

Comment on lines 830 to 840
if not entity_id:
raise DemistoException('Entity ID is a required parameter.')

if not entity_type:
raise DemistoException('Entity Type is a required parameter.')

if not start_time:
raise DemistoException('Start time is a required parameter.')

if not end_time:
raise DemistoException('End time is a required parameter.')
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same as above

Comment on lines 1072 to 1081
if not action:
raise DemistoException('Action is a required parameter.')
if not entity_id:
raise DemistoException('Entity ID is a required parameter.')
if not entity_type:
raise DemistoException('Entity type is a required parameter.')
if not alert_id:
raise DemistoException('Alert ID is a required parameter.')
if not resource_id:
raise DemistoException('Resource ID is a required parameter.')
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please make sure those required in the command as well

Comment on lines 1099 to 1102
if not threat_id:
raise DemistoException('Threat ID is a required parameter.')
if not comment:
raise DemistoException('Comment is a required parameter.')
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please make sure those required in the command as well

@liormgem
Copy link
Contributor Author

liormgem commented May 6, 2024

Hi @maimorag,
Thank you for the review.
All comments were fixed.

I'm working on uploading the example video to the demisto-assets repo and then I'll submit the Contribution form.

Lior

Copy link
Contributor

@ssokolovich ssokolovich left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey @liormgem

A couple of notes from my side.

  1. Note that I usually suggest not using the exact brand GEM in the commands. In case a customer would like to copy your integration, he will need to change all the OOTB playbooks as well (there is an option to choose the non-brand command from the PB itself). This isn't required, however, a tip from my experience.
    image
  2. Usually we add a Done task at the end. Just a cosmetic tip.
    image
  3. Note that you don't have a playbook associated with your incident type. Once the investigation started the user would need to choose how to handle it automatically.
    image
  4. Extracting all indicators from all fields might impact server performance, hence, suggesting to extract indicators from only necessary fields (basically the ones that you mapped).
    image

Let me know if you need me to elaborate more on a point or help with anything else.
Cheers!

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would like to suggest moving some of the fields under the case details to the investigation data (or even expanding the case details section), since if all the fields may be mapped, the user might miss that info.
Not mandatory but nice to have (of course analysts will appreciate it :) ).
image

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @ssokolovich,
Thank you for the review.

  1. I think that would be fine with the amount of content we have.
  2. Will be Added
  3. Customers we consulted prefer to not have a default playbook
  4. There are indicators in other parts of the incident, not just in the mapped fields and we would like it to be processed. The number of incidents is low enough (not less than 10 a day) to not have an impact.
  5. We will leave it as it is, for now, it's extra information.

Lior

Copy link
Contributor

@ssokolovich ssokolovich May 8, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cool! so just let me know when you fix # 2 and also please remember to update the playbook images.
Then waiting for your update @liormgem.
Cheers!

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great, done.
@ssokolovich

@maimorag maimorag added the ready-for-instance-test In contribution PRs, this label will cause a trigger of a build with a modified pack from the PR. label May 9, 2024
@maimorag
Copy link
Contributor

maimorag commented May 9, 2024

Hey @liormgem, the code looks good!

We're ready for a demo. Please check this page, and let me know when you're available for one over DFIR.
Feel free also to send me a recording of a demo.

I'll be on PTO from May 12th to May 19th. Apologies for any delay in advance.

@maimorag maimorag added pending-demo Demo pending and removed pending-contributor The PR is pending the response of its creator labels May 9, 2024
@maimorag maimorag merged commit fa6cb54 into demisto:contrib/Gem-Security_gem_security_pack May 9, 2024
32 of 33 checks passed
Copy link

github-actions bot commented May 9, 2024

Thank you for your contribution. Your external PR has been merged and the changes are now included in an internal PR for further review. The internal PR will be merged to the master branch within 3 business days.

maimorag pushed a commit that referenced this pull request May 9, 2024
* Gem Security pack Commit

Pack includes:

1 Automation
3 Classifiers
16 Incident Fields
1 Incident Type
1 Integration
1 Layout
3 Playbooks
1 Pre-process Rule

* Ci fix (#5)

* Fix post commit validation issues

* Fix tests coverage

* Fix indent (#6)

* Cr fix (#7)

* Fix playbooks

* Format playbooks and fix Gem Alert Classifier

* Fix blank space in Gem Layout

* Fix incident fields

* Fix cr (#8)

* Fix double line

* Update playbooks (#9)

Co-authored-by: Lior Maman <155369912+liormgem@users.noreply.github.com>
pal-xmco pushed a commit to pal-xmco/content that referenced this pull request Jun 19, 2024
* Gem Security pack Commit

Pack includes:

1 Automation
3 Classifiers
16 Incident Fields
1 Incident Type
1 Integration
1 Layout
3 Playbooks
1 Pre-process Rule

* Ci fix (demisto#5)

* Fix post commit validation issues

* Fix tests coverage

* Fix indent (demisto#6)

* Cr fix (demisto#7)

* Fix playbooks

* Format playbooks and fix Gem Alert Classifier

* Fix blank space in Gem Layout

* Fix incident fields

* Fix cr (#8)

* Fix double line

* Update playbooks (demisto#9)

Co-authored-by: Lior Maman <155369912+liormgem@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Contribution Form Filled Whether contribution form filled or not. Contribution Thank you! Contributions are always welcome! docs-approved External PR Partner Partner-Approved pending-demo Demo pending Security Review
Projects
None yet
Development

Successfully merging this pull request may close these issues.

9 participants