Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

New version of threatintel/recordedfuture dataset #30030

Merged
merged 4 commits into from
Feb 1, 2022
Merged

New version of threatintel/recordedfuture dataset #30030

merged 4 commits into from
Feb 1, 2022

Conversation

adriansr
Copy link
Contributor

@adriansr adriansr commented Jan 26, 2022 •
edited
Loading

What does this PR do?

This is a new implementation of the Recorded Future integration in the threatintel module. Uses the risklist API endpoints to fetch threat indicators in CSV format, while also supports ingesting from a custom URL (for Fusion Files) and from already-downloaded CSV files.

Why is it important?

The previous implementation of the Recorded Future API was unsupported as it used the wrong API to download indicators from Recorded Future.

Checklist

  • My code follows the style guidelines of this project
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • I have made corresponding change to the default configuration files
  • I have added tests that prove my fix is effective or that my feature works
  • I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Pending tasks

  • Update the dashboard, most fields are missing from this new API. This will be done in a separate PR.

Screenshots

image

@adriansr
New version of threatintel/recordedfuture dataset
592324e 
This is a new implementation of the Recorded Future integration in the
threatintel module. Uses the `risklist` API endpoints to fetch threat
indicators in CSV format, while also supports ingesting from a custom
URL (for Fusion Files) and from CSV files.

The previous implementation was unsupported as it used the wrong API to
download indicators from Recorded Future.
@elasticmachine
Copy link
Collaborator

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

@botelastic botelastic bot added needs_team Indicates that the issue/PR needs a Team:* label and removed needs_team Indicates that the issue/PR needs a Team:* label labels Jan 26, 2022
@mergify
Copy link
Contributor

mergify bot commented Jan 26, 2022

This pull request does not have a backport label. Could you fix it @adriansr? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

  • backport-v./d./d./d is the label to automatically backport to the 7./d branch. /d is the digit

NOTE: backport-skip has been added to this pull request.

@mergify mergify bot added the backport-skip Skip notification from the automated backport with mergify label Jan 26, 2022
@elasticmachine
Copy link
Collaborator

elasticmachine commented Jan 26, 2022 •
edited by jenkins-beats-ci bot
Loading

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS
Pipeline View Test View Changes Artifacts preview preview

Expand to view the summary

Build stats

  • Reason: null

  • Start Time: 2022-01-27T18:22:02.148+0000

  • Duration: 105 min 39 sec

  • Commit: a17a794

Test stats 🧪

Test Results
Failed 0
Passed 9636
Skipped 1282
Total 10918

💚 Flaky test report

Tests succeeded.

🤖 GitHub comments

To re-run your PR in the CI, just comment with:

  • /test : Re-trigger the build.

  • /package : Generate the packages and run the E2E tests.

  • /beats-tester : Run the installation tests with beats-tester.

  • run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

@adriansr adriansr added the backport-v8.1.0 Automated backport with mergify label Jan 26, 2022
@adriansr adriansr requested review from marius-dr, P1llus and a team and removed request for marius-dr January 26, 2022 17:01
@mergify mergify bot removed the backport-skip Skip notification from the automated backport with mergify label Jan 26, 2022
efd6
efd6 reviewed Jan 27, 2022
View reviewed changes
...ck/filebeat/module/threatintel/_meta/kibana/7/lens/b0837690-df52-11eb-8f2b-753caedf727d.json Outdated
@@ -28,7 +28,7 @@
"size": 10
},
"scale": "ordinal",
"sourceField": "recordedfuture.risk.criticalityLabel"
"sourceField": "recordedfuture.criticality_label"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this field defined? I can't find it referenced anywhere else. Should there be a rename/copy from recordedfuture.evidence_details[:].CriticalityLabel?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It was a mistake to push that change. Turns out the criticality_label field is not reliable from this new API so the code that populated it was removed. We will address the dashboard changes in a different PR as it'll be necessary to remove a few visualisations (like this one) because the data is not available anymore.

@adriansr
Copy link
Contributor Author

/test

@adriansr
Revert dashboard change
a17a794 
Dashboard will be fixed in a separate PR
@adriansr adriansr requested a review from efd6 January 27, 2022 18:22
efd6
efd6 approved these changes Feb 1, 2022
View reviewed changes
Copy link
Contributor

@efd6 efd6 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM but please wait for @P1llus

@adriansr adriansr merged commit fbcece6 into elastic:master Feb 1, 2022
@adriansr adriansr deleted the rf_new_api branch February 1, 2022 09:19
adriansr added a commit to adriansr/beats that referenced this pull request Feb 3, 2022
@adriansr @peasead
Dashboard for new Recorded Future integration
806e8df 
Relates elastic#30030

Co-authored-by: Andrew Pease <7442091+peasead@users.noreply.github.com>
@adriansr adriansr mentioned this pull request Feb 3, 2022
Merged
adriansr added a commit that referenced this pull request Feb 4, 2022
@adriansr @peasead
Dashboard for new Recorded Future integration (#30199)
b8203a3 
Relates #30030

Co-authored-by: Andrew Pease <7442091+peasead@users.noreply.github.com>
mergify bot pushed a commit that referenced this pull request Feb 10, 2022
@adriansr
Dashboard for new Recorded Future integration (#30199)
4100c56 
Relates #30030

Co-authored-by: Andrew Pease <7442091+peasead@users.noreply.github.com>
(cherry picked from commit b8203a3)
adriansr added a commit that referenced this pull request Feb 10, 2022
@mergify @adriansr
Dashboard for new Recorded Future integration (#30199) (#30330)
b9fc658 
Relates #30030

Co-authored-by: Andrew Pease <7442091+peasead@users.noreply.github.com>
(cherry picked from commit b8203a3)

Co-authored-by: Adrian Serrano <adrisr83@gmail.com>
@adriansr adriansr mentioned this pull request Mar 1, 2022
Merged
7 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@P1llus P1llus P1llus approved these changes

@efd6 efd6 efd6 approved these changes

Assignees
No one assigned
Labels
8.1-candidate backport-v8.1.0 Automated backport with mergify enhancement Filebeat Filebeat review
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@adriansr @elasticmachine @P1llus @efd6