Develop #175
Develop #175
Conversation
Signed-off-by: NxPKG <iconmamundentist@gmail.com>
Reviewer's Guide by SourceryThis PR introduces several major changes and improvements to the ThreatMatrix project, including new features for plugin management, improved documentation links, enhanced error handling, and various bug fixes. The changes span across both frontend and backend components, with significant updates to the analyzer, pivot, and playbook functionalities. Class diagram for VirusTotalv3BaseMixin and VirusTotalv3AnalyzerMixinclassDiagram
class VirusTotalv3BaseMixin {
+str url
+str url_sub_path
+str _api_key_name
+headers() dict
+config(runtime_configuration: Dict)
+_perform_get_request(uri: str, ignore_404: bool) Dict
+_perform_post_request(uri: str, ignore_404: bool)
+_perform_request(uri: str, method: str, ignore_404: bool) Dict
+_get_relationship_for_classification(obs_clfn: str, iocs: bool) List
+_get_requests_params_and_uri(obs_clfn: str, observable_name: str, iocs: bool) Tuple
+_fetch_behaviour_summary(observable_name: str) Dict
+_fetch_sigma_analyses(observable_name: str) Dict
+_vt_download_file(file_hash: str) bytes
+_vt_intelligence_search(query: str, limit: int, order_by: str) Dict
+_vt_get_iocs_from_file(sample_hash: str) Dict
}
class VirusTotalv3AnalyzerMixin {
+int max_tries
+int poll_distance
+int rescan_max_tries
+int rescan_poll_distance
+bool include_behaviour_summary
+bool include_sigma_analyses
+bool force_active_scan_if_old
+int days_to_say_that_a_scan_is_old
+list relationships_to_request
+int relationships_elements
+_get_relationship_limit(relationship: str) int
+_vt_get_relationships(observable_name: str, relationships_requested: list, uri: str, result: dict)
+_get_url_prefix_postfix(result: Dict) Tuple
+_vt_scan_file(md5: str, rescan_instead: bool) Dict
+_vt_poll_for_report(observable_name: str, params: Dict, uri: str, obs_clfn: str) Dict
+_vt_include_behaviour_summary(result: Dict, observable_name: str) Dict
+_vt_include_sigma_analyses(result: Dict, observable_name: str) Dict
+_vt_get_report(obs_clfn: str, observable_name: str) Dict
}
VirusTotalv3AnalyzerMixin --|> VirusTotalv3BaseMixin
Class diagram for ScanForm componentclassDiagram
class ScanForm {
+DangerErrorMessage(fieldName)
+selectObservableType(value)
+updateAdvancedConfig(tags, oldClassification, newClassification)
}
ScanForm o-- AnalyzersMultiSelectDropdownInput
ScanForm o-- ConnectorsMultiSelectDropdownInput
ScanForm o-- PlaybookMultiSelectDropdownInput
ScanForm o-- TLPSelectInput
ScanForm o-- ScanConfigSelectInput
ScanForm o-- usePluginConfigurationStore
ScanForm o-- useGuideContext
ScanForm o-- createJob
ScanForm o-- sanitizeObservable
ScanForm o-- SpinnerIcon
ScanForm o-- JobTypes
ScanForm o-- ObservableClassifications
ScanForm o-- ScanTypes
ScanForm o-- Loader
ScanForm o-- MultiSelectDropdownInput
ScanForm o-- ReactSelect
ScanForm o-- TLPTag
ScanForm o-- TLPColors
ScanForm o-- TLPDescriptions
ScanForm o-- MdInfoOutline
ScanForm o-- UncontrolledTooltip
ScanForm o-- FormText
ScanForm o-- FormGroup
ScanForm o-- Label
ScanForm o-- Col
ScanForm o-- Input
ScanForm o-- Field
ScanForm o-- Collapse
ScanForm o-- Button
ScanForm o-- Link
ScanForm o-- JobTag
ScanForm o-- markdownToHtml
File-Level Changes
Tips and commandsInteracting with Sourcery
Customizing Your ExperienceAccess your dashboard to:
Getting Help
|
|
| GitGuardian id | GitGuardian status | Secret | Commit | Filename | |
|---|---|---|---|---|---|
| 13180230 | Triggered | Username Password | 6ca0a96 | tests/auth/test_auth.py | View secret |
🛠 Guidelines to remediate hardcoded secrets
- Understand the implications of revoking this secret by investigating where it is used in your code.
- Replace and store your secret safely. Learn here the best practices.
- Revoke and rotate this secret.
- If possible, rewrite git history. Rewriting git history is not a trivial act. You might completely break other contributing developers' workflow and you risk accidentally deleting legitimate data.
To avoid such incidents in the future consider
- following these best practices for managing and storing secrets including API keys and other credentials
- install secret detection on pre-commit to catch secret before it leaves your machine and ease remediation.
🦉 GitGuardian detects secrets in your source code to help developers and security teams secure the modern development process. You are seeing this because you or someone else with access to this repository has authorized GitGuardian to scan your pull request.
|
Important Review skippedMore than 25% of the files skipped due to max files limit. The review is being skipped to prevent a low-quality review. 152 files out of 292 files are above the max files limit of 75. Please upgrade to Pro plan to get higher limits. You can disable this status message by setting the Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media? 🪧 TipsChatThere are 3 ways to chat with CodeRabbit:
Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments. CodeRabbit Commands (Invoked using PR comments)
Other keywords and placeholders
CodeRabbit Configuration File (
|
|
Here's the code health analysis summary for commits Analysis Summary
|
![sourcery-ai[bot]](https://avatars.githubusercontent.com/in/48477?s=60&v=4){kind=small}
There was a problem hiding this comment.
Hey @NxPKG - I've reviewed your changes - here's some feedback:
Overall Comments:
- Please expand the PR description to better explain the motivation and impact of these changes, particularly around the new elastic search integration and plugin management features.
- Consider adding more documentation around the elastic search functionality given its complexity and importance for the system.
Here's what I looked at during the review
- 🟡 General issues: 5 issues found
- 🟢 Security: all looks good
- 🟡 Testing: 8 issues found
- 🟡 Complexity: 4 issues found
- 🟢 Documentation: all looks good
Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.
| logger.debug(f"text: {response.text}") | ||
| result = response.json() | ||
| # https://developers.virustotal.com/reference/errors | ||
| error = result.get("error", {}) |
There was a problem hiding this comment.
suggestion: Consider enhancing error handling to provide more context about the specific VT API error
The error object from VT API contains additional fields like code and message that could be included in the error message to help with debugging.
| error = result.get("error", {}) | |
| error = result.get("error", {}) | |
| error_code = error.get("code", "unknown") | |
| error_message = error.get("message", "no message") | |
| error_context = f"VT API Error - Code: {error_code}, Message: {error_message}" |
| if "IOC.json" in report and isinstance(report["IOC.json"], List): | ||
| for ioc in report["IOC.json"]: | ||
| try: | ||
| if "url" in ioc["type"].lower(): |
There was a problem hiding this comment.
suggestion: Add explicit error handling for malformed IOC data structure
The IOC parsing could fail if 'type' is missing. Consider adding validation of the IOC structure before accessing fields.
| if "url" in ioc["type"].lower(): | |
| if "type" in ioc and "url" in ioc["type"].lower(): |
| ) | ||
| pdns_reports.append(pdns_report) | ||
| if "rrdata" in report.keys(): | ||
| pdns_report = PDNSReport( |
There was a problem hiding this comment.
suggestion (bug_risk): Consider creating PDNSReport objects only after validating all required fields
Validate all required fields before object creation to avoid creating partial objects that may need to be discarded.
| pdns_report = PDNSReport( | |
| required_fields = ["rrdata", "time_last"] | |
| if all(field in report for field in required_fields): | |
| pdns_report = PDNSReport( |
| const formik = useFormik({ | ||
| initialValues: { | ||
| name: pivotConfig?.name || "", | ||
| description: pivotConfig?.description || "<generated automatically>", | ||
| python_module: | ||
| { | ||
| value: pivotConfig?.python_module, | ||
| label: | ||
| pythonModuleOptions.find( | ||
| (element) => element.value === pivotConfig?.python_module, |
There was a problem hiding this comment.
issue: Consider handling edge cases in form initialization
The form initialization doesn't handle cases where pivotConfig contains unexpected or malformed data. Consider adding validation and fallback values.
| errors.name = "This field is required."; | ||
| } else if (values.name.length < minLength) { | ||
| errors.name = `This field must be at least ${minLength} characters long`; | ||
| } |
There was a problem hiding this comment.
suggestion (bug_risk): Add type validation for payload construction
The payload construction assumes all required fields will be present and correctly typed. Consider adding type validation to prevent runtime errors.
const onSubmit = React.useCallback(
async (values: FormValues, formik) => {
if (!values?.name || !values?.description) {
throw new Error('Required fields are missing');
}
const payloadData = {
name: values.name,
description: values.description,| : "Edit playbook config" | ||
| } | ||
| onClick={() => { | ||
| if (!pluginsLoading) setShowModal(true); |
There was a problem hiding this comment.
suggestion (code-quality): Use block braces for ifs, whiles, etc. (use-braces)
| if (!pluginsLoading) setShowModal(true); | |
| if (!pluginsLoading) { |
Explanation
It is recommended to always use braces and create explicit statement blocks.Using the allowed syntax to just write a single statement can lead to very confusing
situations, especially where subsequently a developer might add another statement
while forgetting to add the braces (meaning that this wouldn't be included in the condition).
| if (validationValue.classification !== "generic") | ||
| observables[validationValue.classification].push( | ||
| validationValue.observable, | ||
| ); |
There was a problem hiding this comment.
suggestion (code-quality): Use block braces for ifs, whiles, etc. (use-braces)
| if (validationValue.classification !== "generic") | |
| observables[validationValue.classification].push( | |
| validationValue.observable, | |
| ); | |
| if (validationValue.classification !== "generic") { | |
| observables[validationValue.classification].push( | |
| validationValue.observable, | |
| ); | |
| } | |
Explanation
It is recommended to always use braces and create explicit statement blocks.Using the allowed syntax to just write a single statement can lead to very confusing
situations, especially where subsequently a developer might add another statement
while forgetting to add the braces (meaning that this wouldn't be included in the condition).
| // remove domain if stringEnd is a number | ||
| if (!Number.isNaN(parseInt(stringEnd, 10))) return null; | ||
| if (!Number.isNaN(parseInt(stringEnd, 10))) return defaultValue; |
There was a problem hiding this comment.
suggestion (code-quality): Use block braces for ifs, whiles, etc. (use-braces)
| if (!Number.isNaN(parseInt(stringEnd, 10))) return defaultValue; | |
| if (!Number.isNaN(parseInt(stringEnd, 10))) { |
Explanation
It is recommended to always use braces and create explicit statement blocks.Using the allowed syntax to just write a single statement can lead to very confusing
situations, especially where subsequently a developer might add another statement
while forgetting to add the braces (meaning that this wouldn't be included in the condition).
| if (!Object.values(hashTypesLength).includes(sanitizedString.length)) | ||
| return null; | ||
| return defaultValue; |
There was a problem hiding this comment.
suggestion (code-quality): Use block braces for ifs, whiles, etc. (use-braces)
| if (!Object.values(hashTypesLength).includes(sanitizedString.length)) | |
| return null; | |
| return defaultValue; | |
| if (!Object.values(hashTypesLength).includes(sanitizedString.length)) { | |
| return defaultValue; | |
| } | |
Explanation
It is recommended to always use braces and create explicit statement blocks.Using the allowed syntax to just write a single statement can lead to very confusing
situations, especially where subsequently a developer might add another statement
while forgetting to add the braces (meaning that this wouldn't be included in the condition).
| if (["phone", "date"].includes(stringClassification)) | ||
| return { | ||
| classification: ObservableClassifications.GENERIC, | ||
| observable: sanitizedString, | ||
| }; |
There was a problem hiding this comment.
suggestion (code-quality): Use block braces for ifs, whiles, etc. (use-braces)
| if (["phone", "date"].includes(stringClassification)) | |
| return { | |
| classification: ObservableClassifications.GENERIC, | |
| observable: sanitizedString, | |
| }; | |
| if (["phone", "date"].includes(stringClassification)) { | |
| return { | |
| classification: ObservableClassifications.GENERIC, | |
| observable: sanitizedString, | |
| }; | |
| } | |
Explanation
It is recommended to always use braces and create explicit statement blocks.Using the allowed syntax to just write a single statement can lead to very confusing
situations, especially where subsequently a developer might add another statement
while forgetting to add the braces (meaning that this wouldn't be included in the condition).
PR Reviewer Guide 🔍Here are some key observations to aid the review process:
|
PR Code Suggestions ✨Explore these optional code suggestions:
|
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
|
|
||
| except Exception as e: | ||
| logging.exception( | ||
| f"Error: {e}. Secret: {secret_name}" |
Check failure
Code scanning / CodeQL
Clear-text logging of sensitive information High
Show autofix suggestion
Hide autofix suggestion
Copilot Autofix AI about 1 month ago
To fix the problem, we should avoid logging sensitive information directly. Instead, we can log a generic message that does not include the sensitive secret_name. This way, we still capture the occurrence of an error without exposing sensitive data.
- Replace the logging statements that include
secret_namewith more generic messages. - Ensure that the functionality of error logging is maintained without exposing sensitive information.
| @@ -85,3 +85,3 @@ | ||
| logging.error( | ||
| f"Failed retrieving of secret {secret_name}. Error: {e}." | ||
| f"Failed retrieving of secret. Error: {e}." | ||
| ) # lgtm [py/clear-text-logging-sensitive-data] | ||
| @@ -89,3 +89,3 @@ | ||
| logging.error( | ||
| f"Error: {e}. Secret: {secret_name}" | ||
| f"Error: {e}. Failed to retrieve secret." | ||
| ) # lgtm [py/clear-text-logging-sensitive-data] | ||
| @@ -93,3 +93,3 @@ | ||
| logging.exception( | ||
| f"Error: {e}. Secret: {secret_name}" | ||
| f"Error: {e}. Failed to retrieve secret." | ||
| ) # lgtm [py/clear-text-logging-sensitive-data] |
Signed-off-by: NxPKG <iconmamundentist@gmail.com>
Signed-off-by: NxPKG <iconmamundentist@gmail.com>
Signed-off-by: NxPKG <iconmamundentist@gmail.com>
Signed-off-by: NxPKG <iconmamundentist@gmail.com>
Signed-off-by: NxPKG <iconmamundentist@gmail.com>
Signed-off-by: NxPKG <iconmamundentist@gmail.com>
Signed-off-by: NxPKG <iconmamundentist@gmail.com>
Signed-off-by: NxPKG <iconmamundentist@gmail.com>
Signed-off-by: NxPKG <iconmamundentist@gmail.com>
Signed-off-by: NxPKG <iconmamundentist@gmail.com>
Signed-off-by: NxPKG <iconmamundentist@gmail.com>
Signed-off-by: NxPKG <iconmamundentist@gmail.com>
Signed-off-by: NxPKG <iconmamundentist@gmail.com>
CI Failure Feedback 🧐(Checks updated until commit 66983d2)
✨ CI feedback usage guide:The CI feedback tool (
In addition to being automatically triggered, the tool can also be invoked manually by commenting on a PR: where Configuration options
See more information about the |
* Update compose-tests.yml Signed-off-by: NxPKG <iconmamundentist@gmail.com> * Update compose.yml Signed-off-by: NxPKG <iconmamundentist@gmail.com> * Update compose-tests.yml Signed-off-by: NxPKG <iconmamundentist@gmail.com> * Update compose.yml Signed-off-by: NxPKG <iconmamundentist@gmail.com> * Update compose-tests.yml Signed-off-by: NxPKG <iconmamundentist@gmail.com> * Update compose.yml Signed-off-by: NxPKG <iconmamundentist@gmail.com> * Update compose.yml Signed-off-by: NxPKG <iconmamundentist@gmail.com> * Update compose-tests.yml Signed-off-by: NxPKG <iconmamundentist@gmail.com> * Update compose.yml Signed-off-by: NxPKG <iconmamundentist@gmail.com> --------- Signed-off-by: NxPKG <iconmamundentist@gmail.com>
* fixed dashboard backend unittest * black format * fix isort * prettier:write fix * Update test_auth.py Signed-off-by: gitworkflows <118260833+gitworkflows@users.noreply.github.com> --------- Signed-off-by: gitworkflows <118260833+gitworkflows@users.noreply.github.com>
| upper_case=True, | ||
| lower_case=True, | ||
| ) | ||
| logger.info(f"Generated fake password input {self.FAKE_PASSWORD_INPUT}") |
Check failure
Code scanning / CodeQL
Clear-text logging of sensitive information High
Show autofix suggestion
Hide autofix suggestion
Copilot Autofix AI 8 days ago
To fix the problem, we should avoid logging sensitive information such as passwords, even if they are fake. Instead, we can log a generic message indicating that the fake password input has been generated without including the actual value. This approach maintains the functionality of the code while ensuring that sensitive information is not exposed in the logs.
- Update the logging statements to avoid including sensitive information.
- Specifically, change the logging statement on line 95 to a more generic message.
- Ensure that similar changes are made to other logging statements that include sensitive information.
| @@ -82,7 +82,5 @@ | ||
| } | ||
| logger.info( | ||
| f"Generated name text input mapping {self._name_text_input_mapping}" | ||
| ) | ||
| logger.info("Generated name text input mapping.") | ||
| self.FAKE_EMAIL_INPUT: str = fake.email() | ||
| logger.info(f"Generated fake email input {self.FAKE_EMAIL_INPUT}") | ||
| logger.info("Generated fake email input.") | ||
| self.FAKE_PASSWORD_INPUT: str = fake.password( | ||
| @@ -94,5 +92,5 @@ | ||
| ) | ||
| logger.info(f"Generated fake password input {self.FAKE_PASSWORD_INPUT}") | ||
| logger.info("Generated fake password input.") | ||
| self.FAKE_TEL_INPUT: str = fake.phone_number() | ||
| logger.info(f"Generated fake tel input {self.FAKE_TEL_INPUT}") | ||
| logger.info("Generated fake tel input.") | ||
|
|
| ) | ||
|
|
||
| logger.info( | ||
| f"Job #{self.job_id}: Sending value {value_to_set} for {input_name=}" |
Check failure
Code scanning / CodeQL
Clear-text logging of sensitive information High
Show autofix suggestion
Hide autofix suggestion
Copilot Autofix AI 8 days ago
To fix the problem, we should avoid logging sensitive information such as passwords, emails, and telephone numbers. Instead, we can log a generic message indicating that a value has been set without including the actual value. This way, we maintain the logging functionality for debugging purposes without exposing sensitive information.
- Replace the logging statement on line 169 to avoid logging the actual value of
value_to_set. - Replace the logging statement on line 176 to avoid logging the actual parameters being sent.
| @@ -168,3 +168,3 @@ | ||
| logger.info( | ||
| f"Job #{self.job_id}: Sending value {value_to_set} for {input_name=}" | ||
| f"Job #{self.job_id}: Sending value for {input_name=}" | ||
| ) | ||
| @@ -175,3 +175,3 @@ | ||
| params, dest_url = self.compile_form_field(form) | ||
| logger.info(f"Job #{self.job_id}: Sending {params=} to submit url {dest_url}") | ||
| logger.info(f"Job #{self.job_id}: Submitting form to url {dest_url}") | ||
| return requests.post( |
⛔ Snyk checks have failed. 7 issues have been found so far.
⛔ security/snyk check is complete. 7 issues have been found. (View Details) |
User description
(Please add to the PR name the issue/s that this PR would close if merged by using a Github keyword. Example:
<feature name>. Closes #999. If your PR is made by a single commit, please add that clause in the commit too. This is all required to automate the closure of related issues.)Description
Please include a summary of the change and link to the related issue.
Type of change
Please delete options that are not relevant.
Checklist
developdumpplugincommand and added it in the project as a data migration. ("How to share a plugin with the community")test_files.zipand you added the default tests for that mimetype in test_classes.py.FREE_TO_USE_ANALYZERSplaybook by following this guide.urlthat contains this information. This is required for Health Checks._monkeypatch()was used in its class to apply the necessary decorators.MockUpResponseof the_monkeypatch()method. This serves us to provide a valid sample for testing.Black,Flake,Isort) gave 0 errors. If you have correctly installed pre-commit, it does these checks and adjustments on your behalf.testsfolder). All the tests (new and old ones) gave 0 errors.DeepSource,Django Doctorsor other third-party linters have triggered any alerts during the CI checks, I have solved those alerts.Important Rules
PR Type
Enhancement, Tests, Configuration changes, Documentation, Formatting, Error handling, Security, Dependencies
Description
Changes walkthrough 📝
55 files
mixins.py
Introduced VirusTotal API interaction mixin classes.api_app/mixins.py
abc,base64,time,datetime,timedelta, and typingutilities.
VirusTotalv3BaseMixinclass with methods for VirusTotal APIinteractions.
_perform_requestmethod for handling HTTP requests.virus_total.py
Implemented VirusTotal ingestor for sample and IOC extraction.api_app/ingestors_manager/ingestors/virus_total.py
VirusTotalclass inheriting fromIngestorandVirusTotalv3BaseMixin.runmethod to execute queries and download samples orIOCs.
_monkeypatchmethod for mocking HTTP requests in tests.doc_info.py
Enhanced document analysis with URL and CVE extraction.api_app/analyzers_manager/file_analyzers/doc_info.py
docxpyandoletoolsmodules.runmethod to extract URLs and CVEs from documents.from documents.
vt3_intelligence_search.py
Refactored VirusTotal intelligence search analyzer.api_app/analyzers_manager/observable_analyzers/vt/vt3_intelligence_search.py
VirusTotalv3Intelligenceclass to useVirusTotalv3AnalyzerMixin.runmethod to utilize_vt_intelligence_search.views.py
Added job rescan and plugin report query functionalities.api_app/views.py
rescanaction toJobViewSetfor rescanning jobs.plugin_report_queriesview for querying plugin reports.basic_observable_analyzer.py
Implemented basic observable analyzer with HTTP request handling.api_app/analyzers_manager/observable_analyzers/basic_observable_analyzer.py
BasicObservableAnalyzerclass for analyzing observables.certificates.
_monkeypatchmethod for mocking HTTP requests in tests.tasks.py
Added task to send plugin reports to Elasticsearch.threat_matrix/tasks.py
send_plugin_report_to_elastictask to send plugin reports toElasticsearch.
handle exceptions.
serializers.py
Enhanced PivotConfigSerializer with additional fields and logic.api_app/pivots_manager/serializers.py
related_analyzer_configs,related_connector_configs,and
plugin_config.PivotConfigSerializer.strings_info.py
Enhanced StringsInfo analyzer with URL extraction.api_app/analyzers_manager/file_analyzers/strings_info.py
StringsInfoanalyzer.updatemethod and disabled mockup connections.boxjs_scan.py
Enhanced BoxJS analyzer with URL extraction.api_app/analyzers_manager/file_analyzers/boxjs_scan.py
BoxJSanalyzer.updatemethod and disabled mockup connections.serializers.py
Enhanced PlaybookConfigSerializer with visualizers and editability.api_app/playbooks_manager/serializers.py
PlaybookConfigSerializerto includevisualizersandis_editablefields.nerd.py
Added NERD observable analyzer with API integration.api_app/analyzers_manager/observable_analyzers/nerd.py
NERDobservable analyzer class.elastic.py
Added serializers for Elasticsearch request and response handling.api_app/serializers/elastic.py
ElasticRequestdataclass and corresponding serializer.plugin.py
Enhanced PluginConfigSerializer with PivotConfig support.api_app/serializers/plugin.py
PivotConfigtoPluginConfigSerializer.PythonConfigSerializerto handle optional parameters.signals.py
Added signal handlers for cache refresh and logging.api_app/signals.py
PythonConfigsave anddelete.
LogEntrypost-save signal.models.py
Enhanced models with report value retrieval and singleton pattern.api_app/models.py
get_valuemethod toAbstractReportfor retrieving nested reportvalues.
SingletonModelandLastElasticReportUpdatefor singletonpattern.
__init__.py
Added utility methods for job creation and sample analysis.tests/init.py
_analyze_samplemethod for sample analysis.dshield.py
Added DShield observable analyzer with API integration.api_app/analyzers_manager/observable_analyzers/dshield.py
DShieldobservable analyzer class.views.py
Enhanced PivotConfigViewSet with CRUD operations and permissions.api_app/pivots_manager/views.py
PivotConfigViewSet.admin.py
Registered LogEntry model in admin with read-only access.api_app/admin.py
LogEntrymodel in admin with read-only permissions.LogEntry.artifacts.py
Simplified artifact analysis command arguments.api_app/analyzers_manager/file_analyzers/artifacts.py
runmethod by removing conflicting options.analyzer_extractor.py
Improved Robtex report extraction with key validation.api_app/visualizers_manager/visualizers/passive_dns/analyzer_extractor.py
rrdatakey in Robtex reports before processing.PDNSReport.compare.py
Simplified value retrieval in Compare pivot.api_app/pivots_manager/pivots/compare.py
_get_valuemethod and usedget_valuefromAbstractReport.lnk_info.py
Added LnkInfo analyzer for LNK file URL extraction.api_app/analyzers_manager/file_analyzers/lnk_info.py
LnkInfoanalyzer class for extracting URLs from LNK files.elastic_templates.py
Added command for managing Elasticsearch index templates.api_app/management/commands/elastic_templates.py
templates.
pdf_info.py
Enhanced PDFInfo analyzer with URI extraction.api_app/analyzers_manager/file_analyzers/pdf_info.py
PDFInfoanalyzer.androguard.py
Added AndroguardAnalyzer for APK file analysis.api_app/analyzers_manager/file_analyzers/androguard.py
AndroguardAnalyzerclass for analyzing APK files.views.py
Enhanced AnalyzerConfigViewSet with CRUD operations and permissions.api_app/analyzers_manager/views.py
AnalyzerConfigViewSet.documents.py
Updated JobDocument with logging and field adjustments.api_app/documents.py
JobDocumentfield definitions.constants.py
Enhanced constants with updated URL regex and HTTP methods.api_app/analyzers_manager/constants.py
ObservableTypes.HTTPMethodsclass for HTTP method choices.classes.py
Enhanced health check logic for authentication-protected endpoints.api_app/classes.py
endpoints.
onenote.py
Enhanced OneNoteInfo analyzer with base64 encoding for files.api_app/analyzers_manager/file_analyzers/onenote.py
OneNoteInfoanalyzer.views.py
Added debug logging and parent job handling in playbook views.api_app/playbooks_manager/views.py
analyze_multiple_observablesand
analyze_multiple_files.app.py
Enhanced error handling in intercept_box_js_result function.integrations/malware_tools_analyzers/app.py
intercept_box_js_resultfunction.serializers.py
Refined serializers for ingestor configuration and reports.api_app/ingestors_manager/serializers.py
IngestorConfigSerializerForMigrationto exclude certainfields.
IngestorReportBISerializer.download_file_from_uri.py
Modified stored_base64 handling in DownloadFileFromUri analyzer.api_app/analyzers_manager/observable_analyzers/download_file_from_uri.py
stored_base64field to a list inDownloadFileFromUrianalyzer.any_compare.py
Refactored field existence check in AnyCompare pivot.api_app/pivots_manager/pivots/any_compare.py
should_runmethod to iterate over related reports.load_file.py
Enhanced base64 decoding logic in LoadFile pivot.api_app/pivots_manager/pivots/load_file.py
get_value_to_pivot_tomethod to handle lists of values.queryset.py
Updated Elasticsearch client references in queryset logic.api_app/queryset.py
ELASTICSEARCH_BI_CLIENT.Elasticsearch.
job.py
Enhanced JobResponseSerializer with sample field and fixed MIME typecheck.api_app/serializers/job.py
is_samplefield toJobResponseSerializer.vt3_get.py
Added update method to VirusTotalv3 analyzer.api_app/analyzers_manager/observable_analyzers/vt/vt3_get.py
updatemethod toVirusTotalv3analyzer.exceptions.py
Added NotImplementedException for unsupported services.api_app/exceptions.py
NotImplementedExceptionclass for unsupported services.APIExceptionto provide custom status and detail.dumpplugin.py
Enhanced plugin dumping with reverse relationship handling.api_app/management/commands/dumpplugin.py
signals.py
Added signal handler for playbooks_choice changes in PivotConfig.api_app/pivots_manager/signals.py
playbooks_choiceofPivotConfig.models.py
Add LNK MIME type to MimeTypes classapi_app/analyzers_manager/models.py
LNKto theMimeTypesclass.droidlysis.py
Increase max_tries for HTTP request polling in DroidLysisapi_app/analyzers_manager/file_analyzers/droidlysis.py
max_triesfor HTTP request polling from 10 to 30.permissions.py
Add PivotActionsPermission class for pivot actionsapi_app/pivots_manager/permissions.py
PivotActionsPermissionclass to handle permissions forpivot actions.
classes.py
Add logging for start and finish of ingestorapi_app/ingestors_manager/classes.py
before_runandafter_runmethods to logstart and finish of ingestor.
permissions.py
Add isPluginActionsPermission class for plugin actionsapi_app/permissions.py
isPluginActionsPermissionclass to handle permissions forplugin actions.
urls.py
Add plugin_report_queries path to urlpatternsapi_app/urls.py
plugin_report_queriesto the urlpatterns.observables.js
Refactor observableValidators to use generic classificationfrontend/src/utils/observables.js
observableValidatorsto return a default value withgenericclassification.
miscConst.js
Add HTTPMethods constant to miscConstfrontend/src/constants/miscConst.js
HTTPMethodsconstant with common HTTP methods.pluginConst.js
Add AllPluginSupportedTypes constant to pluginConstfrontend/src/constants/pluginConst.js
AllPluginSupportedTypesconstant with supported plugin types.utils.js
Add is_sample property to job node in addJobNodefrontend/src/components/investigations/flow/utils.js
is_sampleproperty to job node inaddJobNodefunction.ScanForm.jsx
Refactor ScanForm component with new form utilitiesfrontend/src/components/scan/ScanForm.jsx
ScanFormcomponent to use new form components andutilities.
17 files
test_tasks.py
Added tests for Elasticsearch plugin report integration.tests/threat_matrix/test_tasks.py
SendElasticTestCasefor testing Elasticsearchintegration.
test_api.py
Added tests for job rescan functionality and permissions.tests/api_app/test_api.py
playbooks.
test_mixins.py
Added tests for VirusTotal mixin classes.tests/api_app/test_mixins.py
VirusTotalv3BaseMixinandVirusTotalv3AnalyzerMixin.test_views.py
Added tests for analyzer configuration management.tests/api_app/analyzers_manager/test_views.py
configurations.
test_views.py
Added tests for pivot configuration CRUD operations.tests/api_app/pivots_manager/test_views.py
AnalyzerConfig,Parameter,PluginConfig,Membership,and
Organization.test_create,test_update, andtest_deletemethods fortesting pivot configurations.
configurations with various conditions.
test_doc_info.py
Added tests for DocInfo analyzer functionality.tests/api_app/analyzers_manager/file_analyzers/test_doc_info.py
DocInfoanalyzer.test_classes.py
Added test job creation and health check logic.tests/api_app/analyzers_manager/test_classes.py
.lnkfiles.test_strings_info.py
Added test for URL extraction in StringsInfo analyzer.tests/api_app/analyzers_manager/file_analyzers/test_strings_info.py
StringsInfoanalyzer.test_serializers.py
Added tests for PivotConfig creation with plugin configuration.tests/api_app/pivots_manager/test_serializers.py
PivotConfigwith and without pluginconfiguration.
test_iocextract.py
Added test for IOC extraction in IocExtract analyzer.tests/api_app/analyzers_manager/file_analyzers/test_iocextract.py
IocExtractanalyzer.test_boxjs.py
Added test for URL extraction in BoxJS analyzer.tests/api_app/analyzers_manager/file_analyzers/test_boxjs.py
BoxJSanalyzer.test_onenote_info.py
Added test for base64 extraction in OneNoteInfo analyzer.tests/api_app/analyzers_manager/file_analyzers/test_onenote_info.py
OneNoteInfoanalyzer.files.
test_pdf_info.py
Added test for URL extraction in PDFInfo analyzer.tests/api_app/analyzers_manager/file_analyzers/test_pdf_info.py
PDFInfoanalyzer.test_lnk_info.py
Added test for URL extraction in LnkInfo analyzer.tests/api_app/analyzers_manager/file_analyzers/test_lnk_info.py
LnkInfoanalyzer.observables.test.js
Update observable validators tests for generic classificationfrontend/tests/utils/observables.test.js
genericclassification instead of returning nullfor invalid domains and hashes.
mock.js
Update mock stores with new properties and functionsfrontend/tests/mock.js
mockedUseAuthStore.mockedUseOrganizationStoreNoOrgandmockedUseOrganizationStoreOwnerwithisInOrganization.JobActionBar.test.jsx
Update JobActionsBar tests for new rescan API endpointfrontend/tests/components/jobs/result/utils/JobActionBar.test.jsx
JobActionsBarto use new API endpoint forrescanning.
28 files
0025_ingestor_config_virustotal_example_query.py
Added migration for VirusTotal ingestor configuration.api_app/ingestors_manager/migrations/0025_ingestor_config_virustotal_example_query.py
0120_alter_analyzerconfig_not_supported_filetypes_and_more.py
Updated analyzer configuration for file type support.api_app/analyzers_manager/migrations/0120_alter_analyzerconfig_not_supported_filetypes_and_more.py
AnalyzerConfigmodel fields for supported and not supportedfile types.
0126_analyzer_config_nerd_analyzer.py
Added migration for NERD analyzer configuration.api_app/analyzers_manager/migrations/0126_analyzer_config_nerd_analyzer.py
0033_pivot_config_extractedonenotefiles.py
Added migration for ExtractedOneNoteFiles pivot configuration.api_app/pivots_manager/migrations/0033_pivot_config_extractedonenotefiles.py
PivotConfigforExtractedOneNoteFiles.0124_analyzer_config_androguard.py
Added migration for Androguard analyzer configuration.api_app/analyzers_manager/migrations/0124_analyzer_config_androguard.py
Androguardanalyzer configuration.0127_analyzer_config_dshield.py
Added migration for DShield analyzer configuration.api_app/analyzers_manager/migrations/0127_analyzer_config_dshield.py
DShieldanalyzer configuration.0052_playbook_config_uris.py
Added migration for Uris playbook configuration.api_app/playbooks_manager/migrations/0052_playbook_config_uris.py
Urisplaybook configuration.0121_analyzer_config_lnk_info.py
Added migration for Lnk_Info analyzer configuration.api_app/analyzers_manager/migrations/0121_analyzer_config_lnk_info.py
Lnk_Infoanalyzer configuration.elasticsearch.py
Enhanced Elasticsearch client configuration with security settings.threat_matrix/settings/elasticsearch.py
certificate settings.
0123_basic_observable_analyzer.py
Added migration for BasicObservableAnalyzer module.api_app/analyzers_manager/migrations/0123_basic_observable_analyzer.py
BasicObservableAnalyzerPython module.0125_update_yara_repo.py
Updated Yara repository URLs in migration script.api_app/analyzers_manager/migrations/0125_update_yara_repo.py
0051_add_lnk_info_analyzer_free_to_use.py
Added Lnk_Info analyzer to FREE_TO_USE_ANALYZERS playbook.api_app/playbooks_manager/migrations/0051_add_lnk_info_analyzer_free_to_use.py
Lnk_Infoanalyzer inFREE_TO_USE_ANALYZERSplaybook.
0053_add_androguard_to_free_to_use_analyzers.py
Added Androguard analyzer to FREE_TO_USE_ANALYZERS playbook.api_app/playbooks_manager/migrations/0053_add_androguard_to_free_to_use_analyzers.py
Androguardanalyzer inFREE_TO_USE_ANALYZERSplaybook.0122_alter_soft_time_limit.py
Altered soft time limit for Droidlysis analyzer.api_app/analyzers_manager/migrations/0122_alter_soft_time_limit.py
soft_time_limitforDroidlysisanalyzer.0063_singleton_and_elastic_report.py
Added migration for LastElasticReportUpdate singleton model.api_app/migrations/0063_singleton_and_elastic_report.py
LastElasticReportUpdatemodel with singletonconstraint.
aws.py
Added validation for AWS_USER_NUMBER in AWS settings.threat_matrix/settings/aws.py
AWS_USER_NUMBERwhenAWS_SQSis enabled.celery.py
Scheduled send_plugin_report_to_elastic task in Celery.threat_matrix/celery.py
send_plugin_report_to_elastictask to Celery beat schedule.0034_changed_resubmitdownloadedfile_playbook_to_execute.py
Changed playbook for ResubmitDownloadedFile pivot.api_app/pivots_manager/migrations/0034_changed_resubmitdownloadedfile_playbook_to_execute.py
ResubmitDownloadedFilepivot.__init__.py
Import settings from a_secrets in settings initthreat_matrix/settings/init.py
a_secrets.a_secrets.py
Add a_secrets.py for AWS_REGION configurationthreat_matrix/settings/a_secrets.py
a_secrets.pywith AWS_REGION configuration.mail.py
Import AWS_REGION from a_secrets in mail settingsthreat_matrix/settings/mail.py
AWS_REGIONfromawstoa_secrets.db.py
Import AWS_REGION from a_secrets in db settingsthreat_matrix/settings/db.py
AWS_REGIONfromawstoa_secrets.environment.js
Update THREATMATRIX_DOCS_URL to new documentation sitefrontend/src/constants/environment.js
THREATMATRIX_DOCS_URLto the new documentation site URL.celery_ingestor.sh
Add conditional queue logic for AWS_SQS in celery_ingestordocker/entrypoints/celery_ingestor.sh
AWS_SQSenvironmentvariable.
celery_default.sh
Add conditional queue logic for AWS_SQS in celery_defaultdocker/entrypoints/celery_default.sh
AWS_SQSenvironmentvariable.
celery_long.sh
Add conditional queue logic for AWS_SQS in celery_longdocker/entrypoints/celery_long.sh
AWS_SQSenvironmentvariable.
celery_local.sh
Add conditional queue logic for AWS_SQS in celery_localdocker/entrypoints/celery_local.sh
AWS_SQSenvironmentvariable.
uwsgi.sh
Add ELASTIC_TEMPLATE_COMMAND to uwsgi entrypoint scriptdocker/entrypoints/uwsgi.sh
ELASTIC_TEMPLATE_COMMANDto the uwsgi entrypoint script.3 files
secrets.py
Refactored AWS secret retrieval exception handling.threat_matrix/secrets.py
matchstatement for AWS secretretrieval.
abuse_submitter.py
Added exception handling for missing parent job in AbuseSubmitter.api_app/connectors_manager/connectors/abuse_submitter.py
AbuseSubmitter.mwdb_scan.py
Enhance exception handling in mwdb_scan run methodapi_app/analyzers_manager/file_analyzers/mwdb_scan.py
mwdblib.exc.ObjectNotFoundErrorto the exception handling inrunmethod.
6 files
celery.py
Cleaned up imports in Celery settings.threat_matrix/settings/celery.py
malware_bazaar.py
Simplify logger.info statement in malware_bazaarapi_app/ingestors_manager/ingestors/malware_bazaar.py
verify-email.html
Format verify-email HTML templateauthentication/templates/authentication/emails/verify-email.html
reset-password.html
Format reset-password HTML templateauthentication/templates/authentication/emails/reset-password.html
duplicate-email.html
Format duplicate-email HTML templateauthentication/templates/authentication/emails/duplicate-email.html
base.html
Format base HTML templateauthentication/templates/authentication/emails/base.html
3 files
ldap_config.py
Update LDAP configuration documentation URLconfiguration/ldap_config.py
CHANGELOG.md
Update documentation URLs and format changelog.github/CHANGELOG.md
Pivots.jsx
Update description text for Pivots componentfrontend/src/components/plugins/types/Pivots.jsx
1 files
signals.py
Use DRF ValidationError in signalsapi_app/playbooks_manager/signals.py
django.core.exceptions.ValidationErrortorest_framework.exceptions.ValidationError.1 files
security.py
Add CSRF_TRUSTED_ORIGINS setting for CSRF protectionthreat_matrix/settings/security.py
CSRF_TRUSTED_ORIGINSsetting for CSRF protection.90 files
AnalyzerConfigForm.jsx
...frontend/src/components/plugins/forms/AnalyzerConfigForm.jsx
...
PlaybookConfigForm.test.jsx
...frontend/tests/components/plugins/types/forms/PlaybookConfigForm.test.jsx
...
PlaybookConfigForm.jsx
...frontend/src/components/plugins/forms/PlaybookConfigForm.jsx
...
PivotConfigForm.jsx
...frontend/src/components/plugins/forms/PivotConfigForm.jsx
...
AnalyzerConfigForm.test.jsx
...frontend/tests/components/plugins/types/forms/AnalyzerConfigForm.test.jsx
...
PivotConfigForm.test.jsx
...frontend/tests/components/plugins/types/forms/PivotConfigForm.test.jsx
...
pluginsMultiSelectDropdownInput.jsx
...frontend/src/components/common/form/pluginsMultiSelectDropdownInput.jsx
...
pluginActionsButtons.jsx
...frontend/src/components/plugins/types/pluginActionsButtons.jsx
...
RuntimeConfigurationModal.jsx
...frontend/src/components/scan/utils/RuntimeConfigurationModal.jsx
...
runtimeConfigurationInput.jsx
...frontend/src/components/common/form/runtimeConfigurationInput.jsx
...
AppHeader.test.jsx
...frontend/tests/layouts/AppHeader.test.jsx
...
pull_request_template.md
....github/pull_request_template.md
...
PluginsContainer.jsx
...frontend/src/components/plugins/PluginsContainer.jsx
...
AppHeader.jsx
...frontend/src/layouts/AppHeader.jsx
...
InvestigationFlow.test.jsx
...frontend/tests/components/investigations/flow/InvestigationFlow.test.jsx
...
pluginActionsButtons.test.jsx
...frontend/tests/components/plugins/types/pluginActionsButtons.test.jsx
...
package.json
...frontend/package.json
...
PluginsContainers.test.jsx
...frontend/tests/components/plugins/PluginsContainers.test.jsx
...
traefik_prod.yml
...docker/traefik_prod.yml
...
project-requirements.txt
...requirements/project-requirements.txt
...
pluginTableColumns.jsx
...frontend/src/components/plugins/types/pluginTableColumns.jsx
...
TLPSelectInput.jsx
...frontend/src/components/common/form/TLPSelectInput.jsx
...
ScanConfigSelectInput.jsx
...frontend/src/components/common/form/ScanConfigSelectInput.jsx
...
pluginsApi.jsx
...frontend/src/components/plugins/pluginsApi.jsx
...
PluginData.jsx
...frontend/src/components/user/config/PluginData.jsx
...
Home.jsx
...frontend/src/components/home/Home.jsx
...
JobActionBar.jsx
...frontend/src/components/jobs/result/bar/JobActionBar.jsx
...
CODE_OF_CONDUCT.md
....github/CODE_OF_CONDUCT.md
...
elasticsearch.override.yml
...docker/elasticsearch.override.yml
...
api.jsx
...frontend/src/utils/api.jsx
...
SaveAsPlaybooksForm.jsx
...frontend/src/components/jobs/result/bar/SaveAsPlaybooksForm.jsx
...
ci.override.yml
...docker/ci.override.yml
...
create_elastic_certs
...create_elastic_certs
...
Dockerfile
...integrations/malware_tools_analyzers/Dockerfile
...
OrgConfig.jsx
...frontend/src/components/organization/OrgConfig.jsx
...
jobTableColumns.jsx
...frontend/src/components/jobs/table/jobTableColumns.jsx
...
pull_request_automation.yml
....github/workflows/pull_request_automation.yml
...
investigationTableColumns.jsx
...frontend/src/components/investigations/table/investigationTableColumns.jsx
...
plugin_report.json
...configuration/elastic_search_mappings/plugin_report.json
...
jobApi.jsx
...frontend/src/components/jobs/result/jobApi.jsx
...
compose.yml
...integrations/phoneinfoga/compose.yml
...
UserMenu.jsx
...frontend/src/layouts/widgets/UserMenu.jsx
...
ScanForm.advanced.test.jsx
...frontend/tests/components/scan/ScanForm/ScanForm.advanced.test.jsx
...
useOrganizationStore.jsx
...frontend/src/stores/useOrganizationStore.jsx
...
MyOrgPage.jsx
...frontend/src/components/organization/MyOrgPage.jsx
...
README.md
...frontend/README.md
...
CustomJobNode.jsx
...frontend/src/components/investigations/flow/CustomJobNode.jsx
...
scanApi.jsx
...frontend/src/components/scan/scanApi.jsx
...
default.yml
...docker/default.yml
...
start
...start
...
postgres.override.yml
...docker/postgres.override.yml
...
notifications.jsx
...frontend/src/components/jobs/notifications.jsx
...
SECURITY.md
....github/SECURITY.md
...
release_template.md
....github/release_template.md
...
PluginWrapper.jsx
...frontend/src/components/plugins/types/PluginWrapper.jsx
...
GuideWrapper.jsx
...frontend/src/components/GuideWrapper.jsx
...
traefik_local.yml
...docker/traefik_local.yml
...
TokenPage.jsx
...frontend/src/components/user/token/TokenPage.jsx
...
TokenAccess.jsx
...frontend/src/components/user/token/TokenAccess.jsx
...
.env
...docker/.env
...
compose.yml
...integrations/cyberchef/compose.yml
...
issue_template.md
....github/ISSUE_TEMPLATE/issue_template.md
...
new_connector.md
....github/ISSUE_TEMPLATE/new_connector.md
...
env_file_app_template
...docker/env_file_app_template
...
new_analyzer.md
....github/ISSUE_TEMPLATE/new_analyzer.md
...
ScanForm.observable.test.jsx
...frontend/tests/components/scan/ScanForm/requests/ScanForm.observable.test.jsx
...
MultipleObservablesModal.jsx
...frontend/src/components/scan/utils/MultipleObservablesModal.jsx
...
new_ingestor.md
....github/ISSUE_TEMPLATE/new_ingestor.md
...
test-requirements.txt
...requirements/test-requirements.txt
...
new_playbook.md
....github/ISSUE_TEMPLATE/new_playbook.md
...
new_visualizer.md
....github/ISSUE_TEMPLATE/new_visualizer.md
...
redis.override.yml
...docker/redis.override.yml
...
env_file_app_ci
...docker/env_file_app_ci
...
Ingestors.test.jsx
...frontend/tests/components/plugins/types/Ingestors.test.jsx
...
TagSelectInput.jsx
...frontend/src/components/common/form/TagSelectInput.jsx
...
test.multi-queue.override.yml
...docker/test.multi-queue.override.yml
...
threat_matrix_bi.json
...configuration/elastic_search_mappings/threat_matrix_bi.json
...
test.flower.override.yml
...docker/test.flower.override.yml
...
compose-tests.yml
...integrations/malware_tools_analyzers/compose-tests.yml
...
compose-tests.yml
...integrations/tor_analyzers/compose-tests.yml
...
compose.yml
...integrations/tor_analyzers/compose.yml
...
CONTRIBUTING.md
....github/CONTRIBUTING.md
...
compose.yml
...integrations/malware_tools_analyzers/compose.yml
...
FUNDING.yml
....github/FUNDING.yml
...
test.override.yml
...docker/test.override.yml
...
flower.override.yml
...docker/flower.override.yml
...
compose-tests.yml
...integrations/cyberchef/compose-tests.yml
...
env_file_elasticsearch_template
...docker/env_file_elasticsearch_template
...
elasticsearch_instances.yml
...elasticsearch_instances.yml
...
.prettierignore
...frontend/.prettierignore
...