Merge branch 'develop' into Crowdstrike_Alerts_2

CHANGELOG.md

@@ -14,6 +14,27 @@ We have started this changelogs from version 4.0.0. So, changes on previously re
 ### Dependency update:
 *List the dependecy upgrade or downgrade.*
 
+## 7.1.0 (2024-08-30)
+
+### Breaking changes:
+
+* Moving Flask Out of Core Stix-Shifter [#1730](https://github.com/opencybersecurityalliance/stix-shifter/pull/1730)
+
+### Deprecations:
+
+### Changes:
+
+* Warning added for unmapped from_stix fields. [#1729](https://github.com/opencybersecurityalliance/stix-shifter/pull/1729)
+* Update code-coverage.yml [#1711](https://github.com/opencybersecurityalliance/stix-shifter/pull/1711)
+
+
+### Fixes:
+
+* Domain regex in some connectors had potential for performance issues [#1728](https://github.com/opencybersecurityalliance/stix-shifter/pull/1728)
+* Rest API was not properly pulling in environment variables for proxy [#1727](https://github.com/opencybersecurityalliance/stix-shifter/pull/1727)
+
+### Dependency update:
+
 --------------------------------------
 
 ## 7.0.12 (2024-08-20)
@@ -857,4 +878,4 @@ opencybersecurityalliance/stix-shifter/pull/1448)
 * Bump adal from 1.2.2 to 1.2.7
 * Bump pyopenssl from 20.0.1 to 22.0.0
 * Bump stix2-validator from 1.1.2 to 3.0.2
-* Bump boto3 from 1.17.20 to 1.21.5## 4.0.0 (2022-02-23)
+* Bump boto3 from 1.17.20 to 1.21.5## 4.0.0 (2022-02-23)

docs/adapter-guide/develop-stix-adapter.md

@@ -94,18 +94,27 @@ If the data source is synchronous, you must include `set_async(False)` in the co
 
 #### Testing a new connector using the proxy host
 
-Work on a new stix-shifter connector occurs after the project has been forked and cloned into a local development environment. Stix-shifter contains a **proxy** connector that facilitates a remote instance of the project calling out to a local instance. While in development, a new connector's working branch can be tested in any project using the stix-shifter library without first merging into the master branch on Github. A host is run on the local instance from the CLI. When a `proxy` data source is passed to the remote instance of stix-shifter, the real connection attributes (data source type, host, and port contained in the options) are passed onto the local instance of stix-shifter running the proxy host. The host will then use the new connector and return results back to the remote stix-shifter instance.
+The Proxyhost server routing provides a python file with all of the required mappings to start a flask server that can be used to allow communication through the proxy module. The flask server is run on the local instance from the CLI. When a `proxy` data source is passed to the remote instance of stix-shifter, the real connection attributes (data source type, host, and port contained in the options) are passed onto the local instance of stix-shifter running the proxy host. The host will then use the new connector and return results back to the remote stix-shifter instance.
 
-Open a terminal and navigate to your local stix-shifter directory. Run the host with the following command:
+##### Requirements
+1. stix-shifter-utils must be installed.
+2. flask must be installed.
+3. You need to be able to access the proxyhost_server_routing.py from your CLI.
+
+##### Running the tool
+
+To start a flask server with the proxy host routing map you can run the following command.
 
 ```
-python main.py host "<STIX Identity Object>" "<Host IP address>:<Host Port>"
+flask --app 'proxyhost_server_routing:start_proxyhost_flask_server({"type": "identity","id": "identity--f431f809-377b-45e0-aa1c-6a4751cae5ff","name": "Bundle","identity_class": "events"})' run'
 ```
 
-As an example:
+Running this command starts up the server on localhost and on port 5000.
+
+To start the server up the same way as it was done previously (through the stix-shifter CLI)
 
 ```
-python main.py host '{"type": "identity","id": "identity--f431f809-377b-45e0-aa1c-6a4751cae5ff","name": "Bundle","identity_class": "events"}' "192.168.122.83:5000"
+flask --app 'proxyhost_server_routing:start_proxyhost_flask_server({"type": "identity","id": "identity--f431f809-377b-45e0-aa1c-6a4751cae5ff","name": "Bundle","identity_class": "events"})' run -h 127.0.0.1 -p 5001 --cert "/Users/StixShifter/stix-shifter/cert.pem" --key "/Users/StixShifter/stix-shifter/key.pem"
 ```
 
 ##### Calling the proxy host

stix_shifter/requirements.txt

@@ -5,7 +5,7 @@ antlr4-python3-runtime==4.8
 asyncio==3.4.3
 asynctest==0.13.0
 attrs==23.1.0
-azure-identity==1.15.0
+azure-identity==1.16.1
 colorlog==6.8.2
 flask==3.0.0
 flatten_json==0.1.14

stix_shifter/scripts/stix_shifter.py

@@ -3,20 +3,17 @@
 import json
 import time
 import importlib
-from flask import Flask
 import logging
 import copy
 from stix_shifter.stix_translation import stix_translation
 from stix_shifter.stix_transmission import stix_transmission
-from stix_shifter_utils.utils.proxy_host import ProxyHost
 from stix_shifter_utils.utils.module_discovery import process_dialects, modules_list
 from stix_shifter_utils.utils import logger as utils_logger
 from stix_shifter_utils.utils.logger import exception_to_string
 
 TRANSLATE = 'translate'
 TRANSMIT = 'transmit'
 EXECUTE = 'execute'
-HOST = 'host'
 MAPPING = 'mapping'
 MODULES = 'modules'
 CONFIGS = 'configs'
@@ -170,28 +167,6 @@ def main():
  execute_parser.add_argument('-r', '--results', type=int, default=10,
  help='Maximum number of returned results (default 10)')
 
- host_parser = parent_subparsers.add_parser(HOST, help='Host a local query service, for testing and development')
- host_parser.add_argument(
- 'data_source',
- type=str,
- help='STIX Identity object for the data source'
- )
- host_parser.add_argument(
- 'host_address',
- type=str,
- help='Proxy Host:Port'
- )
- host_parser.add_argument(
- 'ssl_cert',
- type=str,
- help='SSL certificate filename'
- )
- host_parser.add_argument(
- 'ssl_key',
- type=str,
- help='SSL key filename'
- )
-
  args = parent_parser.parse_args()
 
  help_and_exit = args.command is None
@@ -243,56 +218,6 @@ def main():
  if help_and_exit:
  parent_parser.print_help(sys.stderr)
  sys.exit(1)
- elif args.command == HOST:
- # Host means to start a local web service for STIX shifter, to use in combination with the proxy data source
- # module. This combination allows one to run and debug their stix-shifter code locally, while interacting with
- # it inside a service provider such as IBM Security Connect
- app = Flask("stix-shifter")
-
- @app.route('/transform_query', methods=['POST'])
- def transform_query():
- host = ProxyHost()
- return host.transform_query()
-
- @app.route('/translate_results', methods=['POST'])
- def translate_results():
- data_source_identity_object = args.data_source
- host = ProxyHost()
- return host.translate_results(data_source_identity_object)
-
- @app.route('/create_query_connection', methods=['POST'])
- def create_query_connection():
- host = ProxyHost()
- return host.create_query_connection()
-
- @app.route('/create_status_connection', methods=['POST'])
- def create_status_connection():
- host = ProxyHost()
- return host.create_status_connection()
-
- @app.route('/create_results_connection', methods=['POST'])
- def create_results_connection():
- host = ProxyHost()
- return host.create_results_connection()
-
- @app.route('/delete_query_connection', methods=['POST'])
- def delete_query_connection():
- host = ProxyHost()
- return host.delete_query_connection()
-
- @app.route('/ping', methods=['POST'])
- def ping_connection():
- host = ProxyHost()
- return host.ping_connection()
-
- @app.route('/is_async', methods=['POST'])
- def is_async():
- host = ProxyHost()
- return host.is_async()
-
- host_address = args.host_address.split(":")
- app.run(debug=True, port=int(host_address[1]), host=host_address[0], ssl_context=(args.ssl_cert, args.ssl_key))
-
  elif args.command == EXECUTE:
  # Execute means take the STIX SCO pattern as input, execute query, and return STIX as output
 

stix_shifter_modules/okta/stix_transmission/connector.py

@@ -12,7 +12,7 @@ class InvalidMetadataException(Exception):
 
 class Connector(BaseJsonSyncConnector):
  OKTA_MAX_PAGE_SIZE = 1000
- DOMAIN_PATTERN = re.compile(r'^(?:[a-z0-9](?:[a-z0-9-]{0,61}[a-z0-9])?(\.)?)+[a-z0-9][a-z0-9-]{0,61}[a-z0-9]$')
+ DOMAIN_PATTERN = re.compile(r'^(?:(?:[a-z0-9]\.)|(?:[a-z0-9][a-z0-9-]{0,61}[a-z0-9]\.))+[a-z0-9][a-z0-9-]{0,61}[a-z0-9]$')
 
  def __init__(self, connection, configuration):
  self.api_client = APIClient(connection, configuration)

stix_shifter_modules/vectra/stix_translation/transformers.py

@@ -53,8 +53,7 @@ def transform(obj):
  try:
  if obj and isinstance(obj, list):
  ip_pattern = re.compile(r'(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})')
- domain_pattern = re.compile(r'^(?:[a-z0-9](?:[a-z0-9-]{0,61}[a-z0-9])?(\.)?)+[a-z0-9][a-z0-9-]{0,'
- r'61}[a-z0-9]$')
+ domain_pattern = re.compile(r'^(?:(?:[a-z0-9]\.)|(?:[a-z0-9][a-z0-9-]{0,61}[a-z0-9]\.))+[a-z0-9][a-z0-9-]{0,61}[a-z0-9]$')
  for index, row in enumerate(obj):
  if domain_pattern.search(row):
  continue

stix_shifter_tools/External_Tools.MD

@@ -0,0 +1,37 @@
+Stix Shifter Tools
+===============
+These are development and testing tools that can be used to assist with developing STIX Shifter. These tools are used externally to the core of STIX Shifter. The intended purpose behind this folder is to hold useful scripts and code that should not be shipped with the stix-shifter package. Included below will be a list of these tools, including their purpose and how to use them.
+
+# Proxy Host Server Routing
+The Proxyhost server routing provides a python file with all of the required mappings to start a flask server that can be used to allow communication through the proxy module. The flask server is run on the local instance from the CLI. When a `proxy` data source is passed to the remote instance of stix-shifter, the real connection attributes (data source type, host, and port contained in the options) are passed onto the local instance of stix-shifter running the proxy host. The host will then use the new connector and return results back to the remote stix-shifter instance.
+
+# Requirements
+1. stix-shifter-utils must be installed.
+2. flask must be installed.
+3. You need to be able to access the proxyhost_server_routing.py from your CLI.
+
+# Running the tool
+
+To start a flask server with the proxy host routing map you can run the following command.
+
+```
+flask --app 'proxyhost_server_routing:start_proxyhost_flask_server({"type": "identity","id": "identity--f431f809-377b-45e0-aa1c-6a4751cae5ff","name": "Bundle","identity_class": "events"})' run'
+```
+
+Running this command starts up the server on localhost and on port 5000.
+
+To start the server up the same way as it was done previously (through the stix-shifter CLI)
+
+```
+flask --app 'proxyhost_server_routing:start_proxyhost_flask_server({"type": "identity","id": "identity--f431f809-377b-45e0-aa1c-6a4751cae5ff","name": "Bundle","identity_class": "events"})' run -h 127.0.0.1 -p 5001 --cert "/Users/StixShifter/stix-shifter/cert.pem" --key "/Users/StixShifter/stix-shifter/key.pem"
+```
+
+# Calling the proxy host
+
+Each of the translate and transmit CLI commands outlined in the stix-shifter overview can be used to call the proxy host.
+
+As an example:
+
+```
+python main.py transmit proxy '{"options": {"proxy_host": "127.0.0.1", "proxy_port": 5000, "destination": {"connection": {"options": {"result_limit": 10000, "time_range": 5, "timeout": 30}, "host": "<HOST>", "port": <PORT>, "type": "qradar"}, "configuration": {"auth": { "SEC": "<SEC TOKEN>"} } } }}' '{}' ping
+```

stix_shifter_utils/utils/proxy_host.py → stix_shifter_tools/proxyhost_flask_server/proxyhost_server_routing.py

@@ -1,12 +1,63 @@
+import json
+from flask import Flask, request
+
 from stix_shifter.stix_translation import stix_translation
 from stix_shifter.stix_transmission import stix_transmission
 from stix_shifter_utils.utils import logger
 from flask import request
 import json
+
+# Start a local web service for STIX shifter, to use in combination with the proxy data source
+# module. This combination allows one to run and debug their stix-shifter code locally, while interacting with
+# it inside a service provider such as IBM Security Connect
 
+def start_proxyhost_flask_server(data_source):
+ app = Flask("stix-shifter")
 
-class ProxyHost():
+ @app.route('/transform_query', methods=['POST'])
+ def transform_query():
+ host = ProxyHost()
+ return host.transform_query()
+
+ @app.route('/translate_results', methods=['POST'])
+ def translate_results():
+ data_source_identity_object = data_source
+ host = ProxyHost()
+ return host.translate_results(data_source_identity_object)
+
+ @app.route('/create_query_connection', methods=['POST'])
+ def create_query_connection():
+ host = ProxyHost()
+ return host.create_query_connection()
 
+ @app.route('/create_status_connection', methods=['POST'])
+ def create_status_connection():
+ host = ProxyHost()
+ return host.create_status_connection()
+
+ @app.route('/create_results_connection', methods=['POST'])
+ def create_results_connection():
+ host = ProxyHost()
+ return host.create_results_connection()
+
+ @app.route('/delete_query_connection', methods=['POST'])
+ def delete_query_connection():
+ host = ProxyHost()
+ return host.delete_query_connection()
+
+ @app.route('/ping', methods=['POST'])
+ def ping_connection():
+ host = ProxyHost()
+ return host.ping_connection()
+
+ @app.route('/is_async', methods=['POST'])
+ def is_async():
+ host = ProxyHost()
+ return host.is_async()
+
+ return app
+
+class ProxyHost():
  def __init__(self):
  self.logger = logger.set_logger(__name__)
  self.request_args = request.get_json(force=True)
@@ -69,4 +120,4 @@ async def ping_connection(self):
  def is_async(self):
  transmission_module = self.connection['type'].lower()
  transmission = stix_transmission.StixTransmission(transmission_module, self.connection, self.configuration)
- return "{}".format(transmission.is_async())
+ return "{}".format(transmission.is_async())

stix_shifter_utils/modules/base/stix_translation/base_query_translator.py

@@ -114,8 +114,10 @@ def transform_query(self, data):
  unmapped_operator = stripped_parsing.get('unmapped_operator')
  if unmapped_stix:
  unmapped_stix_collection.extend(unmapped_stix)
+ self.logger.warn(f"The following STIX fields are not supported : {set(unmapped_stix_collection)} with dialect {self.dialect}. The request will ignore those fields. This can result in results that do not match the request.")
  if unmapped_operator:
  unmapped_operator_collection.extend(unmapped_operator)
+ self.logger.warn(f"The following STIX operators are not supported : {set(unmapped_operator_collection)} with dialect {self.dialect}. The request will ignore those fields. This can result in results that do not match the request.")
  if antlr_parsing:
  translated_queries = self.transform_antlr(data, antlr_parsing)
  if isinstance(translated_queries, str):