[Comlaude] improvements

[yassine-ouaamou]

This PR contains the code commited by @MohamedMerimi:

Proposed changes

• I added the “_deserialize_json_string” function to convert the JSON-formatted text string into a python object, which eliminates json errors when executing on OpenCTI.
• I Added “_validate_required_fields” function to check that all required data (mandatory fields) are present and not empty, which also prevents errors in OpenCTI executions
• Addition of the “_ping_connector” function, which pings the connector every 5 minutes to keep it active (by modifying the “run” function).
• Status tracking with “last_run” variable in format “{”last_run“: ‘2024-11-18T15:13:36Z’}”.

Related issues

• [Comlaude] improvements #3054

[Megafredo]

Hello @yassine-ouaamou, and @MohamedMerimi,
Since we don't have any identifying information to test this community connector, we can't test the quality of data ingestion from this connector, so we'll just skim the connector code to highlight best practices.

Suggestions for Code Improvement

During this PR, I took the time to examine the code as a whole, and here are a few points I think would be useful with best practices :

1. README

• Add connector_duration_period variable to explain its configuration (see point 6)
• Add connector_queue_threshold variable if relevant (default: 500MB) (see point 6)
• Remove reference to interval (see point 6)
• Delete connector_update_existing_data, as it is depreciated
• Add missing variable comlaude_score
• Correct a bug: it's not config_username, but comlaude_username

2. Docker Compose

• Remove CONFIG_INTERVAL to avoid confusion
• Add CONNECTOR_DURATION_PERIOD and CONNECTOR_QUEUE_THRESHOLD (if necessary)

3. config.yml.sample

• Remove update_existing_data from the comlaude section.
• Add score in the comlaude section.
• Add duration_period in the connector section (for example, PT2H for a 2-hour period).
• Include queue_threshold if necessary in the connector section.

4. Import management

• When using isort, we recommend applying the --profil black to ensure consistent formatting. However, path-related problems can sometimes occur. Here's a suggestion for modifying imports to avoid ignoring the rules imposed by isort.

import datetime  
import json  
import os  
import sys  
import threading  
import time  

import comlaude  
import stix2  
import yaml  
from pycti import (  
    Identity,  
    Indicator,  
    OpenCTIConnectorHelper,  
    StixCoreRelationship,  
    get_config_variable,  
)  
from stix2 import TLP_AMBER, Bundle, DomainName  

5. requirements.txt

• Explicitly add versions of dependencies used to ensure compatibility (e.g. pycti==6.4.2, requests==2.32.2)

6. Code cleaning

1. Obsolete Variable and Method: interval

• The following method, which I believe is never called, and any associated references can be deleted :

def _get_interval(self):
 """
 Get the interval of execution in seconds.
 :return: Interval in seconds.
 """
 return int(self.config_interval) * 60 * 60

• Replace the use of interval with the recommended method: schedule_iso for better scheduling and automatic backpressure management (See blog post for details : https://filigran.io/auto-backpressue-control-octi-connectors/)

def run(self):  
    self.helper.schedule_iso(  
        message_callback=self.your_old_process_run,  
        duration_period=self.duration_period  
    )

2. Deprecated variable: update_existing_data

• Delete this obsolete configuration:

self.update_existing_data = get_config_variable(
    "CONFIG_UPDATE_EXISTING_DATA",
    ["comlaude", "update_existing_data"],
    self.config,
    isNumber=True,
)

• And also its reference in the call to send_stix2_bundle :

self.helper.send_stix2_bundle(
	bundle.serialize(),
	update=self.update_existing_data, # <<<<<< to be deleted
	work_id=self.work_id,
)

3. Improving bundles:

• To improve the quality of STIX bundles, use cleanup_inconsistent_bundle. Note that it is important to add all the markings used (e.g. TLP_AMBER):

self.helper.send_stix2_bundle(
    bundle.serialize(),
    cleanup_inconsistent_bundle=True,
    work_id=self.work_id,
)

4. Modernised logger :

• Replace the old self.helper.log_error() calls with the new format self.helper.connector_logger.error("Message", {"foo": foo, "bar": bar})

5. Closing Work ID:

• Remember to mark the work id as finished, even if you make a mistake, for example in the finally

finally:  
    if self.work_id is not None:  
        self.helper.api.work.to_processed(self.work_id, "Finished")

6. Replacing utcnow() :

• datetime.utcnow()` has been deprecated since Python 3.12. See Python Doc

7. Adding a traceback

• To improve exception handling in the main input of the script, add a traceback to the main block:

if __name__ == "__main__":
    """
    Entry point of the script.
    """
    import traceback

    try:
        connector = ComlaudeConnector()
        connector.run()
    except Exception:  
        traceback.print_exc()  
        sys.exit(1)

[flavienSindou]

Could you please use the logger rather than a print please ? (and preferably when using the method rather than directly in it : this is not this method that skips but the code that uses its result)

[flavienSindou]

Thanks for cleaning the code base 👍

[flavienSindou]

👍