Malcolm v24.11.0

[mmguero]

Malcolm v24.11.0 contains a new threat intelligence feed integration, a few new API calls, other minor improvements, bug fixes, and component version updates.

v24.10.1...v24.11.0

• Features and enhancements
  • Added dashboard-export to the list of Malcolm APIs (add API for exporting dashboard cisagov/Malcolm#401)
  • Added ingest-stats to the list of Malcolm APIs (Add ingest-stats API cisagov/Malcolm#488)
  • Added support for pulling from the Mandiant Threat Intelligence service to feed the Zeek intelligence framework as used by Malcolm's and Hedgehog Linux's Zeek processes. The integration uses the google/mandiant-ti-client library for Python. (Zeek intel support for Mandiant Advantage threat intelligence feed cisagov/Malcolm#358)
  • Improved normalization of Zeek's intel.log to the ECS's threat fields
  • Improved the Zeek Intel dashboard
  • Improved the health/liveness probe for the Logstash container
  • Changed behavior of Malcolm's non-live Zeek container (responsible for processing uploaded PCAPs) so that it becomes available to process data even before an intelligence feed pull is finished
  • Implemented paging for extracted files download dialog (implement paging for extracted files download dialog cisagov/Malcolm#361)
  • Implemented support for sending Zeek logs to Kafka using the SeisoLLC/zeek-kafka plugin (include support for sending Zeek logs to Kafka cisagov/Malcolm#357)
  • Added the NetBox HealthCheck plugin as a default NetBox plugin
  • Updated the Malcolm services readiness status API to use the new LogStash health report API and the NetBox HealthCheck plugin as the basis for reporting the state of LogStash and NetBox, respectively.
  • Added parsing for the new OPCUA-Binary write subscription service log
• Component version updates
  • Arkime to v5.5.0
  • Beats to v8.16.0
  • elasticsearch Python library to v8.16.0
  • elasticsearch-dsl Python library to v8.16.0
  • evtx to v0.8.4
  • LogStash to v8.16.0
  • OpenSearch and OpenSearch Dashboard to v2.18.0
  • watchdog Python library to v6.0.0
  • werkzeug Python library to v3.0.6 to address CVE-2024-49767 and CVE-2024-49766
• Bug fixes
  • Fixed an issue with the ./scripts/configure script not prompting to regenerate the internal NetBox passwords when it should have
  • Fixed errors when running malcolm_appliance_packager.sh on macOS (fixed errors when running appliance packager on macOS cisagov/Malcolm#492, thanks @robrui)
• Configuration changes (in environment variables in ./config/) for Malcolm and in control_vars.conf for Hedgehog Linux
  • Malcolm
    • ZEEK_KAFKA_ENABLED, ZEEK_KAFKA_BROKERS, and ZEEK_KAFKA_TOPIC have been added to ./config/zeek.env, which can be used to enable Zeek's sending of its logs to Kafka (include support for sending Zeek logs to Kafka cisagov/Malcolm#357)
    • ZEEK_DISABLE_DETECT_ROUTERS (default value: true) has been added to ./config/zeek.env which controls an experimental Zeek script for detecting the presence of routers (logging them to known_routers.log) in a network based on packet TTL; it is recommended to leave this set to true as this script is not yet ready for general production use
    • ZEEK_INTEL_REFRESH_ON_STARTUP has been renamed from ZEEK_INTEL_REFRESH_ON_ENTRYPOINT in ./config/zeek.env to more accurately reflect the purpose of the variable
  • Hedgehog Linux
    • ZEEK_KAFKA_ENABLED, ZEEK_KAFKA_BROKERS, and ZEEK_KAFKA_TOPIC have been added to control_vars.conf for the same purpose as described above
    • ZEEK_DISABLE_DETECT_ROUTERS (default value: true) has been added to control_vars.conf for the same purpose as described above
    • ZEEK_INTEL_REFRESH_ON_STARTUP has been renamed from ZEEK_INTEL_REFRESH_ON_ENTRYPOINT in control_vars.conf to more accurately reflect the purpose of the variable
• Code and project maintenance
  • All open issues and the project board have been migrated from the Idaho National Lab fork to the upstream CISA fork. The repos will continue to be kept in sync going forward. (migrate all GitHub issues from idaholab to cisagov repo cisagov/Malcolm#350)