Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Memory leak when ingesting an opencti stream configured with an empty starting synchronization date #9358

Open
JeremyCloarec opened this issue Dec 16, 2024 · 0 comments · May be fixed by #9555
Assignees
Labels
bug use for describing something not working as expected
Milestone

Comments

@JeremyCloarec
Copy link
Contributor

JeremyCloarec commented Dec 16, 2024

Description

The current handling of stream ingestion in syncManager can lead to a memory leak when trying to ingest a stream with a lot of data.
The problem is identified: the manageBackPressure method in syncManager doesn't work properly, leading to the eventsQueue length increasing without any limit

Environment

  1. OS (where OpenCTI server runs): { e.g. Mac OS 10, Windows 10, Ubuntu 16.4, etc. }
  2. OpenCTI version: { e.g. OpenCTI 1.0.2 }
  3. OpenCTI client: { e.g. frontend or python }
  4. Other environment details:

Reproducible Steps

Steps to create the smallest reproducible scenario:

  1. Add an opencti stream ingestion on a platform with a large amount of data

Expected Output

The data is ingested at a steady pace with a steady memory footprint

Actual Output

The memory used by openCTI increases without stopping

Additional information

Screenshots (optional)

@JeremyCloarec JeremyCloarec added bug use for describing something not working as expected needs triage use to identify issue needing triage from Filigran Product team labels Dec 16, 2024
@JeremyCloarec JeremyCloarec self-assigned this Dec 16, 2024
@JeremyCloarec JeremyCloarec removed the needs triage use to identify issue needing triage from Filigran Product team label Dec 16, 2024
@romain-filigran romain-filigran added this to the Bugs backlog milestone Dec 17, 2024
@SamuelHassine SamuelHassine added the solved use to identify issue that has been solved (must be linked to the solving PR) label Jan 9, 2025
@SamuelHassine SamuelHassine reopened this Jan 10, 2025
@SamuelHassine SamuelHassine removed the solved use to identify issue that has been solved (must be linked to the solving PR) label Jan 10, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug use for describing something not working as expected
Projects
None yet
3 participants