Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[stream replication] first version of connectors #2730

Open
wants to merge 15 commits into
base: master
Choose a base branch
from

Conversation

axelfahy
Copy link
Contributor

This PR adds two new connectors, a stream/stream-exporter and a external-import/stream-importer. The goal is to be able to synchronize two OpenCTI instances that don't have access to each other.

The stream-exporter reads a live-stream and save all events on minio.
The stream-importer reads the files from minio and send them to rabbitmq to be inserted in OpenCTI by the workers.

Proposed changes

  • Add a stream/stream-exporter connector that reads events from a live stream and save them to minio (in bulk).
  • Add a external-importer/stream-importer connector that reads files from minio (containing the events of the live stream) and send them to rabbitmq to be imported in OpenCTI.

Related issues

  • N/A

Checklist

  • I consider the submitted work as finished
  • I tested the code for its functionality using different use cases
  • I added/update the relevant documentation (either on github or on notion)
  • Where necessary I refactored code to improve the overall quality

Further comments

@axelfahy axelfahy force-pushed the feature/stream-replication branch from 24784a1 to 4d7998a Compare October 1, 2024 07:26
@axelfahy axelfahy marked this pull request as ready for review October 1, 2024 07:50
@axelfahy axelfahy force-pushed the feature/stream-replication branch from 4d7998a to ec440a7 Compare October 15, 2024 06:29
@axelfahy axelfahy force-pushed the feature/stream-replication branch 2 times, most recently from adf7919 to b08032c Compare November 8, 2024 08:45
@helene-nguyen helene-nguyen self-requested a review November 15, 2024 08:31
stream/stream-exporter/Dockerfile Outdated Show resolved Hide resolved
stream/stream-exporter/Dockerfile Outdated Show resolved Hide resolved
stream/stream-exporter/entrypoint.sh Outdated Show resolved Hide resolved
external-import/stream-importer/entrypoint.sh Outdated Show resolved Hide resolved
external-import/stream-importer/Dockerfile Outdated Show resolved Hide resolved
external-import/stream-importer/Dockerfile Outdated Show resolved Hide resolved
@axelfahy axelfahy force-pushed the feature/stream-replication branch from eff399a to d582448 Compare November 22, 2024 09:21
@helene-nguyen helene-nguyen added the community use to identify PR from community label Dec 5, 2024
@axelfahy axelfahy force-pushed the feature/stream-replication branch from d582448 to bf887ea Compare December 19, 2024 09:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
community use to identify PR from community
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants