-
Notifications
You must be signed in to change notification settings - Fork 1.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[DRAFT] core: signal blob implementation #9326
Draft
edsiper
wants to merge
24
commits into
master
Choose a base branch
from
signal-blob
base: master
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Draft
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
edsiper
requested review from
fujimotos,
niedbalski,
patrick-stephens,
celalettin1286,
leonardo-albertovich and
koleini
as code owners
September 2, 2024 18:21
Signed-off-by: Eduardo Silva <eduardo@calyptia.com>
Signed-off-by: Eduardo Silva <eduardo@calyptia.com>
Signed-off-by: Eduardo Silva <eduardo@calyptia.com>
Signed-off-by: Eduardo Silva <eduardo@calyptia.com>
Recent patch series add support to process/route large binary files through a zero-copy strategy. This new in_blob plugin allows to scan a path from the file system and register files that matched the pattern. service: flush: 1 log_level: info pipeline: inputs: - name: blob path: '~/logs/blob/*' database_file: blob.db outputs: - name: stdout match: '*' - name: azure_blob match: '*' path: kubernetes container_name: blobs auto_create_container: on database_file: azure.db part_size: 4M upload_parts_timeout: 1s workers: 10 Signed-off-by: Eduardo Silva <eduardo@calyptia.com>
The recent changes in Fluent Bit, allows to process Blob signal types which represents large binary files. When a blob arrives to the plugin, it's enqueued and processed through parts and uploaded as a Block Blob. Part sizes are configurable and survives service restart. example usage: service: flush: 1 log_level: info pipeline: inputs: - name: blob path: '~/logs/blob/*' database_file: blob.db outputs: - name: stdout match: '*' - name: azure_blob match: '*' path: kubernetes container_name: blobs auto_create_container: on database_file: azure.db part_size: 4M upload_parts_timeout: 1s workers: 10 account_name: abcdefghijk shared_key: asdkljaskldjaskldjaskldjasioduasoudaskldjaskld tls: on Signed-off-by: Eduardo Silva <eduardo@calyptia.com>
Signed-off-by: Eduardo Silva <eduardo@calyptia.com>
Signed-off-by: Eduardo Silva <eduardo@calyptia.com>
edsiper
force-pushed
the
signal-blob
branch
from
September 16, 2024 19:30
2edf300
to
e7defa8
Compare
Signed-off-by: leonardo-albertovich <leonardo.alminana@chronosphere.io>
leonardo-albertovich
had a problem deploying
to
pr
September 17, 2024 12:32
— with
GitHub Actions
Failure
leonardo-albertovich
had a problem deploying
to
pr
September 17, 2024 12:32
— with
GitHub Actions
Failure
leonardo-albertovich
had a problem deploying
to
pr
September 17, 2024 12:32
— with
GitHub Actions
Failure
Signed-off-by: leonardo-albertovich <leonardo.alminana@chronosphere.io>
leonardo-albertovich
had a problem deploying
to
pr
September 17, 2024 12:33
— with
GitHub Actions
Failure
leonardo-albertovich
had a problem deploying
to
pr
September 17, 2024 12:33
— with
GitHub Actions
Failure
leonardo-albertovich
had a problem deploying
to
pr
September 17, 2024 12:33
— with
GitHub Actions
Failure
Interesting, I did a POC a while back using Fluent Bit to essentially rsync files for "reasons" but that has issues with the line-driven approach not guaranteeing ordering which this would I presume now. |
Yes, you are right @patrick-stephens |
leonardo-albertovich
had a problem deploying
to
pr
September 20, 2024 11:10
— with
GitHub Actions
Failure
leonardo-albertovich
had a problem deploying
to
pr
September 20, 2024 11:10
— with
GitHub Actions
Failure
leonardo-albertovich
had a problem deploying
to
pr
September 20, 2024 11:10
— with
GitHub Actions
Failure
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The following pull request is an active work in process implementation of Blob signal.
Blob
There are use cases where is desired to move very large files with Fluent Bit, and most of the time these are binary files like videos, AI models or others.
We implement the handling of binary files through the new signal called Blob which has the following design principles:
Note that the code base in this branch is in active development and and might be updated with breaking changes.
Components and status
High-level list of things to implement in this pull request:
in_blob
core
out_azure_blob
Other changes in out_azure_blob:
Database example
Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.