Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add ability to throttle exports when reading from disk. #638

Open
breedx-splk opened this issue Oct 17, 2024 · 5 comments
Open

Add ability to throttle exports when reading from disk. #638

breedx-splk opened this issue Oct 17, 2024 · 5 comments
Assignees
Labels
enhancement New feature or request

Comments

@breedx-splk
Copy link
Contributor

If the user is offline or in airplane mode for a long time and a lot of data has buffered on disk, it is possible for the agent to use a lot of bandwidth. There is currently nothing that limits how much or how fast the data should be sent. Instead, we just try and export all the data on disk as fast as we can...and this can be a problem because it hogs a limited resource.

This feature could be opt-in or opt-out, but there should be a way to configure the max "bandwidth" used by the SignalFromDisk exporters. I used the quotes around "bandwidth" because I'm not sure that it needs to be strict bytes/time style, and we might just be able to do batch/time or spans/time or something similar.

This would ideally allow the data to flow more smoothly/consistently instead of "all at once".

@breedx-splk breedx-splk added the enhancement New feature or request label Oct 17, 2024
@breedx-splk
Copy link
Contributor Author

There is some prior art here in the Splunk ThrottlingExporter, but this is not exactly what we want because it favors dropping data instead of allowing it to stay buffered (on disk).

@Victorsesan
Copy link

@breedx-splk Can i be allowed to help with this one? thanks

@breedx-splk
Copy link
Contributor Author

Sure, thanks!

@Victorsesan
Copy link

@breedx-splk Please help me out, can we create a new class/file in the android/export directory in this repo with a name like RateLimitedExporter.java to implement a SpanExporter interface which can help with this? or in which file can we implement the bandwidth-limiting feature in?. Thanks for the help

@breedx-splk
Copy link
Contributor Author

@Victorsesan Yes, you would likely want to create a new package and new class(es) to implement this new functionality. Please review CONTRIBUTING.md and make sure include tests and to format with ./gradlew spotlessApply.

Victorsesan added a commit to Victorsesan/opentelemetry-android that referenced this issue Oct 27, 2024
…idth usage when exporting spans, allowing for smoother data flow and preventing resource hogging. It can further refine the size estimation logic based on a specific use case.

Relate to Add ability to throttle exports when reading from disk. open-telemetry#638
Victorsesan added a commit to Victorsesan/opentelemetry-android that referenced this issue Oct 30, 2024
Victorsesan added a commit to Victorsesan/opentelemetry-android that referenced this issue Nov 1, 2024
…nterface and Duration with a plain long value for timeWindowInMillis

 ref open-telemetry#638
Victorsesan added a commit to Victorsesan/opentelemetry-android that referenced this issue Nov 2, 2024
Ref: Add ability to throttle exports when reading from disk. open-telemetry#638
Victorsesan added a commit to Victorsesan/opentelemetry-android that referenced this issue Nov 14, 2024
Victorsesan added a commit to Victorsesan/opentelemetry-android that referenced this issue Nov 14, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

2 participants