Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[WIP] Add redis storage extension #31731

Closed

Conversation

pepperkick
Copy link
Contributor

Description:
Add Redis storage extension to utilize it as exporter queue.

Link to tracking Issue:
#31682

Testing:

Did basic functionality testing.

  • Had 2 collectors, A and B
  • A exports to B with redis queue enabled
  • Took down B and sent 100 traces to A
  • Ensured 100 entries were in redis
  • Started collector B and ensured 100 traces were received by it

Documentation:
TBD

@pepperkick pepperkick requested a review from a team March 13, 2024 14:54
@pepperkick pepperkick marked this pull request as draft March 13, 2024 14:54
@pepperkick
Copy link
Contributor Author

I do have one major concern.

I was able to implement basic setter and getter with Redis. But one of the main of the main feature that I wanted was that multiple collectors could connect to the same Redis DB and during dequeue they can distribute the work.

The way l implemented it before was to add a lock function in exporterhelper (collector v0.70.0) which would create a lock key against the actual key with short TTL. That way a collector can lock a key, process it and then delete it. In case the collector fails, so other collector can pick it up after the lock expires.

Doing the same in the current storage interface will require an additional Lock operation.

@atoulme atoulme changed the title [WIP} Add redis storage extension [WIP] Add redis storage extension Mar 13, 2024
@atoulme atoulme added the Sponsor Needed New component seeking sponsor label Mar 13, 2024
Copy link
Contributor

This PR was marked stale due to lack of activity. It will be closed in 14 days.

Copy link
Contributor

Closed as inactive. Feel free to reopen if this PR is still being worked on.

@github-actions github-actions bot closed this Apr 12, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
extension/storage Sponsor Needed New component seeking sponsor Stale
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants