Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Persistence queue #861

Merged
merged 31 commits into from
Sep 6, 2023
Merged

Persistence queue #861

merged 31 commits into from
Sep 6, 2023

Conversation

VihasMakwana
Copy link
Contributor

@VihasMakwana VihasMakwana commented Jul 27, 2023 •
edited
Loading

Things included in this PR:

  • enable persistent queues configuration in the helm chart for an agent,

  • Don't enable the persistent queues for following scenarios:

    • If the gateway mode is enabled. Gateway uses multiple replicas by default, and our filestorage extension will try to acquire an exclusive lock for the directory and it will cause a deadlock. Only one pod will be in a healthy state if we enable a persistent queue for gateway.
    • If the gke/autopilot or eks/fargate are used. Both of these distributions don't allow hostPath mounts.

  • Will add the same thing in the docs

wojtekzyla and others added 14 commits May 4, 2023 23:48
@wojtekzyla
fix: Add persistent queue
7d75c48 
Delete .DS_Store

Delete .DS_Store

remove persistent queue from cluster receiver and gateway

Delete .DS_Store

Delete .DS_Store

update .gitignore
@atoulme
Merge branch 'main' into persistent-queue-updated
0429ea5
@wojtekzyla
fix: rename splunkSendingQueue to splunkPlatformSendingQueue, add a c…
57e3fbb 
…hangelog entry, remove from gitignore .DS_Store
@wojtekzyla
Merge branch 'main' into persistent-queue-updated
c47660b
@wojtekzyla @jvoravong
Update helm-charts/splunk-otel-collector/values.yaml
2a4c183 
Co-authored-by: jvoravong <47871238+jvoravong@users.noreply.github.com>
@wojtekzyla
fix: describe where persistent queue is currently supported
bb3588c
fix: Add persistent buffering for cluster receiver, agent and gateway
c76ed52
Merge remote-tracking branch 'origin/main' into persistent-queue
9fe3bb0
Fix pre-commit
c281b62
@VihasMakwana
Merge branch 'main' into persistent-queue-updated
81f9c57
Merge remote-tracking branch 'origin/main' into wojciech/persistent-q…
2f640b3 
…ueue-updated
Exclude persistent buffering for eks/fargate and gke/autopilot
2c81de2
Merge remote-tracking branch 'origin/main' into persistent-queue-updated
a330d6c
remove persistence queue for gateway
e79bbf4
@VihasMakwana VihasMakwana requested review from a team as code owners July 27, 2023 12:46
@atoulme
Copy link
Contributor

atoulme commented Jul 27, 2023

You did the work for logs and metrics. What about traces? Do they also need that support?

Would you please add an example with persistence enabled so we can see what that looks like?

@VihasMakwana VihasMakwana force-pushed the persistent-queue-updated branch from f30582e to f04cad1 Compare July 28, 2023 07:10
@VihasMakwana VihasMakwana force-pushed the persistent-queue-updated branch from f04cad1 to ab14b57 Compare July 28, 2023 07:12
@VihasMakwana @jvoravong
Update docs/advanced-configuration.md
825af68 
Co-authored-by: jvoravong <47871238+jvoravong@users.noreply.github.com>
@VihasMakwana VihasMakwana force-pushed the persistent-queue-updated branch from 5ff7489 to 4ba2577 Compare August 29, 2023 19:29
@VihasMakwana VihasMakwana force-pushed the persistent-queue-updated branch from 4ba2577 to 10c7488 Compare August 29, 2023 19:30
Vihas Splunk and others added 2 commits August 30, 2023 13:15
@dmitryax
Copy link
Contributor

dmitryax commented Sep 1, 2023

The PR LGTM overall. just couple nits

@VihasMakwana VihasMakwana force-pushed the persistent-queue-updated branch from 6786a54 to 231277a Compare September 2, 2023 12:49
@VihasMakwana VihasMakwana force-pushed the persistent-queue-updated branch from 875e36b to 90b802d Compare September 3, 2023 13:13
@VihasMakwana
Copy link
Contributor Author

@dmitryax I added functional tests covering persistent storage, I guess it's helpful.

@jvoravong jvoravong merged commit d0e8bd9 into signalfx:main Sep 6, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@jvoravong jvoravong jvoravong approved these changes

@jinja2 jinja2 jinja2 left review comments

@rmfitzpatrick rmfitzpatrick rmfitzpatrick left review comments

@dmitryax dmitryax dmitryax approved these changes

@atoulme atoulme Awaiting requested review from atoulme

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
7 participants
@VihasMakwana @atoulme @dmitryax @rmfitzpatrick @jinja2 @jvoravong @wojtekzyla