-
Notifications
You must be signed in to change notification settings - Fork 1.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Exit non-zero when Vector fails to gracefully shutdown #13731
Comments
Hi @Sh4d1 ! Can you share the full log output and full Vector configuration? I think I reproduced this, but the behavior is just that the sink continuously tries to send the data, which is the correct behavior since the error might recover (in my case DNS resolution). Vector doesn't shut down, but it would if it had consumed all of the input. If Vector does start to shut down, then the sink will eventually time out shutting down and exit as you saw. I could see having a different exit code for hard shut downs though. For systemd, you can configure it to always restart even if it exited 0 with |
Indeed I ended up changing the For the config it's the same than #13718 (the first block)
Yes that's my case too (well some timeout but same behaviour I guess) |
Aha, I see. That's interesting that you managed to hit a panic too, which seems to be related to reload. I think there are two issues here:
I'll repurpose this issue for the second point: that Vector could exit non-zero when it fails to gracefully shutdown. If you are able to reproduce that panic, would you mind opening it as a separate issue? |
Ah yeah I forgot about the panic, yeah it totally not related, apparently it panicked with (on a filter transform):
I have some logs:
I was trying to make the previous filter work, I'm not sure which change cause the panicked though, however now I have to:
in order to to get those errors (not sure if I should need it, but yeah different issue). I'll see if I can reproduce but the panick but now I don't have any timeout issues so not sure it'll work 😅 |
Hi Folks: Just wanted to know if there is any targeted release for fixing this issue. Thanks. |
👍 |
Hey! No not yet unfortunately. Happy to accept a PR in this area though. |
…shutdown times out (#17676) <!-- **Your PR title must conform to the conventional commit spec!** <type>(<scope>)!: <description> * `type` = chore, enhancement, feat, fix, docs * `!` = OPTIONAL: signals a breaking change * `scope` = Optional when `type` is "chore" or "docs", available scopes https://github.com/vectordotdev/vector/blob/master/.github/semantic.yml#L20 * `description` = short description of the change Examples: * enhancement(file source): Add `sort` option to sort discovered files * feat(new source): Initial `statsd` source * fix(file source): Fix a bug discovering new files * chore(external docs): Clarify `batch_size` option --> Issue: [Exit non-zero when Vector fails to gracefully shutdown](#13731)
Closed by #17676 |
A note for the community
Problem
Hello, when I reload vector, and there is an issue with a sink (
http
in my case), this happens:It looks like it exits with a exit code 0, which then stop systemd from restarting Vector.
I'll take a look when I have some time, but if somebody has an idea before, go for it 😄 (Can't really run with RUST_BACKTRACE atm)
Configuration
Version
0.23.0
Debug Output
No response
Example Data
No response
Additional Context
No response
References
No response
The text was updated successfully, but these errors were encountered: