Vector panicked at 'async fn resumed after completion'

[xdatcloud]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Vector Version

vector 0.18.1 (x86_64-apple-darwin c4adb60 2021-11-30)

Vector Configuration File

Origin Configuration

data_dir = "/tmp"

[api]
enabled = true
address = "127.0.0.1:8686"

[sources.demo_logs]
type = "demo_logs"
format = "shuffle"
lines = ["hello", "world"]

[sinks.demo_logs_to_loki]
type = "loki"
inputs = [
    "demo_logs",
]
endpoint = "<endpoint>"
encoding.codec = "json"
batch.max_events = 10000
tenant_id = "<tenant_id>"
request.rate_limit_duration_secs = 1
request.rate_limit_num = 500

[sinks.demo_logs_to_loki.auth]
user = "<user>"
password = "<password>"
strategy = "basic"

[sinks.demo_logs_to_loki.labels]
host = "{{ host }}"

Configuration After Change (In order to trigger the panic)

data_dir = "/tmp"

[api]
enabled = true
address = "127.0.0.1:8686"

[sources.demo_logs]
type = "demo_logs"
format = "shuffle"
lines = ["hello", "world"]

[sinks.demo_logs_to_loki]
type = "loki"
inputs = [
    "demo_logs",
]
endpoint = "<endpoint>"
encoding.codec = "json"
batch.max_events = 10000
tenant_id = "<tenant_id>"
request.rate_limit_duration_secs = 1
request.rate_limit_num = 400 # this line modified, indeed any changes in config file will probably trigger the panic

[sinks.demo_logs_to_loki.auth]
user = "<user>"
password = "<password>"
strategy = "basic"

[sinks.demo_logs_to_loki.labels]
host = "{{ host }}"

Debug Output

https://gist.github.com/xdatcloud/9af88c2652a283a693b942bf8d99f930

Expected Behavior

Vector keep working.

Actual Behavior

Vector exited due to the panic.

Example Data

Additional Context

Run vector with --watch-config enabled:

RUST_BACKTRACE=full vector -vvv -w true -c panic-test-config.toml

Please try to modify config file several times and the panic will probably occurred.

References

[xdatcloud]

I wonder this issue is related to vector_core::stream::driver. I ran the test above with vector built under debug mode, and the backtrace shows 'async fn resumed after completion' probably occurred at: https://gist.github.com/xdatcloud/6424ef6be706b505490fa2754e6626ea#file-vector-output-log-L122-L127

[xdatcloud]

Please leave any information about this issue, I would like to make a hotfix on my local at this weekend, thanks!

[jszwedko]

Hi @xdatcloud ! Thanks for filing this! We think #10490 will fix it. Would you want to give that a try?

[xdatcloud]

Hi @xdatcloud ! Thanks for filing this! We think #10490 will fix it. Would you want to give that a try?

Great! Thanks for this quick fix! I will make a hot patch on my local branch 😄