Improve Vector's throughput for UDP protocols

[jszwedko]

Currently we process UDP packets serially in the syslog source when mode udp is used:

https://github.com/timberio/vector/blob/410fac0b7fbb0c7361056105ba362f7ee6c112ca/src/sources/syslog.rs#L422-L444

It seems like we should be able to improve throughput introducing concurrency here given that processing the packet does involve some decoding and parsing work.

This is also true for the socket source:

vector/src/sources/socket/udp.rs

Line 92 in 801ee21

loop {

Perhaps one worker per "connection" similar to the tcp mode where we could use a source ip/port pair to partition them; or just by having a fixed (configurable?) number of workers processing packets.

[hhromic]

Perhaps one worker per "connection" similar to the tcp mode where we could use a source ip/port pair to partition them; or just by having a fixed (configurable?) number of workers processing packets.

The idea of using the (source ip/port) tuple for marking a "connection" or "session" is indeed common in many applications. However, from my experience in my company where we deal with a big number of observability data feeds, many of those systems send data using a single source ip/port that rarely rotates. So, parallelising by this tuple won't do much in these cases unfortuntely.

Mentioning that for your consideration on the design 👍

[davidpellcb]

We're interested in potentially replacing the dogstatsd portion of Datadog Agent with a Vector agent dedicated to receiving statsd applicatino metrics, due to performance issues we've had with dogstatsd under load. It would be nice to see if this provided better performance once optimized!