Tail Workers

[marcusdarmstrong]

Hi there;

As I've been prototyping what it would look like to move our workload onto workerd I've been working through the observability story. I've managed to implement a workable observability layer (collecting prometheus-style metrics/logs and pushing them over to a service tasked with being the collection point) but it feels to me like a bit of a hack, given that tail workers are clearly the intended mechanism here and obviously fit a lot more cleanly.

Poking around the code, it looks like top-level support for tail workers has been pretty explicitly removed from the public version of workerd... Is that the intended situation long term, or is there a roadmap here toward expressing tail workers in the workerd.capnp schema?

[kentonv]

This is an oversight. I believe all the implementation code exists in workerd to enable tail workers, but it wasn't ever wired up at the top level, as you discovered in your link.

Probably we should extend struct Worker in the config with tail :ServiceDesignator, which causes tail events to be delivered to the given service. I think this can be implemented entirely in server.c++ without too much work.

One weird thing: Technically a tail event can include multiple events in a batch. The intent of this was supposed to be that if you have several workers operating in a "pipeline" making requests to each other, you'd get one tail event batch for each incoming request from the network, and it would include all of the worker invocations that resulted from that one incoming request, transitively. So if the worker receiving the request made a subrequest to another worker (in the same workerd instance), you'd get both invocation reports in a single batch.

However, I think in practice we don't ever deliver batches with more than one event in production? Like, if a tailed worker invokes a service binding, I don't think we actually include the other worker's invocation in the tail batch... and we probably shouldn't, since the service binding could belong to someone else. I wonder if this "batch" functionality should be made available in workerd at all, or if maybe we should remove it entirely.

cc @irvinebroque and @mrbbot

[marcusdarmstrong]

I gave this a shot, as you described (without further consideration for batching—in the OSS env I think collecting subrequests into a single TailEvent makes sense, but I never really got to a point where I needed to think about it), but couldn't quite get it working... It was easy enough to add the configuration mechanism and wire the tail events to the tail worker, but I was unable to get the tail worker to actually "wake up" and ingest the events that it was provided. I suspect that's due to a very poor understanding on my part of the waitUntil lifetimes involved and what the intended configuration of them is to be. My intuition was that both the PipelineTracer's onComplete and the TraceCustomEventImpl should be added/bound to the parent WorkerService's waitUntilTasks, but that 1. didn't work 2. perhaps conflicts with the commentary in IoContext surrounding getWaitUntilTasks.

I also tried various mechanisms to extend the IoContext appropriately, but those failed rather violently as I was clearly doing so incorrectly.

Some other notes:

• Stack traces, even with llvm-symbolizer (macos, arm) are ?? at ??:0:0—I've had success with the symbolizer in a linux/container env with the NPM builds of workerd, so I'm assuming there's some bazel invocation secret I'm missing.
• The CustomEvent::type seems ill-defined in the OSS release. For my attempt I've used 6 and that doesn't appear to be problematic, but presumably there's some specification external to the repository for a correct value here?
• Am I correct in believing that tail worker dispatch is not supposed to be a subrequest? That was the only way I could make sense of the flow, since presumably the PipelineTracer needs to be able to complete prior to making it to the tail worker, but that's obviously a pretty different flow from other worker-to-worker interactions in server.c++.
• I was surprised to see my code wasn't running through the RPC codepath given it was specifically called out in the IoContext::getWaitUntilTasks commentary, but I guess that makes sense given it's all in-process. Given the configuration, I don't really see how another case could exist, so additional context on the codepaths here would be helpful if there is any.