RFC: vsock / virtio-serial support for agent and collector

[jodh-intel]

Requirement - what kind of business use case are you trying to solve?

We are adopting Jaeger for Kata Containers. Kata Containers integrates with container manager systems like Docker and k8s and allows a workload to run transparently (to the container manager) inside a highly optimised virtual machine (VM).

Architectural overview image:

• https://katacontainers.io/media/uploads/katacontainers/images/data.jpg

As shown in that diagram, Kata is comprised of a number of components. We've already added basic Jaeger support to our runtime component but are trying to find the best way to do the same for our agent process (kata-agent), which runs inside the VM we create to host the workload.

Further information:

• https://github.com/kata-containers/documentation/blob/master/architecture.md

Problem - what in Jaeger blocks you from solving the requirement?

We don't want to run a Jaeger agent inside the VM so are looking for way to allow spans to be reported from the kata-agent inside the VM, "out" through the VM to the Jaeger agent running in the host context.

The two main options here are virtio-serial and VSOCK. See:

• Adding OpenTracing support for Kata kata-containers/kata-containers#27 (comment)

Proposal - what do you suggest to solve the problem or improve the existing situation?

Unless there is a simple solution that we've missed (?), we're interested in helping to add VM support in the form of VSOCK and/or virtio-serial support to Jaeger:

• Allow the Jaeger client library (https://godoc.org/github.com/uber/jaeger-client-go atleast) to be configured to use VSOCK or virtio-vsock rather than UDP.

• Allow the Jaeger agent to be configured to act as a VSOCK server (or be able to connect to the serial device on the host corresponding to the virtio-serial socket in the VM) so it can receive the spans from the kata-agent running inside the VM.

I think the VSOCK route would potentially be cleaner, but it does presuppose a 4.8 host kernel, hence the possibility that both VSOCK and virtio-serial support might be required to cater for all environments.

Any open questions to address

n/a at this stage.

---

/cc @devimc.

[black-adder]

Hi there,

I'm concerned that adding support for these new senders will add extra bloat to the client with new dependencies that will have only one user. If we could somehow vendor the new dependencies so that they're only pulled in for the kata use case, I'll be all for it. However, given how the client is written, that maybe difficult to do. I'll take a look and see if it's possible.

In the ticket you referenced, it looks like there were 2 other options available, are those out of the question?

[jpkrohling]

I know the Go client was specifically mentioned, but the Java client does support loading custom senders dynamically, via Service Loaders.

[devimc]

Hi @black-adder

please take a look kata-containers/kata-containers#27 (comment)

[vprithvi]

Is using TCP an option? The go client supports transporting traces over HTTP to collectors.

[jodh-intel]

@black-adder - As @devimc has pointed to, the other two options on kata-containers/kata-containers#27 are out of the question I'm afraid as they are not secure enough.

However, given how the client is written, that maybe difficult to do. I'll take a look and see if it's possible.

Thanks!

@vprithvi - given the above, no, unfortunately we cannot use TCP.

@jpkrohling - good to know, but our stack is entirely based on golang.

[jodh-intel]

I've started to have a look at how this might work given the requirement to avoid linking the jaeger client package to the vsock package. I'm still not very familiar with the codebases, but here's what I've found out so far (warning: there may be mistakes herein ;-)...

Generic tasks

• Create a new TVSockTransport Thrift implementation for VSOCK. This would implement thrift.TTransport.

• Put the implementation in a new package (to avoid binding it into Jaeger)

Jaeger Agent

• Add a new "--transport-plugin /path/to/plugin.so" CLI option (that could be used for "vsock-thrift-plugin.so").

• Each plugin would implement something like:

  type TransportPlugin interface {
  
      // GetTransport returns the name of the transport and a Thrift Transport implementation
      GetTransport(config map[string]interface{}) (string, thrift.TTransport, error)
  }
  

• Create a map to hold all the available transports:

  type transportFunc func(config map[string]interface{}) (thrift.TTransport, error)
  
  var transportMap = map[string]transportFunc{
    // TODO: function will need its parameter changed
    "udp" : NewTUDPServerTransport,
  }
  

• Load --transport-plugin plugin and append entries to the transportMap map.

• Rename ServerConfiguration.getUDPServer() to ServerConfiguration.getServer(transportName string) and find the appropriate transport (defaulting to the existing "udp" one).

Jaeger client

• Again, to ensure vsock is decoupled from jaeger, we'd need a plugin that implements something like:

  type TransportPlugin interface {
  
      // GetTransport returns the name of the transport and a Jaeger Transport implementation
      GetTransport(config map[string]interface{}) (string, jaeger.Transport, error)
  }
  

• Add support for "JAEGER_TRANSPORT_PLUGIN=/path/to/plugin.so" environment variable.

• Modify ReporterConfig.newTransport() to load and use the plugin if variable set, but default to calling the existing jaeger.NewUDPTransport().

[yurishkuro]

@jodh-intel fyi #422 and #927 are sort of pre-requisites to what you're suggesting with the pkg/plugin.

[yurishkuro]

This seems highly unlikely to be supported, so closing.