RFE: Polling interface

[FGasper]

As I look at the interface to embed a server, I only see custom functionality to connect() to the peer; beyond that, Dqlite expects to do all I/O.

As best I can tell this presents at least two significant barriers to adoption:

1. Encrypted client-server connections only seem possible with kernel TLS, or a VPN like WireGuard.
2. It’s not possible to embed Dqlite transmissions in some other transport, e.g., WebSocket.

It would be nice if Dqlite’s I/O were pluggable … has that been considered?

Thank you!

[cole-miller]

LXD, which is the primary client for dqlite right now, proxies dqlite node-to-node connections from a local Unix-domain socket over an HTTPS transport, using WebSockets. You can see how that's set up here (and the rest of that source file):

https://github.com/lxc/lxd/blob/5938f747f6460d74be89df151d851dbf31540796/lxd/cluster/gateway.go#L1136-L1146

Would something like that work for the application you have in mind? I suspect that "deeper" I/O pluggability would run into the problem of integrating with dqlite's use of the libuv event loop.

[FGasper]

At this point I don’t have a specific use case, but I’ve wanted something like this before, so I’m teasing out what it can & can’t do. (Also wrote a freestanding client library … have yet to publish, though.)

It sounds like the expectation is that peer-to-peer and client-server interactions happen via abstract-domain local sockets, which in turn get proxied out?

[cole-miller]

It sounds like the expectation is that peer-to-peer and client-server interactions happen via abstract-domain local sockets, which in turn get proxied out?

I'll leave it to the other maintainers to speak on our expectations for dqlite consumers -- just wanted to share an example of how one existing consumer accomplishes those two goals (security and transport substitution) using the existing API.

[freeekanayaka]

At this point I don’t have a specific use case, but I’ve wanted something like this before, so I’m teasing out what it can & can’t do. (Also wrote a freestanding client library … have yet to publish, though.)

It sounds like the expectation is that peer-to-peer and client-server interactions happen via abstract-domain local sockets, which in turn get proxied out?

It's not an expectation, since you can use plain TCP if that's good enough for you, but yes it's definitely the way to go if you need a custom transport (TLS, Websockets or whatever).

Ideally there should be native C-level support for at least TLS, but that hasn't been done yet. In the go-dqlite client/bindings support for TLS is available and implemented using a proxy, you can look at the code there to understand the details, or ask us more if it's not clear.

[FGasper]

Along this line—and I’m sorry if this isn’t the best forum to ask—what is the difference between the address given to dqlite_node_set_bind_address versus that given to dqlite_node_create? Both are described as client interfaces, but it seems the one given to dqlite_node_create is used for raft, which seems more like that would be a peer-to-peer interface … when would a client use that address to connect?

Incidentally: https://metacpan.org/pod/Protocol::Dqlite

[freeekanayaka]

Both are described as client interfaces, but it seems the one given to dqlite_node_create is used for raft, which seems more like that would be a peer-to-peer interface … when would a client use that address to connect?

This is a fine place to ask. We have a Discourse forum at dqlite.io, but it's not really used.

The address you pass to dqlite_node_create is the location of the node as seen by other nodes. In other words, other nodes will use that address when trying to connect to this node. The dqlite_node_set_bind_address is mainly used to set up proxies, for example for TLS: you pass a local abstract unix socket to dqlite_node_set_bind_address, which dqlite will bind, then some other part of your code (e.g. the go-dqlite package) will bind the original address that you passed to dqlite_node_create setting up TLS termination on it and proxying the traffic back and forth to the local abstract unix socket.

Hope that's clear.

@cole-miller @MathieuBordere this might be a spot were documentation/docstrings need improvement, if it's not clear enough.

[cole-miller]

@freeekanayaka added to my checklist in canonical/dqlite-docs#4

[FGasper]

@freeekanayaka I found the Discourse forum, but as you say, it seems little-used. (Maybe Github Discussions would fare better?)

If I’m understanding it correctly, it sounds like dqlite.h’s description of dqlite_node_create should omit the mention of clients? And dqlite_node_set_bind_address is apropos for either clients or peers across a proxy (e.g., TLS)?

Thank you!

[cole-miller]

Note that we also have an IRC channel, #dqlite on Libera.Chat. It's mostly the developers chatting about CI/CD failures right now, but we do monitor it and you're welcome to post questions there.

[freeekanayaka]

@freeekanayaka I found the Discourse forum, but as you say, it seems little-used. (Maybe Github Discussions would fare better?)

If I’m understanding it correctly, it sounds like dqlite.h’s description of dqlite_node_create should omit the mention of clients?

Mentioning clients is correct. This is the externally visible address both for clients and other nodes.

And dqlite_node_set_bind_address is apropos for either clients or peers across a proxy (e.g., TLS)?

The docstring here says:

 * Instruct the dqlite node to bind a network address when starting, and
 * listening for incoming client connections.

probably we should say just "connections" instead of "client connections", because the purpose is really proxying connections, of any type.

[FGasper]

OK, I think I’m getting it. Thank you!

One last question: can a dqlite cluster (and clients) use only proxied connections?

[freeekanayaka]

OK, I think I’m getting it. Thank you!

One last question: can a dqlite cluster (and clients) use only proxied connections?

Yes, that's exactly what happens in go-dqlite when you use TLS. The Go bindings for dqlite will start listening on the address passed to dqlite_node_create, using a TLS setup: all connections from clients and from other dqlite nodes will go through that endpoint. The Go bindings will also call dqlite_node_set_bind_address to configure the C dqlite engine to accept connections on a local abstract unix socket. Every time a connection is accepted by Go on the "official" TLS node endpoint, the go-dqlite code will internally connect to the local abstract unix socket and copy data back and forth between the two connections.

In that sense the dqlite cluster (and clients) use only proxied connections. I hope that's what you meant.

[freeekanayaka]

Ideally there should be native TLS support in dqlite, but that hasn't been implemented. That was the main motivation for the proxy support (TLS in Go is straightforward, while with libuv it requires some work).

[freeekanayaka]

Also, regardless of TLS, some applications (like LXD) want to have a single endpoint for both their REST API and for dqlite traffic. Using a proxy makes that possible. I think it's a bit of a unique requirement though, in most cases having 2 separate endpoints (one for dqlite traffic and one for the application API) should be fine.