Skip to content

Pluggable Message Routing

Jump to bottom Edit New page
Mike Blackstock edited this page Apr 30, 2018 · 20 revisions

Moved to Node-RED wiki: https://github.com/node-red/node-red/wiki/Pluggable-Message-Routing Please make changes and comments there.

Trello: https://trello.com/c/J7UDbQVP/66-pluggable-message-routing

The mechanism by which messages are passed from one node to the next should be a pluggable component of the runtime. This would enable, for example, a flow that spans multiple runtime instances. Other use cases:

  • flow debugger
  • adding custom low-level logging of node send/receive events including the full message data

Expanding on multiple runtime instances:

  • instances running on separate cores routed manually or based on a policy or algorithm.
  • runtimes on separate machines in a cluster
  • runtimes in a fog deployment, e.g. on devices, gateways, cloud routed dynamically depending on the mobility of a device, associated connectivity, location, etc..

Thinking about at least two approaches. The first requires the user to manually specify, and possibly configure the router to use on a given wire, the next is about providing a runtime component that handles the delivery of some or all messages between nodes, closer to the intent of this epic I think.

Note: The method for managing and distributing instances running in different cores or machines is outside the scope of this document. This document discusses how to 'tap' messages sent between wire-connected nodes in flows.

1. Manual routing using wire nodes

Note: This is really the same as creating a new node, and dropping it between wires.

In this idea, Routers are a special type of node that are associated with wires. They always have one input and one output.

They are not completely hidden in the UI, but do not appear in the palette, and do not appear as nodes on the canvas. Instead they have a UI on top of a wire such as the name and status. The routers available could also appear in the info when you click on any wire. (Currently wire info is not used.)

The default routing is 'none', meaning send directly to connected node.

To change a wire routing:

  1. User double clicks on a wire.
  2. Wire configuration dialog appears. There we can choose what type of routing to use depending on router nodes installed, e.g. MQTT broker, IPC, TCP/UDP in a pop up.
  3. User chooses 'MQTT router' for example.
  4. The configuration for the MQTT wire routing node appears.
  5. User configures router, configures associated configuration node if needed, leveraging existing node configurations.
  6. User hits OK.
  7. Wire node is added between source and destination node on the wire by the runtime. Wire is somehow annotated on the canvas indicating there is a router node associated with the wire, e.g name hovering over the wire as shown.
  8. User hits deploy.
  9. Flow works as usual. Router node implementation handles communications with outside systems or protocols.

Wire nodes could be shown as a dot on a wire (for example). When the user hovers, the node name and status could be shown.

2. Routing using a pluggable routing module

The first idea doesn't handle the case where the type of routing to choose between specific nodes in a flow is not configured by the user beforehand. Another approach is needed to handle the use cases where we want to tap all messages between nodes or need to make decisions on how to route messages between nodes at runtime.

To do this type of message routing, we need a way to tap any message between nodes, and an (optional) way for the user to configure the current routing system.

Runtime changes

An interface to a module that can intercept messages sent by any node is required. An implementation can then decide what to do with the message, e.g. log, forward to an external system or protocol.

Considerations

  • There are potentially two points to tap: the send or receive node methods.
  • There are a number of (potential) optimizations for local routing that should be maintained.

From scanning the code, it looks like the send side is the best option since this allows a routing implementation to optimize how messages are sent between nodes on different or the same systems as is done in the local node-node case.

The default implementation would be the same as it is now, i.e. call node.receive() on downstream nodes (with current message cloning and other optimizations).

The minimal interface for the pluggable routing module could look something like this:

function init(_runtime)

Initialise the router, connecting to external systems as needed to make routing decisions, deliver messages.

function send(Node source, Object msg)

Send message to downstream nodes in the port/wire lists of the source node.

This simple interface would mean that a router implementation takes responsibility for sending all messages. There is no way for a router to tap only specific messages or wires, leaving some to be delivered by the default local routing implementation, or to log or copy messages sent between wires before they are delivered locally.

To address this we need a way for the router to indicate which wires it has taken responsibility for, and which wires can should be handled by the default local implementation.

This could be done in an all-or-nothing manner, returning true or false if the router handles message delivery, or on a per-wire basis. The router could return a subset of the source port/wire lists containing the wires that have not been handled by the router plug in. The default local routing could then handle delivery along the ports/wires not handled by the plug in.

A method signature with this capability could look something like this:

function send(Node source, Object msg): [[String]]

or

function send(nodeId, [[String]] wires, Object msg): [[String]]

A noop implementation would simply do nothing on initialization and return the source wire list on send().

There could be a single router installed at any one time. Perhaps a composite router could be developed to chain routers if desired, e.g. a router to log messages that returns the full port/wire list, followed by a router that handles delivery of some messages to different cores, followed by the default local router implementation.

Open Issues/Questions:

  • performance and optimizations may be more difficult or limited. Code currently optimizes for no wires, single port & message, and anticipates pre-fetching downstream nodes. Not sure what is possible.
  • should we first query the router for the revised wire list, then send the message? This might allow the local implementation to optimize, and allow local messages to be sent before routed messages.
  • how will this work with extension to APIs needed to provide message delivery guarantees (ref)? Does the router provide a guarantee that it has sent the message to all wires it has handled? Is there a promise or callback?

User interface

A router implementation could supply a user interface to configure itself using a node implementation in the same package. This node could extend the node-red user interface by adding (for example) an additional 'Router' side panel as the Dashboard and Debug nodes do. If needed, this panel could be used to communicate with an endpoint added by the node for configuration.

Router configuration storage.

A router can use any way it wants to store its configuration, but the router module will have access to the runtime in case it wants to store and retrieve configuration in a flow.

Module runtime configuration

To configure node-red to use a router, the router module would be set in the settings, in a similar way to a storage module.

Discussion

Discussion from slack and elsewhere - these may be edited!

**dceejay** - another approach could be (rather than start to give wires properties) - for nodes to declare a “location” where location could be physical, or abstract, and the deploy process would then take care of interpreting what it means to go from location A (localhost) to B(mqtt://foo.bar.com:1883) to C(http://moo.com:8080) . If two connected nodes are in the same location then the existing node to node comms would be the default etc.

mike - Yes. I was thinking a router module (idea #2) could use node properties on each side of the wire to decide when and how to route. I will write that down. Good use of system wide node properties also on the board

dceejay - Yes indeed - in the general case it is “just” a system wide property for each node - that code be instantiated config node style (ie not in palette per se) - and itself have config nodes to config sub-properties.

mike - Hmm. I see. The pluggable router code could be instantiated on some or all sending nodes. If it exists, node send() could delegate to that.

knolleary - The intention of the item is pluggable message routing that is - at its core - invisible to the user so its very much more your #2 rather than #1

dceejay - (I think the original thought came more from the ability to be able to replace the internal messaging (for debug / trace etc) - and grew out to other transports - which then of course intersects with the distributed Node-RED flows thoughts - so in my mind there is quite a bit of overlap)

so both debates need to happen… … at some point.

Add a custom footer
Add a custom sidebar
Clone this wiki locally