SolarNode SolarFlux Protobuf Guide

This guide describes how SolarNode can be configured to use the SolarFlux Upload plugin to publish datum to SolarFlux (MQTT) as Protobuf messages, instead of using the default CBOR encoding.

Why would I want to do this?

Good question! The default CBOR encoding is simple to use. It requires no configuration and all datum properties are saved without any fuss. The main reasons to consider Protobuf for your use case are:

1. Conserving network bandwidth is essential, for example your node is deployed with an expensive mobile network connection.
2. You are integrating with a system that already uses Protobuf.

For point #1, consider the following example of how much network bandwidth Protobuf could save over the default CBOR encoding.

The OS Statistics SolarNode plugin can collect various operating system metrics. In its default configuration it collects a lot of metrics. Here's an example datum collected by the OS Statistics plugin, shown in JSON form:

{
  "created" : 1619587693010,
  "sourceId" : "OS Stats",
  "cpu_user" : 0,
  "cpu_system" : 0,
  "cpu_idle" : 1E+2,
  "fs_size_/" : 3664707584,
  "fs_used_/" : 896360448,
  "fs_used_percent_/" : 25,
  "fs_size_/run" : 484552704,
  "fs_used_/run" : 10645504,
  "fs_used_percent_/run" : 3,
  "ram_total" : 969105408,
  "ram_avail" : 5.4102016E+8,
  "ram_used_percent" : 44.2,
  "sys_load_1min" : 0.55,
  "sys_load_5min" : 1.2,
  "sys_load_15min" : 1.02,
  "cpu_temp" : 47.774,
  "net_bytes_in_eth0" : 0,
  "net_bytes_out_eth0" : 0,
  "net_packets_in_eth0" : 0,
  "net_packets_out_eth0" : 0,
  "sys_up" : 3289.57,
  "_v" : 2
}

When posted to SolarFlux, the CBOR serialization format is used by default. This results in a message payload of 452 bytes:

BF67 6372 6561 7465 641B 0000 0179 16F2 
F9D2 6873 6F75 7263 6549 6468 4F53 2053 
7461 7473 6863 7075 5F75 7365 72C4 8200 
006A 6370 755F 7379 7374 656D C482 0000 
6863 7075 5F69 646C 65C4 8202 0169 6673 
5F73 697A 655F 2FC4 8200 1B00 0000 00DA 
6F00 0069 6673 5F75 7365 645F 2FC4 8200 
1A35 6D60 0071 6673 5F75 7365 645F 7065 
7263 656E 745F 2FC4 8200 1819 6C66 735F 
7369 7A65 5F2F 7275 6EC4 8200 1A1C E1B0 
006C 6673 5F75 7365 645F 2F72 756E C482 
001A 00A2 7000 7466 735F 7573 6564 5F70 
6572 6365 6E74 5F2F 7275 6EC4 8200 0369 
7261 6D5F 746F 7461 6CC4 8200 1A39 C360 
0069 7261 6D5F 6176 6169 6CC4 8201 1A03 
3988 0070 7261 6D5F 7573 6564 5F70 6572 
6365 6E74 C482 2019 01BA 6D73 7973 5F6C 
6F61 645F 316D 696E C482 2118 376D 7379 
735F 6C6F 6164 5F35 6D69 6EC4 8220 0C6E 
7379 735F 6C6F 6164 5F31 356D 696E C482 
2118 6668 6370 755F 7465 6D70 C482 2219 
BA9E 716E 6574 5F62 7974 6573 5F69 6E5F 
6574 6830 C482 0000 726E 6574 5F62 7974 
6573 5F6F 7574 5F65 7468 30C4 8200 0073 
6E65 745F 7061 636B 6574 735F 696E 5F65 
7468 30C4 8200 0074 6E65 745F 7061 636B 
6574 735F 6F75 745F 6574 6830 C482 0000 
6673 7973 5F75 70C4 8221 1A00 0504 FD62 
5F76 02FF

Reduced CBOR message

We can use filters on the SolarFlux Upload plugin to remove properties that we don't need, which is a good way of reducing the message size. After defining a filter, a reduced message looks like the following, in JSON form:

{
  "cpu_user" : 0,
  "cpu_system" : 0,
  "cpu_idle" : 1E+2,
  "ram_used_percent" : 47.1,
  "sys_load_5min" : 0.42,
  "cpu_temp" : 46.698,
  "sys_up" : 3709.57,
  "_v" : 2
}

When posted to SolarFlux as CBOR this results in a message payload of 119 bytes (26.3% of the original size):

BF68 6370 755F 7573 6572 C482 0000 6A63 
7075 5F73 7973 7465 6DC4 8200 0068 6370 
755F 6964 6C65 C482 0201 7072 616D 5F75 
7365 645F 7065 7263 656E 74C4 8220 1901 
D76D 7379 735F 6C6F 6164 5F35 6D69 6EC4 
8221 1822 6863 7075 5F74 656D 70C4 8221 
1912 0866 7379 735F 7570 C482 211A 0005 
C07D 625F 7602 FF

OsStats Protobuf message

Here's a Protobuf definition for the same reduced set of properties:

syntax = "proto3";

message OsStats {
  float 	cpuUser 	= 1;
  float 	cpuSystem 	= 2;
  float 	cpuIdle		= 3;
  float 	ramUsed		= 5;
  float		loadFiveMin	= 6;
  float		cpuTemp		= 7;
  double	upTime		= 8;
}

After configuring the SolarFlux Upload plugin to encode this datum into the OsStats Protobuf message, then the data posted to SolarFlux is only 29 bytes (6.4% of the original size, or 24.4% of the reduced property CBOR size):

1D00 00C8 422D 6666 3C42 35CD CC4C 3E3D 
D7A3 3842 4185 EB51 B81E DBAE 40

Required Plugins

The following plugins are required for SolarNode to post Protobuf messages to SolarFlux:

1. The SolarFlux Upload plugin for posting datum MQTT messages.
2. The SolarNode Protobuf I/O plugin for mapping datum properties into Protobuf message fields, and for generating Java sources from .proto files via the protoc tool.
3. The Java Compiler Service — JDT plugin for compiling Protobuf Java sources generated by protoc into Java classes.

The protoc tool is required as well. On Debian-based operating systems — including SolarNodeOS — this can be installed via the protobuf-compiler package. For SolarNodeOS the solarnode-app-protobuf package can be used to install these plugins as well as protoc.

Configuration

First configure a Protobuf Compiler (protoc) component, which will handle compiling .proto files into Java classes. You'll need to configure a Service Name that can be referenced by other plugins. In SolarNodeOS the Path should work as-is. For other systems you should change the Path to match where protoc is available. The Service Name Default is shown here:

Next configure a Protobuf Datum Encoder/Decoder component, which will handle encoding datum into Protobuf messages. The following core settings must then be configured:

Setting	Description
Service Name	A unique name for this encoder, to be referenced by the SolarFlux Upload plugin. A good approach here is to use a name that describes the Protobuf message type. In this example we use OsStats Protobuf.
Protobuf Compiler	The Service Name of the Protobuf Compiler component configured previously. Default in our example here.
Message Class	The Java class of the Protobuf message to encode datum into. This will be derived from the .proto file(s) supplied. We use OsStats in our example here, based on the OsStats shown previously.

Use the Upload Files button to upload your .proto file. In this example we uploaded one file: os-stats.proto as shown previously.

Then add Property Configurations for each datum property you want to encode. Shown here are 7 property configurations for each of the fields defined in the OsStats message:

Finally, configure a SolarFlux Upload component. You must add a Filter configuration that references the Service Name of the encoder component we previously configured. That is OsStats Protobuf in our example here.

In this example we have not configured a Source ID in our filter. In a real deployment that will probably also need to be configured, because you'll usually need to be explicit about how different datum sources are encoded into different Protobuf messages. In that scenario you would configure multiple Protobuf Datum Encoder components, one per Protobuf message, and one corresponding Filter configuration for each encoder with the specific source ID to encode messages for.

Conclusion

From this point you should see your datum messages in SolarFlux encoded as Protobuf messages. 🎉