SolarNet aggregation

SolarNet provides several API methods that return aggregated results derived from the raw datum posted by SolarNodes over fixed time windows, such as days and months. This guide describes how SolarNet performs that aggregation.

Instantaneous vs. accumulating vs. status properties

SolarNode datum are collected as general datum sample objects, which have their properties grouped into these classifications:

Classification	Description	Example
instantaneous	An absolute measurement of some number.	A reading from a speedometer in a car, for example 50 km/h.
accumulating	A relative measurement of some number that accumulates value over time.	A reading from an odometer in a car, for example 28,189 km.
status	Some value, not necessarily a number.	A warning message from a car, for example Check engine: E290.

SolarNet treats these classifications in the following ways when aggregating datum into time windows:

Classification	Aggregation
instantaneous	average of samples within time window
accumulating	sum of differences over time, projected across time window boundaries
status	most frequently seen value

Time windows are equal in length to the aggregation level being queried, and include everything on and after the start date up to but not including the end date. You might think of that logic like StartDate <= window < EndDate or (window >= StartDate) AND (window < EndDate).

List aggregation

The /datum/list endpoint allocates accumulating data using time projection across time window boundaries, so that only the amount within the window is allocated to the final aggregate value. For example, imagine calculating the accumulated aggregate for a time window starting at 10:00 given two readings like this:

Date	Energy (Wh)
2000-01-01 09:59	200
2000-01-01 10:02	500

SolarNet would report 200 Wh for the 10:00 window. That is because there are 3 minutes between the two readings, and a difference of 300 Wh. If we project that 300 Wh evenly over those 3 minutes, only 2 of those minutes fall within our 10:00 time window, so SolarNet would allocate 2/3 of the 300 Wh to the window, which is 200 Wh.

Example list aggregation

Imagine we have a SolarNode collecting data from a car that goes on a trip. The collected data looks like this:

Date	Speed (km/h)	Odometer (km)	Message
2000-01-01 10:00	0	53456
2000-01-01 10:15	50	53460
2000-01-01 10:30	70	53466
2000-01-01 10:45	30	53470
2000-01-01 11:00	20	53472
2000-01-01 11:15	90	53478	Check oil
2000-01-01 11:30	10	53485	Check oil
2000-01-01 11:45	5	53486	Check oil
2000-01-01 12:00	0	53486	Check oil
2000-01-01 12:15	0	53487
2000-01-01 12:30	60	53490
2000-01-01 12:45	95	53498
2000-01-01 13:00	2	53504

SolarNet would aggregate this into hourly time windows that look like this:

Date	Speed (km/h)	Odometer (km)	Message
2000-01-01 10:00	37.5	16
2000-01-01 11:00	31.25	14	Check oil
2000-01-01 12:00	38.75	18

Here the Speed column represents the average speed and Odometer represents the distance travelled, over each hour.

In a similar fashion, SolarNet would aggregate this into a single daily time window that would look like this:

Date	Speed (km/h)	Odometer (km)	Message
2000-01-01 00:00	35.83	48	Check oil

List partial aggregation

The /datum/list endpoint supports including partial aggregated results for time ranges that do not align with the requested aggregation level. This feature only works in conjunction with aggregate queries, so the aggregation and partialAggregation query parameters must be both provided.

⚠️ NOTE: the start/end date ranges will be implicitly treated as node-local dates, that is without any time zone component.

An example scenario where this can be used is when you want to query for monthly results between two mid-month dates, like 15 Jan 2020 - 15 Mar 2020. By specifying the query parameters like this:

startDate=2020-01-15&endDate=2020-03-15&aggregation=Month&partialAggregation=Day

The results will include 3 result time ranges:

Result #	Start Date	End Date
1	2020-01-15	2020-01-31
2	2020-02-01	2020-02-29
3	2020-03-01	2020-03-14

If the partialAggregation parameter had not been provided, the results would include only full month results like this:

Result #	Start Date	End Date
1	2020-02-01	2020-02-29
2	2020-03-01	2020-03-31

Supported partial aggregation types

Both the aggregation and partialAggregation parameters accept aggregation type values, but partialAggregation must be a smaller aggregate (shorter time range) than the aggregation value, and only Month, Day, and Hour values are supported, as outlined here:

Aggregation	Supported Partial Aggregations
Year	Month, Day
Month	Day, Hour
Day	Hour

Reading aggregation

The /datum/reading method can be used to query the difference of accumulating property values collected by SolarNodes without any time projection applied. SolarNet will apply the Difference reading type logic to each aggregate time window. In the previous example aggregation, the end result would be the same because the datums all fall on exact minute dates. Real data collected by SolarNodes will not have such nicely convenient dates, and the normal aggregation uses time based projection to allocate portions of accumulation fairly between time window boundaries. The CalcualtedAtDifference reading type uses this same approach, but reading aggregation uses the Difference approach, which does not use time based projection. Instead it looks for the reading closest to but before the start/end boundaries of the time window.

Here's an example to illustrate the difference. Let's calculate the 10:00 hourly aggregate value for this data:

Date	Energy (Wh)
2000-01-01 09:59	200
2000-01-01 10:02	500
2000-01-01 10:11	900
2000-01-01 10:19	1600
2000-01-01 10:55	2900
2000-01-01 11:01	3500

Here's what SolarNet would return, for both /datum/list and /datum/reading style aggregations:

Style	Result	Calculation
/list	3100	((500-200) * (2/3)) + (900-500) + (1600-900) + (2900-1600) + ((3500 - 2900) * (5/6))
/reading	2700	(2900 - 200)

Note the two cases of time projection involved in the /list result: (500-200) * (2/3) and (3500 - 2900) * (5/6).

Time window tolerance

When SolarNet aggregates for a given time window, using either the /list or /reading style, it must search for the previous and/or next datum samples in order to either perform time projection or simply find the starting/ending readings. In order to find those values efficiently, SolarNet imposes a maximum tolerance limitation on how far in time it will look for the necessary datum. The tolerance amount differs based on the aggregation style:

Style	Tolerance
/list	1 hour
/reading	3 months

Essentially, if there is a gap in the data larger than this tolerance, then SolarNet will allocate no accumulation for that period. Here's an example:

Date	Energy (Wh)
2000-01-01 10:55	2900
2000-01-01 12:01	6500
2000-01-01 12:03	6600

Notice the gap of 66 minutes between the 10:55 and 12:01 samples. Here's what SolarNet would return when aggregating the 12:00 hourly aggregate:

Style	Result	Calculation
/list	100	(6600 - 6500)
/reading	3700	(6600 - 2900)

Notice how for the /list result there is no time projection included like (6500 - 2900) * (1/66), because the amount of time is larger than the maximum tolerance of 1 hour. The /reading result has a much larger tolerance, and so does use the 10:55 sample as the starting value for this window.