Skip to content

SolarQuery Stream API

Jump to bottom Edit New page
Matt Magoffin edited this page May 19, 2024 · 13 revisions

The SolarQuery Stream API provides methods to query the data collected by SolarNodes, in an efficient datum stream format. The methods in this API do not limit the number of returned results by default, so your client must be prepared to handle potentially large results.

Response formats

The SolarQuery Stream API supports different response data formats, which are based on the Accept HTTP header used in the request. The supported Accept header values are:

  • application/cbor - CBOR
  • application/json - JSON
  • text/csv - CSV

In addition the responses can be compressed with the gzip compressor by adding an Accept-Encoding: gzip HTTP header to the request.

Endpoints

Verb Endpoint Description
GET /datum/stream/datum Query raw or aggregate data by time, node, source, or stream.
GET /datum/stream/reading Query for data at specific dates or the difference between two dates.

JavaScript SDK

The solarnetwork-api-core package provides support for working with the datum stream metadata and datum stream datum returned by the SolarQuery Stream API endpoints. For example:

Datum stream datum list

This method returns datum stream values, either raw data or aggregated if the aggregation parameter is provided.

This method supports paged query parameters.

GET /solarquery/api/v1/sec/datum/stream/datum
localStartDate An inclusive starting date to use in the query, in YYYY-MM-dd or YYYY-MM-ddTHH:mm patterns. The date will be interpreted in the time zone of the node associated with each datum.
localEndDate An exclusive ending date to use in the query, in YYYY-MM-dd or YYYY-MM-ddTHH:mm patterns. The date will be interpreted in the time zone of the node associated with each datum.
startDate An inclusive starting date to use in the query, in YYYY-MM-dd or YYYY-MM-ddTHH:mm patterns. The date must be provided in the UTC time zone.
endDate An exclusive ending date to use in the query, in YYYY-MM-dd or YYYY-MM-ddTHH:mm patterns. The date must be provided in the UTC time zone.
mostRecent If true then no date range is required, and the most recently posted datum for all matching nodes/sources will be returned. This can also be combined with the aggregation parameter to return the most recent calculated aggregate values instead of raw datum (only the Hour, Day, and Month types are supported). Date parameters may be included to restrict the results to the most recently available datum within the given date range.
aggregation An optional aggregation type. If not provided, then non-aggregated data will be returned. See the SolarNet aggregation guide for more info. ⚠️ Note the *Minute types enforce a maximum date range of 5 weeks.
partialAggregation An optional aggregation type for partial aggregation results to be included. See the partial aggregation guide for more info.
nodeId The ID of a single node.
nodeIds A comma-delimited list of node IDs. This parameter may be also provided multiple times.
sourceId The ID of a single source to restrict the results to. A wildcard pattern can be used to restrict the results to only source IDs that match the pattern. Also, read more about how source IDs resolve with security policy restrictions.
sourceIds A comma-delimited list of source IDs to restrict the results to. This parameter may also be provided multiple times.
streamId The ID of a single stream to restrict the results to.
streamIds A comma-delimited list of stream IDs to restrict the results to. This parameter may also be provided multiple times.
combiningType An optional combining type to combine raw results into virtual ones. Specifying a type here requires some combintation of nodeIdMaps and sourceIdMaps parameters also be provided. See Virtual IDs for more information.
nodeIdMaps Return combined node IDs as virtual node IDs using the specified combiningType. The format of this parameter is VID:IDLIST where VID is a virtual node ID (integer) and IDLIST is a comma-delimited list of real node IDs. Multiple virtual IDs can be assigned by providing multiple nodeIdMaps query parameters. See Virtual IDs for more information.
sourceIdMaps Return combined source IDs as virtual source IDs using the specified combiningType. The format of this parameter is VID:IDLIST where VID is a virtual source ID and IDLIST is a comma-delimited list of real source IDs. Multiple virtual IDs can be assigned by providing multiple sourceIdMaps query parameters. This parameter requires the withoutTotalResultsCount also be set to true. See Virtual IDs for more information.

Notes:

  • Use either the local*Date or *Date parameters; do not mix the two styles. For example, providing localStartDate and endDate parameters is not supported.

Datum stream datum list sorting

Datum stream list queries support the following sort query parameter keys:

  • created
  • node
  • source
  • stream

The default sort order, if none provided, is (stream, created), both in ascending order.

💡 Sorting can be an expensive operation and slow your query down. The best-performing sort is by stream then created.

Datum stream datum list response

The response contains a general response object with the following properties:

Property Type Description
meta object An array of datum stream metadata, one element for every stream returned in the result data.
data array An array of datum stream datum arrays for each datum matching the request criteria.

An example response object for a non-aggregate query looks like this (see datum stream datum for more detailed information):

{
  "success" : true,
  "meta": [
    {
      "streamId": "7714f762-2361-4ec2-98ab-7e96807b32a6",
      "zone": "Pacific/Auckland",
      "kind": "n",
      "objectId": 123,
      "sourceId": "/power/1",
      "i": [
        "watts",
        "current",
        "voltage",
        "frequency"
      ],
      "a": [
        "wattHours"
      ]
    }
  ],
  "data": [
    [
      0,
      1650667326308,
      12326,
      null,
      230.19719,
      50.19501,
      6472722
    ]
  ]
}

That same example as CSV looks like this:

ts,streamId,objectId,sourceID,watts,current,voltage,frequency,wattHours,tags
2022-04-22T22:42:06.308Z,7714f762-2361-4ec2-98ab-7e96807b32a6,123,/power/1,12326,,230.19719,50.19501,6472722,

An example JSON response object for an aggregate query looks like this (see datum stream aggregate for more detailed information):

{
  "success" : true,
  "meta": [
    {
      "streamId": "7714f762-2361-4ec2-98ab-7e96807b32a6",
      "zone": "Pacific/Auckland",
      "kind": "n",
      "objectId": 123,
      "sourceId": "/power/1",
      "i": [
        "watts",
        "current",
        "voltage",
        "frequency"
      ],
      "a": [
        "wattHours"
      ]
    }
  ],
  "data": [
    [
      0,
      [
        1650667326308,
        null
      ],
      [
        12326,
        600,
        8290,
        14222
      ],
      null,
      [
        230.19719,
        600,
        228.2922,
        233.12324
      ],
      [
        50.19501,
        600,
        49.94322,
        50.20012
      ],
      [
        6472722,
        2819093834849,
        2819100307571
      ]
    ]
  ]
}

That same example as CSV looks like this:

ts_start,ts_end,streamId,objectId,sourceID,watts,watts_count,watts_min,watts_max,current,current_count,current_min,current_max,voltage,voltage_count,voltage_min,voltage_max,frequency,frequency_count,frequency_min,frequency_max,wattHours,wattHours_start,wattHours_end,tags
2022-04-22T22:42:06.308Z,,7714f762-2361-4ec2-98ab-7e96807b32a6,123,/power/1,12326,600,8290,14222,,,,,230.19719,600,228.2922,233.12324,50.19501,600,49.94322,50.20012,6472722,2819093834849,2819100307571,

Datum stream reading list

This method returns datum stream reading values, either raw data or aggregated if the aggregation parameter is provided.

This method supports paged query parameters.

GET /solarquery/api/v1/sec/datum/stream/reading
readingType The type of reading to perform. See Datum reading types for possible values.
localStartDate An inclusive starting date to use in the query, in YYYY-MM-dd or YYYY-MM-ddTHH:mm patterns. The date will be interpreted in the time zone of the node associated with each datum.
localEndDate An exclusive ending date to use in the query, in YYYY-MM-dd or YYYY-MM-ddTHH:mm patterns. The date will be interpreted in the time zone of the node associated with each datum.
startDate An inclusive starting date to use in the query, in YYYY-MM-dd or YYYY-MM-ddTHH:mm patterns. The date must be provided in the UTC time zone.
endDate An exclusive ending date to use in the query, in YYYY-MM-dd or YYYY-MM-ddTHH:mm patterns. The date must be provided in the UTC time zone.
aggregation An optional aggregation type. If provided, then individual aggregate results covering the time range between the start/end dates will be returned, instead of just one result for the overall range. Only Hour, Day, and Month values are supported. Only the Difference reading type is supported. See the SolarNet aggregation guide for more info.
nodeId The ID of a single node.
nodeIds A comma-delimited list of node IDs. This parameter may be also provided multiple times.
sourceId The ID of a single source to restrict the results to. A wildcard pattern can be used to restrict the results to only source IDs that match the pattern. Also, read more about how source IDs resolve with security policy restrictions.
sourceIds A comma-delimited list of source IDs to restrict the results to. This parameter may also be provided multiple times.
streamId The ID of a single stream to restrict the results to.
streamIds A comma-delimited list of stream IDs to restrict the results to. This parameter may also be provided multiple times.
tolerance An optional time duration to restrict how far in time the query is allowed to look for matching raw datum. This duration is applied to both the start and end dates. The syntax is ISO 8601 and defaults to P1M (one month). Generally the shorter the duration, the faster the query can execute.

Notes:

  • Use either the local*Date or *Date parameters; do not mix the two styles. For example, providing localStartDate and endDate parameters is not supported.
  • When aggregation is provided, results for the specified aggregation are returned. For example, if the date range is specified as 2000-01-01 to 2001-01-01 and Month aggregation is used, 12 results will be returned per stream, one for each month. Each month's results shows the reading values for just that month.
  • If no datum exists before a given start date, either because there simply isn't any data or there isn't any data within the tolerance range of the start date, the queries will include the first available datum within the given start/end dates. This can happen if datum are posted infrequently or at the very start of a datum stream, for example when querying for datum in the month the stream started and the stream starts in the middle of the month.

Datum stream reading list sorting

Datum stream reading queries support the following sort query parameter keys:

  • created
  • node
  • source
  • stream

The default sort order, if none provided, is (stream, created), both in ascending order.

💡 Sorting can be an expensive operation and slow your query down. The best-performing sort is by stream then created.

Datum stream reading list response

The response contains a general response object with the following properties:

Property Type Description
meta object An array of datum stream metadata, one element for every stream returned in the result data.
data array An array of either datum stream datum or datum stream aggregate arrays for each datum matching the request criteria.

The CalculatedAt reading type will return datum stream datum while all other types will return datum stream aggregate results.

An example response object looks like this (see datum stream aggregate for more detailed information):

{
  "success": true,
  "meta": [
    {
      "streamId": "5514f762-2361-4ec2-98ab-7e96807b3255",
      "zone": "America/New_York",
      "kind": "n",
      "objectId": 123,
      "sourceId": "/pyrometer/1",
      "i": [
        "irradiance",
        "temperature"
      ],
      "a": [
        "irradianceHours"
      ]
    }
  ],
  "data": [
    [
      0,
      [
        1650945600000,
        1651032000000
      ],
      [
        3.6,
        2,
        0,
        7.2
      ],
      [
        19.1,
        2,
        18.1,
        20.1
      ],
      [
        1.422802,
        1138.446687,
        1139.869489
      ]
    ]
  ]
}

That same example as CSV looks like this:

ts_start,ts_end,streamId,objectId,sourceID,irradiance,irradiance_count,irradiance_min,irradiance_max,temperature,temperature_count,temperature_min,temperature_max,irradianceHours,irradianceHours_start,irradianceHours_end,tags
2022-04-26T04:00:00Z,2022-04-27T04:00:00Z,7714f762-2361-4ec2-98ab-7e96807b32a6,123,/pyrometer/1,3.6,2, 0,7.2,19.1,2,18.1,20.1, 1.422802,1138.446687, 1139.869489,
Add a custom footer
Clone this wiki locally