Resolve "external values" in Things via search and connections

[thjaeckle]

The idea of this issue is to have a special kind of JSON values inside a Ditto Thing (e.g. as attribute or as feature property) which are not directly available in Ditto's twin persistence, but which are resolved by Ditto from an "external source" (like another API, e.g. an HTTP endpoint, or a Kafka topic).

Such an "external value" would have a special json format in the Thing.

Some ideas of what could be "externally resolved":

• perform Ditto searches
  • e.g. for relationships: all "Devices" belonging to a "Room" (also managed in Ditto) refer to the room's thingId in an attribute attributes/roomThingId - the Room however is not directly aware of its devices (as this would be a bidirectional relationship and therefore e.g. difficult to update consistently)
    • the "Room" Thing could simply have an "externally resolved" attribute with a Ditto search query like e.g. filter=eq(attributes/roomThingId,{{ thing:id }}),fields=thingId
    • placeholders could be used in order to e.g. use the room's data dynamically (see above)
    • a fields selector could be used in order to define certain fields of the search results to resolve as the "external value"
• perform Ditto aggregations, e.g. calculating the average temperature of room
  • currently, Ditto's search cannot yet perform aggregations - that's however an idea to do in Provide "aggregation" functions in Ditto search #1525
  • again, in the "Room" example, the room could define an "external value" resolving the room's current average temperature which is resolved via a search aggregation over all devices of that room reporting a temperature
• invoke foreign APIs, resolving external values
  • a "Digital Twin" is often used to also orchestrate other APIs, e.g. providing a lower level API for the device data managed by another system (like e.g. a LoRa Server or a mobile network provider)
  • to have a "single point of truth", data from other APIs often want to also be mirrored to the twin (and e.g. automatically updated as well)

Some ideas of when "external values" are resolved:

• on access: whenever an "external value" is retrieved via Ditto, it could first be resolved "from externally"
  • this strategy could also define a caching strategy of how often to do the external lookup (which can potentially be costly to look up or has a big latency)
  • this would block the retrieve until the value was retrieved externally
• scheduled: define eg cron syntax when an external value should be resolved

Some ideas of what to do with resolved "external values":

• the twin could be updated with the external value
  • to also be findable via Ditto search for example
  • or as part of a caching strategy
  • or to have the value available in the Thing's history API
• the resolved value could just be returned back, without persisting/caching it
  • to e.g. always externally resolve the value
  • if the result is very large, it maybe is not wanted to persist it in the twin

To get an idea how this could look like, this is a first rough idea of the format:

{
  "thingId": "foo:my-room-1",
  "policyId": "foo:my-room-policy-1",
  "attributes": {
    "roomDevices": {
      "_ditto-external-value-config": { // TODO define a good "reserved keyword"
        "source": "connection:<the-connection-id>/<the-target-address>",
        "updateStrategy": {
          "type": "cron",
          "config": "<the cron config>"
        },
        "responseStrategy": "updateTwin" // "updateTwin" | "dispatchOnly"
      }
      "value": ... // this is the Json Value where the retrieved external Json value (e.g. Json array or object or string) is stored
    } 
  }
}

And here an example for an internal Ditto search:

{
  "thingId": "foo:my-room-1",
  "policyId": "foo:my-room-policy-1",
  "attributes": {
    "roomDevices": {
      "_ditto-external-value-config": { // TODO define a good "reserved keyword"
        "source": "search?filter=eq(attributes/roomThingId,{{ thing:id }}),fields=thingId",
        "updateStrategy": {
          "type": "onAccess", // would update the value when e.g. `/attributes/roomDevices/value` is retrieved
          "config": "always" // also retrieve, when complete thing or all attributes are retrieved, alternative could be "exact"
        },
        "responseStrategy": "updateTwin" // "updateTwin" | "dispatchOnly"
      }
      "value": [
        {"thingId": "foo:my-device-1"},
        {"thingId": "foo:my-device-2"}
      ]
    } 
  }
}

To clarify:

• could the "ditto-external-value-config" also be a metadata (not even shown in the Thing JSON by default)?
  • that way, the "end user" of the thing would not even be aware of the "external value" and just use the twin's data like any other data, which would be great
  • the json structure would not have to follow a certain structure, but could be completely defined by the user
  • the stuff inside the "ditto-external-value-config" would not be indexed in the twin's values in search and would not be part of the full thing json
  • downside: metadata does not yet provide concepts for access control, see Allow access to metadata to be restricted via policies #1403
    • so every subject with write permission would be able to define/change the external value mapping
• the Ditto "things" service does not know of "connectivity" or "search" service .. how to solve where to address such a "external value" lookup message to?
  • maybe the configuration of things could define some templates (eg for search and connections) - configuring messages to send via pub/sub and the topic to use for publishing

The "scheduled" variant could also supersede issue #1542 - the existing "HTTP-push" connection type would be sufficient to also solve what #1542 was aiming for - the cron/schedule would be defined in the things which would want to poll data.
And great benefit would be that this then works for all connection types, not only http.