Ditto history: Getting events only from specific feature

[ondrej-charvat]

Hi everyone,
I'd like to use the history feature to show a log of changes that occurred for a particular thing feature. I thought it should be possible to "query the changes made to a single property value of a thing" as mentioned in the original idea but I'm not sure how to achieve this.

I was following docs for history feature but didn't find anything about limiting what events to get (maybe this option was not implemented ?). I did some tryouts (Ditto 3.5.10) and so far the results are:

1. Over HTTP API, I'm able to retrieve a specific feature at a specific revision using http://<host>:<port>/api/2/things/<thingId/features/<featureId>
2. Over SSE I'm not able to get events only from a specific feature. I managed to filter out what features I'm getting in the event by adding parameter ?from-historical-revision=193160&fields=thingId,policyId,features/State,_revision,_modified but this way I'm still getting all the events and just the ones without changes to the State feature are coming without any feature (I see this is expected behavior).
3. Over Ditto protocol (sent through WebSockets) I'm neither able to limit what events I'm getting. I've tried to use path as it's usual when retrieving the current state of a single feature but this seems to have no effect as I'm still getting all the stored events:

{
  "topic": "<thingId>/things/twin/streaming/subscribeForPersistedEvents",
  "path": "/features/State",
  "headers": {},
  "value": {
    "fromHistoricalRevision": 193160,
    "toHistoricalRevision": 193166
  }
}

{
  "topic": "<thingId>/things/twin/streaming/request",
  "path": "/features/State",
  "headers": {},
  "value": {
    "subscriptionId": "4",
    "demand": 25
  }
}

Am I doing something wrong or is it currently simply not possible to limit what history events will be streamed?

By the way, for our use case It would be best to be able to limit also what part of the thing history is being stored (e.g. only subset of the features) to limit DB volume (we need to persist the events for 1 month). But if I get it right the main raison d`etre of the event journal is something internal to Ditto and not providing the events over the API. So the limitation (if implemented) would harm the Ditto functionality, right?

Thank you in advance for any hints/replies.

[thjaeckle]

Hi @ondrej-charvat

Thanks a lot for formulating your question so well 👍

What you describe should be possible with the added #1583 (search in history).
So you should be able to e.g. open a SSE:

http://<host>:<port>/api/2/things/<thingId>?filter=exists(features/State)

And with this only get events/changes back in which the features/State was modified.

Opening a SSE for a specific feature like you described is not yet supported - only the "Thing level" currently is. Same goes for the WebSocket - which should behave the same as SSE.

By the way, for our use case It would be best to be able to limit also what part of the thing history is being stored (e.g. only subset of the features) to limit DB volume (we need to persist the events for 1 month).

I am afraid that this is not possible with Ditto's "Event Sourcing" approach.
What the Ditto "History" API provides, is basically only exposing what the "Pekko Persistence" (Event Sourcing library) functionality.
For having e.g. only the history of single properties, I would recommend including a timeseries database which you update based on Ditto events of e.g. a single feature property.

[ondrej-charvat]

Hello @thjaeckle,

thank you for the quick answer. That is exactly what I need! Now I realized I was reading documentation for Ditto v3.4 where this filtering was not yet present. My bad.

Yes, the timeseries DB is our second option. We'll record some of the events to AWS Timestream anyway as a long term archive but we wanted to evaluate Ditto history as well for a quick and easy access to data over Ditto API which we're using already.