HTTP 1.1 Connectivity API - Source (aka Polling)

[jonaswre]

Discussed in #1541

^{Originally posted by jonaswre December 3, 2022}
I have read the whole documentation twice.

Maybe I've missed something obvious, but in many scenarios its necessary to poll devices/gateways/services for data.

Is there any build in way to achieve this?
I would prefer not have to run my own service that polls http endpoints and sends the result to a message broker.

Thanks alot.

[thjaeckle]

I think that is a good idea to use HTTP connection "sources" for configuring polling of HTTP APIs.

In order to do that, a "Consumer" for HTTP connection has to be created and started in HttpPushClientActor (currently, only a HttpPublisherActor is started there).
How to create and start the "Consumer" role can be looked at from the other connection types. It e.g. involves creating a new class extending the abstract BaseConsumerActor class.

Regarding the scheduling you asked in #1541: you can use Akka's scheduler to schedule repeating tasks (like the polling).
The polling interval should be configurable via the connection configuration (done in class HttpPushSpecificConfig).

The use of Akka's scheduler could look similar to (used in an Akka actor):

Object message = ....;
Cancellable cancellable = getContext().getSystem().scheduler()
   .scheduleAtFixedRate(initialDelayFromConfig, intervalFromConfig, getSelf(),
        message, context.dispatcher(), ActorRef.noSender())

This will send the message each intervalFromConfig to getSelf() (the own actor) - this message should trigger the polling of the HTTP endpoint.

[thjaeckle]

Thinking about this issue I wondered how the polling would come to its thingIds to work on.
So you would e.g. define:

• each 30 minutes, poll an HTTP endpoint via GET
  • but what if the HTTP endpoint already includes a "device id" or a "thing id"? where does that come from?

For other connection sources which are always reacting on an "outside" message, there are several ways to get a thingId:

• as part of protocol headers
• as part of e.g. protocol topic (e.g. for MQTT, Kafka)
• as part of the received payload

When a source now is triggered by a cron-job/polling, those information sources do not exist, so where do they come from? Ideas:

• one connection per thingId to poll for
  • does not scale very well - only usable for "small use cases"
• static configuration in the connection of a list of thing IDs
  • not very dynamic :/
• result of an internal search query, determining thingIds ... (e.g. searching for all devices of type "gateway" in Ditto, e.g. identified by an attribute)
  • could internally done via "streaming" the found thingIds in order to not need paging
• call of another HTTP API returning the list of ids first (this could also be Ditto's own search API endpoint maybe?)
  • but what about paging?

I think I tend into determining the list of thingIds to poll an HTTP API for via either static configuration of a list or by using an internal search.
Do you have ideas on that @jonaswre ?

[thjaeckle]

This issue could be superseded by #1609