State Management

[MainRo]

Hello @dbrattli,
I started playing with state management and its propagation from the subscription up to the source observables.
Propagating the state store is straightforward since this is similar to what we do with the scheduler.

However, this is not sufficient to implement a state that could be saved and restored. We need additional topology information so that each observable/subscription is uniquely and deterministically identified. In Rxsci this is something that I added with a combination of topology discovery (for each operator) and key management (for subscription instances). Currently, we have none of them in RxPY.

I looked quickly at Reaqtor and I do not understand how this is managed.
I see that each operator manages its own data internally and exposes load/save methods that are called by a visitor. But I do not see how each operator subscription is uniquely identified.

You can see my initial experiment here:
https://github.com/MainRo/RxPY/tree/feature/state-store

It exposes a state store API where the states is be directly managed. This is similar to implementations like Kafka and Faust, different from the save/load approach of the Reaqtor implementation. I am not sure what approach would be better for RxPY, but both need some topology information.

Do you have any clues on how Reaqtor handles this?

[dbrattli]

If I remember things correctly then every operator has a name property. Thus you can build a unique name for an operator while traversing the subscription tree e.g op1:op2:op3:op4. So for an operator that needs to save state, then you can use that unique name for saving (and for validating that the pipeline looks the same when restoring)

[dbrattli]

But it could perhaps make sense to also add e.g an optional name argument to stateful operators so the developer can override the topo generated name (as I mentioned above). Sometimes you want the state to survive for e.g a scan operator even if the topo changes. PS: I think your branch looks really interesting. This will be awesome!! I btw implemented something very similar in my previous job so would be great to get similar features in RxPY.

[MainRo]

yes, I agree that adding a name parameter would be a nice additon, even if it does allow updating the topology only in some specific cases.
I will look deeper at the reaqtor code base to better understand how it's done.