Some food-for-thought for supporting Key-Value API, this shows we can… #143
Conversation
viktorklang
added
enhancement
| final KVEntity.Builder builder = KVEntity.newBuilder(); | ||
| unparsed.forEach( | ||
| (k, v) -> { | ||
| builder.putEntries(k, com.google.protobuf.ByteString.copyFrom(v.asByteBuffer())); |
There was a problem hiding this comment.
@viktorklang
This is the serialization for the user choosen for his values in the KVS, right?
Does this conflict or is it compatible with the stable serialization rules by Cloudstate? Is this relevant here too? I think about how, if done, a separate/different user lib, language wise, would parse the values back is they're are marshalled as ByteStrings here.
There was a problem hiding this comment.
Yeah, this needs work. I am not sure how to integrate it though—I'd need to go through AnySupport I think.
There was a problem hiding this comment.
Could be a good example how to use AnySupport.
There was a problem hiding this comment.
Indeed. I'll have a look at what that could look like.
|
@viktorklang <https://github.com/viktorklang>
This is the serialization for the user choosen for his values in the KVS,
right?
Does this conflict or is it compatible with the stable serialization
rules by Cloudstate
<https://cloudstate.io/docs/user/lang/java/serialization.html>? Is this
relevant here too? I think about how, if done, a separate/different user
lib, language wise, would parse the values back is they're are marshalled
as ByteStrings here.
It’s just food-for-thought, but you’re right—for this to be correct we’d
have to use the same serialization scheme as for the rest. I just did it
like this for illustrative purposes 🙂
--
Cheers,
√
|
@viktorklang no worries. I just wanted to be sure I got the idea right. cool stuff. |
|
Thought: Since this is more of a pattern than a distinct different storage mode, should we make this a documentation feature only? |
In #144 @jroper argues, that the user should not have to know that any of the CRDTs he uses is a CRDT, by using one. Similarly a user also should not have to know a "Cloudstate KVS" he uses is an event sourced entity underneath? |
… implement it as a pure lang-support alternative.
viktorklang
force-pushed
the
wip-kv-api-√
branch
from
93b60df
to
3cc52f5
Compare
| // We'll most likely want to add a specific CommandContext for KeyValueEntity | ||
| // so you'd instead of state().set(…) would do ctx.setState(…), and then it'd be automatically | ||
| // flushing ctx.emit(state.toProtoModification()) if there are any changes | ||
| @CommandHandler |
There was a problem hiding this comment.
@jroper What do you think about the comments above, James? Do you think this is something we could put together rather quickly?
There was a problem hiding this comment.
@jroper ping :)
There was a problem hiding this comment.
@viktorklang Any progress here? This seems to be stopped for a long time
There was a problem hiding this comment.
@sleipnir No, unfortunately not. I guess the question is: Should we go for an API-only approach or support KV natively in the proxy as a different way of storing data.
There was a problem hiding this comment.
The CRUD support will provide the underlying native KV support discussed in this thread, and @ralphlaude is making progress there. (Side note: not sure if "CRUD" entities is the best name, maybe we can improve the naming there too.) Until that's complete, language supports could also experiment with including a setState or setValue style API directly on event sourced entities, setting a snapshot value, if we want to try this out earlier.
Also agree that it would be good to try out some use cases, to see how useful a KV API like in this PR might be, since this would need to be added to all the language supports. This currently gives a more generic Map-like API in place of defining protobuf message objects directly (which can also be map-like as needed). The shopping cart samples also use a map for state — would that example be improved with a generic key-value API, compared with the types it uses now for event sourcing? In these kinds of examples, would just setting a map as the value (and using CRUD support) already be clean enough?
Entity services themselves provide a KV-style store, keyed by entity id, and can be backed by event sourced, or the upcoming CRUD for a simple set-value approach, or using CRDTs for distributed, in-memory, eventually consistent storage. Within an entity the state can be a map (be key-value-based) like in this PR.
And then there's the concept of a KV store that's available across Entity or Action instances — like the Cloudflare Workers KV, which is an eventually-consistent LWW (last writer wins) key-value store distributed across the worker nodes. For Cloudstate, the pattern that we've demonstrated that's similar to this is having multiple entity types. Like the "hot items" CRDT entity used alongside the shopping cart. We have forwarding and side effects to combine entities. Maybe we should look at supporting an "ask pattern", an async get on another entity type, where values can be fetched from other entity types and then continue processing with the responses.
There was a problem hiding this comment.
@jroper and I discussed this further. Some things we discussed:
We don't currently see clear use cases for a key-value API like this in the language supports. Especially when compared with simply using an object or map as the value (with CRUD or event sourcing support). For persisting deltas/changes only, there's the event sourcing already, and having an additional KV API increases the surface area that language supports need to implement.
For CRUD support naming, we're suggesting to refer to this as a value-based state model, and it could be the default state model for entities. From the user perspective, we could simply call these Entities. And then there are Event Sourced Entities and Replicated Entities (CRDTs). Within code or discussions, we could use value entities or value-based entities or persisted value entities as needed. And for services, entities themselves provide key-value storage (with the entity key and its value), which can be backed by simple key-value storage, event sourcing, or in-memory replication.
I think the main thing to do here is to complete the basic entities with value-based state model (CRUD support). In terms of roadmap, we also discussed other forward-looking things, like replicated and active-active event-sourcing, durable actions (as in long-lasting, that can fetch asynchronously, implement workflows), durable CRDTs, and product CRDTs.
So suggesting that we close this PR and put the idea of KV APIs in language supports on hold, and focus on finishing the value entities (CRUD support) and making those the default starting point. Interested to hear more thoughts on this, and it can be a topic for the next contributors' call.
There was a problem hiding this comment.
@pvlugter thanks for summing everything up. It is a clear roadmap and the proposed names are fine for me.
There was a problem hiding this comment.
I agree that this is the best path forward.
There was a problem hiding this comment.
Having a good "CRUD" solution might obviate the need for a KV solution, especially if the proxy can diff.
|
Closing this, as discussed. |
… implement it as a pure lang-support alternative.
Related to #51