-
Notifications
You must be signed in to change notification settings - Fork 2
All node metadata re-design #1105
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Comments
My vote is to keep the design as is with Edit: might change my mind pending the CRUD decision. |
Partially addressed in #1107 |
Also considered just using struct MetadataT
int::Dict{Symbol, Vector{Int}}
float::Dict{Symbol, Vector{Float64}}
string::Dict{Symbol, Vector{String}}
bool::Dict{Symbol, Vector{Bool}}
end |
SmallDataTypes good, that is equivalent to just using existing JSONable content. That also allows nesting of dicts (which we want). I'm against hard top level definition such as Dict{String, String/Any}. Way to restrictive or loose. Just stay with JSON, the whole GQL universe is simply JSON. |
Yes, idea of SmallDataTypes is to force JSONable content.
Can you elaborate on this please? We cannot currently nest dicts, metadata in essence is just key value pairs. The value itself can be a json serialized dict as a string. Do you agree on metadata as just key value pairs to help keep it small as anything bigger should be stored in a blob? |
Does metadata have CRUD in the SDKs? Currently add/update/delete would have to: get, add/update/delete, set. That makes Isolation not possible, so we would have to change the structure and implement it in the backend for proper CRUD. Using |
No this this the same thing that keeps coming back. NEVER put a JSON.write inside a value of a key=>value pair. That is the whole point. Pure JSON includes the idea of nested dicts.
No, that is up to the user. Keeping size of content in metadata small is not forced, that is simply convention.
Yes
What do you mean by isolation. Do you mean cannot guarantee atomic updates? metadata in SDKs need CRUD, no way around that I think.
yes, agree |
"Isolation" in ACID
I've been using metadata without CRUD as "get" -> modify -> "set", so not needed. |
The current design is that smallData/metadata can't be nested, they are just key-value pairs. |
Packing metadata as a vector of key-value pairs can avoid the whole nested JSON string issue. |
Okay, then I have been referencing SmallDataTypes incorrectly in previous comments. Ignore reference to SmallDataTypes and lets simplify the discussion to "Nested JSON" vs "key=>value JSON only".
Okay, new name for atomic+globalized operation order. I have been saying atomic instead of isolation. I'll restrict my use of From #1118
Two risks when using "key-value JSON pattern" with random data (incl already serialized JSON/JSONstring buried in the data):
I think it is fine to have one condition on One layer of nesting is already included upon transmission or persistence:
Think at this point we should talk more on the phone about this, but if Neo4j does not support nested JSON then it is starting to become a pretty poor choice. If that forces us to into short-term-long-term tandem plans then we should consider it.
maybe as a workaround only, but feels like that would result in lots of wasted effort overall. Side question: this sounds like
do you mean CRUD at driver/DB level vs CRUD at SDK level? I've been talking at SDK level. |
Just copying link from duplicate 1118 - This should be just a JSON dict, not a JSON3.write ::String:
|
This is currently still the case and smalldata == metadata as we should absolutely not be storing blobs in metadata for performance reasons. The definition from the dictionary is still valid on "store a couple of bytes". Smalldata: Pollutes the graph, but a convenient way to store a couple of bytes directly in the graph. Use this with caution and keep it small and simple |
I don't make a distinction between SDK and db, as a user I expect update to be ACID everywhere. DFG currently has update, but it is only valid in GraphsDFG as a reference to a variable is returned on In case it's not clear:
|
Decisions from call on Apr 8 2025:
Note: we can look into expanding support in SmallDataTypes in the future. One level is a small change in DFG by adding a |
Graphql has 2 options:
With the decision to support CRUD, the julia options giving the current limitations:
To support CRUD, key-value is the best option with current db. With value either:
These 2 choices basically comes down to: |
Hi @Affie, which option are we going with?
|
Hi @dehann, I wanted to get your input on which you prefer as well before we proceed, I'm still leaning towards |
Okay so decision is we work towards Shall we separate the time pressures by using blobless entries for CRUD in the meantime? (see other discussion E&T 187) |
Crud on variable metadata is already supported in dfg, the rest should follow over the next few minor releases to standardize to only one design. In the mean time we can use blobless entries as you suggest or the get-modify-set pattern |
right, think we can move this out of the Active Discussions column? |
To decide
Metadata Structure
Look into:
JSON3.Object
fast enough forCompute_Node_
level?Reminder SmallDataTypes are basically JSON values excluding null and objects (nesting):
Serialization
TAC:
The text was updated successfully, but these errors were encountered: