Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Kafka JSON Spec Does Propagates DH Nulls Instead of JSON Null #5701

Closed
nbauernfeind opened this issue Jun 30, 2024 · 0 comments · Fixed by #5702
Closed

Kafka JSON Spec Does Propagates DH Nulls Instead of JSON Null #5701

nbauernfeind opened this issue Jun 30, 2024 · 0 comments · Fixed by #5702
Assignees
Labels
bug Something isn't working core Core development tasks json kafka kafka-exhaust release blocker A bug/behavior that puts is below the "good enough" threshold to release.
Milestone

Comments

@nbauernfeind
Copy link
Member

nbauernfeind commented Jun 30, 2024

To reproduce:

import random
from deephaven import kafka_producer as pk

source_group = time_table("PT00:00:00.1").update(
    formulas=["X = (int) random.randint(1, 5)", "Y = X == 3 ? null : i"]
)

write_topic_group = pk.produce(
    source_group,
    {"bootstrap.servers": "redpanda:9092"},
    "time-topic_group",
    pk.json_spec(["X"]),
    pk.json_spec(
        [
            "X",
            "Y",
        ]
    ),
    True,
)

This produces deephaven null values to the stream instead of real null values; see redpanda snapshot below:

Screenshot 2024-06-30 at 11 11 42 AM

(You will have to ignore that X == "3", Y == non-null instance in my image; that was because I [accidentally] had more than one producer running on the same topic at the time of the shot.)

@nbauernfeind nbauernfeind added bug Something isn't working core Core development tasks kafka release blocker A bug/behavior that puts is below the "good enough" threshold to release. kafka-exhaust json labels Jun 30, 2024
@rcaudy rcaudy added this to the June 2024 milestone Jun 30, 2024
devinrsmith added a commit to devinrsmith/deephaven-core that referenced this issue Jul 1, 2024
This fixes what was likely a copy-paste error, where the Kafka JSON producing logic checked for NULL_SHORT for int fields. None of the other field types were mishandled. It's worth noting that the json producer will omit null fields by default, and to explicitly output a JSON null (`{..., "myIntField": null, ...}`), the `boolean outputNulls` must be set to `true` (exposed as `output_nulls` in python).

Fixes deephaven#5701
devinrsmith added a commit to devinrsmith/deephaven-core that referenced this issue Jul 18, 2024
This fixes what was likely a copy-paste error, where the Kafka JSON
producing logic checked for `NULL_SHORT` for int fields. None of the
other field types were mishandled. It's worth noting that the json
producer will omit null fields by default, and to explicitly output a
JSON null (`{..., "myIntField": null, ...}`), the `boolean outputNulls`
must be set to `true` (exposed as `output_nulls` in python).

Fixes deephaven#5701
devinrsmith added a commit that referenced this issue Jul 18, 2024
This fixes what was likely a copy-paste error, where the Kafka JSON
producing logic checked for `NULL_SHORT` for int fields. None of the
other field types were mishandled. It's worth noting that the json
producer will omit null fields by default, and to explicitly output a
JSON null (`{..., "myIntField": null, ...}`), the `boolean outputNulls`
must be set to `true` (exposed as `output_nulls` in python).

Fixes #5701
Cherry-pick of #5702
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working core Core development tasks json kafka kafka-exhaust release blocker A bug/behavior that puts is below the "good enough" threshold to release.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants