feat: Display JSON columns in anywidget mode #2138
Open
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
product-auto-label
bot
added
size: s
|
Check out this pull request on See visual diffs & provide feedback on Jupyter Notebooks. Powered by ReviewNB |
product-auto-label
bot
added
size: m
tswast
reviewed
Oct 6, 2025
tswast
requested changes
Oct 6, 2025
shuoweil
force-pushed
the
shuowei-anywidget-col
branch
2 times, most recently
from
43a938c to
4a33ccf
Compare
tswast
reviewed
Oct 15, 2025
tswast
reviewed
Oct 15, 2025
tswast
reviewed
Oct 15, 2025
shuoweil
force-pushed
the
shuowei-anywidget-col
branch
from
30dfa7d to
237c134
Compare
tswast
reviewed
Oct 15, 2025
Comment on lines
786
to
788
| # anywdiget mode uses the same display logic as the "deferred" mode | ||
| # for faster execution | ||
| if opts.repr_mode in ("deferred", "anywidget"): |
shuoweil
force-pushed
the
shuowei-anywidget-col
branch
from
f18fd9e to
8601e52
Compare
product-auto-label
bot
added
size: l
shuoweil
force-pushed
the
shuowei-anywidget-col
branch
from
533541a to
860db3e
Compare
product-auto-label
bot
added
size: m
shuoweil
changed the title
Previously, SQL queries in anywidget mode would fall back to deferred execution, showing a dry run instead of an interactive table. This change modifies the display logic to directly use the anywidget interactive display for SQL queries, providing a more consistent and responsive user experience. A test case has been added to verify this behavior.
Adds a test case to verify that a DataFrame with a STRUCT column is correctly displayed in anywidget mode. This test confirms that displaying a STRUCT column does not raise an exception that would trigger the fallback to the deferred representation. It mocks `IPython.display.display` to capture the `TableWidget` instance and asserts that the rendered HTML contains the expected string representation of the STRUCT data.
This reverts commit 153e1d2.
This commit fixes an AttributeError that occurred when displaying a DataFrame with a JSON column in anywidget mode. The dtype check was incorrect and has been updated. Additionally, the SQL compilation for casting JSON to string has been corrected to use TO_JSON_STRING.
shuoweil
force-pushed
the
shuowei-anywidget-col
branch
from
36e58ba to
48ba1a0
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Previously, SQL queries in anywidget mode would fall back to deferred execution, showing a dry run instead of an interactive table.
This change enables the display of DataFrames containing JSON columns when using the anywidget representation mode.
Due to a limitation in Apache Arrow (#45262), JSON columns are not natively supported by the to_pandas_batches() method, which is used by the anywidget backend. This change implements a workaround by automatically converting JSON columns to strings before rendering the table in anywidget. A warning is issued to inform users of this temporary conversion.
Key changes include:
(1) Updated DataFrame.repr_html() to detect JSON columns using bigframes.dtypes.contains_db_dtypes_json_dtype() and apply a conversion to string when in anywidget mode using _apply_unary_op(ops.json_ops.ToJSONString()).
Simplified JSON-to-string casting in scalar_op_registry.py by unconditionally using to_json_string(x) for JSON-to-string conversions, replacing the previous conditional logic that distinguished between safe and unsafe casting modes.
(2) Updated TableWidget.init() to detect and convert JSON columns to strings before processing, ensuring compatibility with the Arrow-based to_pandas_batches() method.
(3) Added comprehensive system tests including test_json_column_anywidget_mode(), test_struct_column_anywidget_mode(), and test_sql_anywidget_mode() to verify the display of JSON and STRUCT columns in anywidget mode.
(4) Updated anywidget_mode.ipynb notebook with an example demonstrating the display of generative AI results from AI.GENERATE, which returns JSON columns, showing the automatic conversion to strings for display.
(5) Removed test test_json_astype_others_raise_error[to_string] that expected JSON-to-string conversions to raise errors in strict mode, as this behavior is no longer supported with the simplified casting logic.
Fixes #<448126500 and 453561268> 🦕