DH-17242: PartitionedTable display widget does not render properly when there are non-key, non-constituent columns

[rcaudy]

Description

When a PartitionedTable is created that has extra columns (not just key columns and the constituent columns), the display widget fails to render the partition filter widget or any table data. A box at the bottom of the component informs the user that it is "Reloading partition".

Steps to reproduce

You can reproduce with this script in a Groovy console:

tt = timeTable("PT00:00:01") // Make a TimeTable
it = tt.update("A=ii%2==0", "B=k%10") // Add some columns
pt = it.partitionBy("A", "B") // Partition by those columns
ptt = pt.table() // Grab the underlying table
pttu = ptt.update("C=A ? B : -B") // Derive a new column
pt2 = io.deephaven.engine.table.PartitionedTableFactory.of(pttu, List.of("A","B"), true, "__CONSTITUENT__", pt.constituentDefinition(), false) // Make a new PartitionedTable with the same key columns

Another way to reproduce is with a partitionedAggBy:

pt = it.partitionedAggBy([AggLast("Last=Timestamp")], false, null, "A", "B")

In both cases an extra column in the underlying table seems to be confusing the UI. This is an important, permitted use case for PartitionedTables.

Expected results

I expect a partition filtering widget to show as normal. Ideally, though, we'd have some better way to display this, since showing the extra columns of the partitioned table would be valuable. I'm not certain that PartitionedTable and PartitionAwareSourceTable should be displayed in the same way.

Additional details and attachments

DH-17242

Versions

• Engine Version: latest
• Web UI Version: ...
• Java Version: ...
• Barrage Version: ...
• OS: ...
• Browser: ...
• Docker: ...

[dsmmcken]

Ideally, though, we'd have some better way to display this, since showing the extra columns of the partitioned table would be valuable. I'm not certain that PartitionedTable and PartitionAwareSourceTable should be displayed in the same way.

Can you elaborate on what you are thinking?

[vbabich]

There's a ticket for differentiating the UI for PartitionedTable and PartitionAwareSourceTable - #2049

[rcaudy]

Ideally, though, we'd have some better way to display this, since showing the extra columns of the partitioned table would be valuable. I'm not certain that PartitionedTable and PartitionAwareSourceTable should be displayed in the same way.

Can you elaborate on what you are thinking?

If I were designing a "perfect" PartitionedTable widget, I might want the following:

1. The underlying table() displayed by default. Color-coding or some other visual indicator identifying the key columns and the constituent columns.
2. Key column selectors and a merge() button available, as today, for selecting a single constituent to display or merging the entire partitioned table.
3. The ability to select a constituent to display from the table, rather than requiring the user to fiddle with the dropdowns.
4. The ability to filter or sort the underlying table being displayed (this might call for more of the PartitionedTable Java API to be exposed over gRPC).

[dsmmcken]

Same as example in ticket just as python, except for one line I am not sure on the equivalent

from deephaven import agg
from deephaven import time_table

tt = time_table("PT00:00:01") # Make a TimeTable
it = tt.update(["A=ii%2==0", "B=k%10"]) # Add some columns
pt = it.partition_by(["A", "B"]) # Partition by those columns
ptt = pt.table # Grab the underlying table

pttu = ptt.update("C=A ? B : -B") # Derive a new column
# unsure how make this python
# pt2 = io.deephaven.engine.table.PartitionedTableFactory.of(pttu, List.of("A","B"), true, "__CONSTITUENT__", pt.constituentDefinition(), false) // Make a new PartitionedTable with the same key columns

# aggregate it
ptagg = it.partitioned_agg_by(
    aggs=[agg.last(cols=["Last = Timestamp"])], by=["A","B"]
)

[mofojed]

@rcaudy can we get some clarification on point 1?

The underlying table() displayed by default. Color-coding or some other visual indicator identifying the key columns and the constituent columns.

Is there another case that would demonstrate this a little bit better? In the example above, there are the two key columns, and then there is a column _CONSTITUENT_ which looks automatically generated... you want to be able to see that?

We'll need to have a JS API added for this at least, as there is no way to get the underlying table for a partitioned table at the moment.

[rcaudy]

@rcaudy can we get some clarification on point 1?

The underlying table() displayed by default. Color-coding or some other visual indicator identifying the key columns and the constituent columns.

Is there another case that would demonstrate this a little bit better? In the example above, there are the two key columns, and then there is a column _CONSTITUENT_ which looks automatically generated... you want to be able to see that?

We'll need to have a JS API added for this at least, as there is no way to get the underlying table for a partitioned table at the moment.

I had a fairly extensive meeting with @dsmmcken and @rbasralian on Friday afternoon, and I don't want to recap that here unless there's a need for clarification. If you look at the partitionedAggBy example, you'll see more about what I mean about the importance of the underlying table, because of the additional non-key, non-constituent columns that might be present. __CONSTITUENT__ is an automatically-generated name, but users are also able to supply their own names; QueryTable in this case is the description of the table, and need not always be so boring.

[mofojed]

I should have asked the previous question on #2079 instead of this ticket, let's keep this ticket focussed on just the fix.

Debugging a little bit, it looks like the JsPartitionedTable.keyColumns returns just two key columns, ['A', 'B']; however, the JsPartitionedTable.keyTable returns a table with 3 columns, ['A', 'B', 'Last'], which doesn't seem correct.

If you run the code on the Java side looking at the keys table on the ptagg object, it is returning a table with only two columns:

print(ptagg.key_columns) # outputs ['A', 'B']
print(ptagg.keys().columns) # outputs [Column(name='A', data_type=java.lang.Boolean, component_type=None, column_type=NORMAL), Column(name='B', data_type=long, component_type=None, column_type=NORMAL)]

So it seems correct there, so for some reason the Last column is being added in somewhere, and I don't think it should be.