refactor: Cache dtypes for scalar expressions for SQLGlot compiler

[sycai]

Make DerefOp hold a field id_or_field, and resolve the column ID to actual Field instance upon node creation.

[TrevorBergeron]

Can we just create a new FieldRefOp instead of overloading the existing minimal reference class with optional semantics?

[sycai]

I think that would require a lot more code changes than necessary.

I updated the deref's attribute to this union type: ColumnId | Field. ColumnId means type unresolved. Field means otherwise. This can keep the code change to minimum.

[TrevorBergeron]

There is a certain irreducible complexity here. So something somewhere is checking. I'd rather have the callers deal with this than trying to paper over the differences with a single class.

[TrevorBergeron]

bind_refs should be able to do this, don't think we need a new method

[sycai]

I think that would make things more complicated:

We cannot bake in any short-circuit mechanism. For type resolution, if a expression already has a resolved type, we can skip the binding. This is crucial for gradually updating deref ops with types. We cannot do so with bind_refs because we don't know if the caller's intention is to resolve types or replace expressions.

[TrevorBergeron]

What we want is essentially this:

def resolved(expr, node) -> Expression:
   if expr.resolved:
      return expr
   else:
      return expr.bind_refs({id: ex.field_ref(node.id_to_field[id]) for id in node.ids})

[sycai]

I have encountered an edge case where the ProjectionNode has a resolved expression whose column reference does not appear in the child node. The column ID is "level_0", so it looks like a default index.

For cases like this, this logic would still fail because "level_0" is not present in the map from column IDs to Fields.

[TrevorBergeron]

that sounds like a bug, if it doesn't fail later, that is due to sheer luck. Every reference should match a column in the child schema.

[sycai]

For specifically, the case comes from this test:

python-bigquery-dataframes/tests/unit/core/compile/sqlglot/test_compile_projection.py

Lines 24 to 31 in 50dca4c

	def test_compile_projection(
	scalars_types_pandas_df: pd.DataFrame, compiler_session: bigframes.Session, snapshot
	):
	bf_df = bpd.DataFrame(
	scalars_types_pandas_df[["int64_col"]], session=compiler_session
	)
	bf_df["int64_col"] = bf_df["int64_col"] + 1
	snapshot.assert_match(bf_df.sql, "out.sql")

. Not sure whether it is ought to happen.

[sycai]

What we want is essentially this:

def resolved(expr, node) -> Expression:
   if expr.resolved:
      return expr
   else:
      return expr.bind_refs({id: ex.field_ref(node.id_to_field[id]) for id in node.ids})

This worked! Thanks

[TrevorBergeron]

dealing with the absentdtype in post-binding contexts will be a pain. maybe a strict mode with typing overload?

[sycai]

Right. Removed the AbsentDtype

Instead, I added a property is_type_resolved, which behaves like that field in Spark.

[TrevorBergeron]

I would prefer to just use the field object. Nullability information can be useful for some operation simplifications.

[sycai]

Now deref ops are able to hold both ColumnId and Field values

[TrevorBergeron]

There is a certain irreducible complexity here. So something somewhere is checking. I'd rather have the callers deal with this than trying to paper over the differences with a single class.

[TrevorBergeron]

Not sure this really needs to be cached? Will build a lot of expressions and not care about type a lot of the time.

[sycai]

This is for enabling dynamic dispatch of the SQLGlot compiler.