[WIP] Report all unsupported operations for a query in cudf.polars #16960
Conversation
Matt711
added
feature request
github-actions
bot
added
Python
| def evaluate(self, *, cache: MutableMapping[int, DataFrame]) -> DataFrame: | ||
| return pl.DataFrame() |
There was a problem hiding this comment.
| def evaluate(self, *, cache: MutableMapping[int, DataFrame]) -> DataFrame: | |
| return pl.DataFrame() | |
| def evaluate(self, *, cache: MutableMapping[int, DataFrame]) -> DataFrame: | |
| return DataFrame([]) |
The object evaluate returns should be an internal DataFrame, rather than a polars DataFrame.
But, plausibly we don't want to define this method at all, and just let the default implementation raise.
There was a problem hiding this comment.
It would be nice if we could keep track of the exception so that we can report it later.
There was a problem hiding this comment.
I agree that we should keep track of the errors. There's two ways off the top of my head, either maintain a global list that we append to in debug mode, or, have the ErrorNode constructor accept a string error message which attaches it to the instance itself. Either way I think in debug mode we'll need to read once more over the errors and print them in some reasonable way.
| from __future__ import annotations | ||
|
|
||
| import os | ||
|
|
||
|
|
||
| def _env_get_int(name, default): | ||
| """Get the integer value of the environment variable.""" | ||
| try: | ||
| return int(os.getenv(name, default)) | ||
| except (ValueError, TypeError): | ||
| return default | ||
|
|
||
|
|
||
| def _env_get_bool(name, default): | ||
| """Get the the boolean value of the environment variable.""" | ||
| env = os.getenv(name) | ||
| if env is None: | ||
| return default | ||
| as_a_int = _env_get_int(name, None) | ||
| env = env.lower().strip() | ||
| if env == "true" or env == "on" or as_a_int: | ||
| return True | ||
| if env == "false" or env == "off" or as_a_int == 0: | ||
| return False | ||
| return default |
There was a problem hiding this comment.
suggestion: in the long run we probably want to use the polars Config object for this, rather than having our own parallel configuration.
| if other._env_get_bool("CUDF_POLARS_DEBUG_MODE", default=False): | ||
| return ir.ErrorNode(args[0].get_schema()) |
There was a problem hiding this comment.
This kind of action-at-distance stateful modification of the environment makes me think (and @brandon-b-miller wants it too for other reasons) that we need to carry some kind of config object around in the translate_ir
There was a problem hiding this comment.
The other advantage of sending the config through to translate_ir would be the ability to configure per-query rather than per-session
Description
Closes #16690. The purpose of this PR is to list all of the unique operations that are unsupported by
cudf.polarswhen running a query. The current approach is to create a new node (ErrorNode) in the IR when translating polars IR tocudf.polarsIR if the translation step fails with aNotImplementedError. And then traverse the new tree and report where ErrorNodes occured to the user.Checklist