[SPARK-40770][PYTHON] Improved error messages for applyInPandas for schema mismatch #38223

EnricoMi · 2022-10-12T14:37:23Z

What changes were proposed in this pull request?

Improve the error messages when a Python method provided to DataFrame.groupby(...).applyInPandas / DataFrame.groupby(...).cogroup(...).applyInPandas returns a Pandas DataFrame that does not match the expected schema.

With

gdf = spark.range(2).join(spark.range(3).withColumnRenamed("id", "val")).groupby("id")

Mismatching column names, matching number of columns:

gdf.applyInPandas(lambda pdf: pdf.rename(columns={"val": "v"}), "id long, val long").show()
# was: KeyError: 'val'
# now: RuntimeError: Column names of the returned pandas.DataFrame do not match specified schema.
#      Missing: val  Unexpected: v

Mismatching column names, different number of columns:

gdf.applyInPandas(lambda pdf: pdf.assign(foo=[3, 3, 3]).rename(columns={"val": "v"}), "id long, val long").show()
# was: RuntimeError: Number of columns of the returned pandas.DataFrame doesn't match specified schema.
#      Expected: 2 Actual: 3
# now: RuntimeError: Column names of the returned pandas.DataFrame do not match specified schema.
#      Missing: val  Unexpected: foo, v

Expected schema matches but has duplicates (id) so that number of columns match:

gdf.applyInPandas(lambda pdf: pdf.rename(columns={"val": "v"}), "id long, id long").show()
# was: java.lang.IllegalArgumentException: not all nodes and buffers were consumed.
#      nodes: [ArrowFieldNode [length=3, nullCount=0]]
#      buffers: [ArrowBuf[304], address:139860828549160, length:0, ArrowBuf[305], address:139860828549160, length:24]
# now: RuntimeError: Column names of the returned pandas.DataFrame do not match specified schema.
#      Unexpected: v

In case the returned Pandas DataFrame contains no column names (none of the column labels is a string):

gdf.applyInPandas(lambda pdf: pdf.assign(foo=[3, 3, 3]).rename(columns={"id": 0, "val": 1, "foo": 2}), "id long, val long").show()
# was: RuntimeError: Number of columns of the returned pandas.DataFrame doesn't match specified schema.
#      Expected: 2 Actual: 3
# now: RuntimeError: Number of columns of the returned pandas.DataFrame doesn't match specified schema.
#      Expected: 2  Actual: 3

Mismatching types (ValueError and TypeError):

gdf.applyInPandas(lambda pdf: pdf, "id int, val string").show()
# was: pyarrow.lib.ArrowTypeError: Expected a string or bytes dtype, got int64
# now: pyarrow.lib.ArrowTypeError: Expected a string or bytes dtype, got int64
#      The above exception was the direct cause of the following exception:
#      TypeError: Exception thrown when converting pandas.Series (int64) with name 'val' to Arrow Array (string).


gdf.applyInPandas(lambda pdf: pdf.assign(val=pdf["val"].apply(str)), "id int, val double").show()
# was: pyarrow.lib.ArrowInvalid: Could not convert '0' with type str: tried to convert to double
# now: pyarrow.lib.ArrowInvalid: Could not convert '0' with type str: tried to convert to double
#      The above exception was the direct cause of the following exception:
#      ValueError: Exception thrown when converting pandas.Series (object) with name 'val' to Arrow Array (double).

with self.sql_conf({"spark.sql.execution.pandas.convertToArrowArraySafely": True}):
  gdf.applyInPandas(lambda pdf: pdf.assign(val=pdf["val"].apply(str)), "id int, val double").show()
# was: ValueError: Exception thrown when converting pandas.Series (object) to Arrow Array (double).
#      It can be caused by overflows or other unsafe conversions warned by Arrow. Arrow safe type check can be disabled
#      by using SQL config `spark.sql.execution.pandas.convertToArrowArraySafely`.
# now: ValueError: Exception thrown when converting pandas.Series (object) with name 'val' to Arrow Array (double).
#      It can be caused by overflows or other unsafe conversions warned by Arrow. Arrow safe type check can be disabled
#      by using SQL config `spark.sql.execution.pandas.convertToArrowArraySafely`.

Why are the changes needed?

Existing errors are generic (KeyError) or meaningless (not all nodes and buffers were consumed). The errors should help users in spotting the mismatching columns by naming them.

The schema of the returned Pandas DataFrames can only be checked during processing the DataFrame, so such errors are very expensive. Therefore, they should be expressive.

Does this PR introduce any user-facing change?

This only changes error messages, not behaviour.

How was this patch tested?

Tests all cases of schema mismatch for GroupedData.applyInPandas and PandasCogroupedOps.applyInPandas.

EnricoMi · 2022-10-12T14:41:25Z

The ... Schema: id, val bit can grow arbitrarily large, maybe this should be limited to the first X columns or characters.

If there are multiple calls to applyInPandas in a DataFrame's plan, this should help to spot which one cased the error.

EnricoMi · 2022-10-13T06:56:56Z

@HyukjinKwon @cloud-fan @zhengruifeng @itholic

zhengruifeng · 2022-10-13T07:08:48Z

also cc @itholic since it's to improve pyspark error message

AmplabJenkins · 2022-10-14T02:19:00Z

Can one of the admins verify this patch?

cloud-fan · 2022-10-19T12:53:56Z

Yea I think we should provide better error message for this user error. @HyukjinKwon can you take a look?

HyukjinKwon · 2022-10-21T02:29:07Z

python/pyspark/sql/pandas/serializers.py

I think you can get the name from Series e.g., s.name instead of passing extra n?

I know, this parameter is ugly as it is only used for the exception, but s.name is 1 instead of mean (test test_apply_in_pandas_returning_incompatible_type) when I apply this change.

Looks like s[field.name] and s[s.columns[i]] returns a Series that thinks it's name is 1. Sad.

But s[s.columns[i]].rename(field.name) fixes that. Thanks for the idea!

HyukjinKwon · 2022-10-21T02:30:37Z

Looks pretty fine from a cursory look. @zhengruifeng @xinrong-meng @itholic would be great if you guys find some time for a second look.

itholic

Looks good except #38223 (comment).

Nice work!

EnricoMi · 2022-10-21T10:00:18Z

I have also limited the schema string in the error message to 1024 characters / missing and unexpected columns to 5.

EnricoMi · 2022-10-31T18:54:59Z

@HyukjinKwon I have addressed your comment.

HyukjinKwon · 2022-11-01T01:44:47Z

Since this touches the core Python worker side, let's make sure to get reviewed from other people like @BryanCutler @viirya @ueshin too.

HyukjinKwon · 2022-11-01T01:46:21Z

python/pyspark/sql/tests/pandas/test_pandas_cogrouped_map.py

I think regex match is fine. this is too long .e.g, just column_with_long_column_name_.* would be good enough.

this is gone

HyukjinKwon · 2022-11-01T01:47:01Z

python/pyspark/sql/tests/pandas/test_pandas_cogrouped_map.py

Maybe you could leverage r"ValueError: ..." to avoid backslashes.

HyukjinKwon · 2022-11-01T01:48:31Z

python/pyspark/worker.py

Hm, I wonder if this abbreviation is too much (and the character limit) ...

e.g., would have to consider having a config to control this too ..

Hm, I wonder if this abbreviation is too much

Too much logic (better remove) or the limit too high?

@HyukjinKwon I need your input here. Should I remove the abbreviation logic, or even printing the schema at all, to simplify things?

@HyukjinKwon I have removed the schema and columns limit.

ueshin · 2022-11-03T23:39:21Z

python/pyspark/worker.py

I guess we can call assign_cols_by_name(runner_conf) outside of wrapped?

Yes we could, but what is the benefit of moving that out of wrapped?

wrapped will be called many times. We want to reduce as much overhead as possible.

Very good point, thanks for clarification.

ueshin · 2022-11-03T23:39:28Z

python/pyspark/worker.py

python/pyspark/sql/tests/pandas/test_pandas_cogrouped_map.py

python/pyspark/worker.py

viirya · 2022-11-24T09:23:29Z

python/pyspark/worker.py

Suggested change

"Number of columns of the returned pandas.DataFrame "

"doesn't match specified schema. "

"Expected: {} Actual: {}".format(

len(return_type),

len(result.columns)

"Number of columns of the returned pandas.DataFrame "

"doesn't match specified schema. "

"Expected: {}. Actual: {}".format(

len(return_type),

len(result.columns)

Reverted this error message to before PR since that message is always short and easy to read without spaces or periods.

viirya

Looks good from a curiosity look.

HyukjinKwon · 2023-02-09T00:52:01Z

Merged to master.

…chema mismatch ### What changes were proposed in this pull request? Improve the error messages when a Python method provided to `DataFrame.groupby(...).applyInPandas` / `DataFrame.groupby(...).cogroup(...).applyInPandas` returns a Pandas DataFrame that does not match the expected schema. With ```Python gdf = spark.range(2).join(spark.range(3).withColumnRenamed("id", "val")).groupby("id") ``` **Mismatching column names, matching number of columns:** ```Python gdf.applyInPandas(lambda pdf: pdf.rename(columns={"val": "v"}), "id long, val long").show() # was: KeyError: 'val' # now: RuntimeError: Column names of the returned pandas.DataFrame do not match specified schema. # Missing: val Unexpected: v ``` **Mismatching column names, different number of columns:** ```Python gdf.applyInPandas(lambda pdf: pdf.assign(foo=[3, 3, 3]).rename(columns={"val": "v"}), "id long, val long").show() # was: RuntimeError: Number of columns of the returned pandas.DataFrame doesn't match specified schema. # Expected: 2 Actual: 3 # now: RuntimeError: Column names of the returned pandas.DataFrame do not match specified schema. # Missing: val Unexpected: foo, v ``` **Expected schema matches but has duplicates (`id`) so that number of columns match:** ```Python gdf.applyInPandas(lambda pdf: pdf.rename(columns={"val": "v"}), "id long, id long").show() # was: java.lang.IllegalArgumentException: not all nodes and buffers were consumed. # nodes: [ArrowFieldNode [length=3, nullCount=0]] # buffers: [ArrowBuf[304], address:139860828549160, length:0, ArrowBuf[305], address:139860828549160, length:24] # now: RuntimeError: Column names of the returned pandas.DataFrame do not match specified schema. # Unexpected: v ``` **In case the returned Pandas DataFrame contains no column names (none of the column labels is a string):** ```Python gdf.applyInPandas(lambda pdf: pdf.assign(foo=[3, 3, 3]).rename(columns={"id": 0, "val": 1, "foo": 2}), "id long, val long").show() # was: RuntimeError: Number of columns of the returned pandas.DataFrame doesn't match specified schema. # Expected: 2 Actual: 3 # now: RuntimeError: Number of columns of the returned pandas.DataFrame doesn't match specified schema. # Expected: 2 Actual: 3 ``` **Mismatching types (ValueError and TypeError):** ```Python gdf.applyInPandas(lambda pdf: pdf, "id int, val string").show() # was: pyarrow.lib.ArrowTypeError: Expected a string or bytes dtype, got int64 # now: pyarrow.lib.ArrowTypeError: Expected a string or bytes dtype, got int64 # The above exception was the direct cause of the following exception: # TypeError: Exception thrown when converting pandas.Series (int64) with name 'val' to Arrow Array (string). gdf.applyInPandas(lambda pdf: pdf.assign(val=pdf["val"].apply(str)), "id int, val double").show() # was: pyarrow.lib.ArrowInvalid: Could not convert '0' with type str: tried to convert to double # now: pyarrow.lib.ArrowInvalid: Could not convert '0' with type str: tried to convert to double # The above exception was the direct cause of the following exception: # ValueError: Exception thrown when converting pandas.Series (object) with name 'val' to Arrow Array (double). with self.sql_conf({"spark.sql.execution.pandas.convertToArrowArraySafely": safely}): gdf.applyInPandas(lambda pdf: pdf.assign(val=pdf["val"].apply(str)), "id int, val double").show() # was: ValueError: Exception thrown when converting pandas.Series (object) to Arrow Array (double). # It can be caused by overflows or other unsafe conversions warned by Arrow. Arrow safe type check can be disabled # by using SQL config `spark.sql.execution.pandas.convertToArrowArraySafely`. # now: ValueError: Exception thrown when converting pandas.Series (object) with name 'val' to Arrow Array (double). # It can be caused by overflows or other unsafe conversions warned by Arrow. Arrow safe type check can be disabled # by using SQL config `spark.sql.execution.pandas.convertToArrowArraySafely`. ``` ### Why are the changes needed? Existing errors are generic (`KeyError`) or meaningless (`not all nodes and buffers were consumed`). The errors should help users in spotting the mismatching columns by naming them. The schema of the returned Pandas DataFrames can only be checked during processing the DataFrame, so such errors are very expensive. Therefore, they should be expressive. ### Does this PR introduce _any_ user-facing change? This only changes error messages, not behaviour. ### How was this patch tested? Tests all cases of schema mismatch for `GroupedData.applyInPandas` and `PandasCogroupedOps.applyInPandas`. Closes #38223 from EnricoMi/branch-pyspark-apply-in-pandas-schema-mismatch. Authored-by: Enrico Minack <github@enrico.minack.dev> Signed-off-by: Hyukjin Kwon <gurwls223@apache.org> (cherry picked from commit bdf56c4) Signed-off-by: Hyukjin Kwon <gurwls223@apache.org>

EnricoMi · 2023-03-02T13:45:44Z

@HyukjinKwon why has this been reverted in branch-3.4? a86324c

CI looked pretty green: https://github.com/apache/spark/actions/runs/4129838944/jobs/7135888364

This contribution had been finished in November, see above comments. Though branch-3.4 has been cut, this should still have gone into the 3.4.0 release.

…das for schema mismatch ### What changes were proposed in this pull request? Similar to #38223, improve the error messages when a Python method provided to `DataFrame.mapInPandas` returns a Pandas DataFrame that does not match the expected schema. With ```Python df = spark.range(2).withColumn("v", col("id")) ``` **Mismatching column names:** ```Python df.mapInPandas(lambda it: it, "id long, val long").show() # was: KeyError: 'val' # now: RuntimeError: Column names of the returned pandas.DataFrame do not match specified schema. # Missing: val Unexpected: v ``` **Python function not returning iterator:** ```Python df.mapInPandas(lambda it: 1, "id long").show() # was: TypeError: 'int' object is not iterable # now: TypeError: Return type of the user-defined function should be iterator of pandas.DataFrame, but is <class 'int'> ``` **Python function not returning iterator of pandas.DataFrame:** ```Python df.mapInPandas(lambda it: [1], "id long").show() # was: TypeError: Return type of the user-defined function should be Pandas.DataFrame, but is <class 'int'> # now: TypeError: Return type of the user-defined function should be iterator of pandas.DataFrame, but is iterator of <class 'int'> # sometimes: ValueError: A field of type StructType expects a pandas.DataFrame, but got: <class 'list'> # now: TypeError: Return type of the user-defined function should be iterator of pandas.DataFrame, but is iterator of <class 'list'> ``` **Mismatching types (ValueError and TypeError):** ```Python df.mapInPandas(lambda it: it, "id int, v string").show() # was: pyarrow.lib.ArrowTypeError: Expected a string or bytes dtype, got int64 # now: pyarrow.lib.ArrowTypeError: Expected a string or bytes dtype, got int64 # The above exception was the direct cause of the following exception: # TypeError: Exception thrown when converting pandas.Series (int64) with name 'v' to Arrow Array (string). df.mapInPandas(lambda it: [pdf.assign(v=pdf["v"].apply(str)) for pdf in it], "id int, v double").show() # was: pyarrow.lib.ArrowInvalid: Could not convert '0' with type str: tried to convert to double # now: pyarrow.lib.ArrowInvalid: Could not convert '0' with type str: tried to convert to double # The above exception was the direct cause of the following exception: # ValueError: Exception thrown when converting pandas.Series (object) with name 'v' to Arrow Array (double). with self.sql_conf({"spark.sql.execution.pandas.convertToArrowArraySafely": True}): df.mapInPandas(lambda it: [pdf.assign(v=pdf["v"].apply(str)) for pdf in it], "id int, v double").show() # was: ValueError: Exception thrown when converting pandas.Series (object) to Arrow Array (double). # It can be caused by overflows or other unsafe conversions warned by Arrow. Arrow safe type check can be disabled # by using SQL config `spark.sql.execution.pandas.convertToArrowArraySafely`. # now: ValueError: Exception thrown when converting pandas.Series (object) with name 'v' to Arrow Array (double). # It can be caused by overflows or other unsafe conversions warned by Arrow. Arrow safe type check can be disabled # by using SQL config `spark.sql.execution.pandas.convertToArrowArraySafely`. ``` ### Why are the changes needed? Existing errors are generic (`KeyError`) or meaningless (`'int' object is not iterable`). The errors should help users in spotting the mismatching columns by naming them. The schema of the returned Pandas DataFrames can only be checked during processing the DataFrame, so such errors are very expensive. Therefore, they should be expressive. ### Does this PR introduce _any_ user-facing change? This only changes error messages, not behaviour. ### How was this patch tested? Tests all cases of schema mismatch for `DataFrame.mapInPandas`. Closes #39952 from EnricoMi/branch-pyspark-map-in-pandas-schema-mismatch. Authored-by: Enrico Minack <github@enrico.minack.dev> Signed-off-by: Xinrong Meng <xinrong@apache.org>

…InPandas for schema mismatch ### What changes were proposed in this pull request? This merges #39952 into 3.5 branch. Similar to #38223, improve the error messages when a Python method provided to `DataFrame.mapInPandas` returns a Pandas DataFrame that does not match the expected schema. With ```Python df = spark.range(2).withColumn("v", col("id")) ``` **Mismatching column names:** ```Python df.mapInPandas(lambda it: it, "id long, val long").show() # was: KeyError: 'val' # now: RuntimeError: Column names of the returned pandas.DataFrame do not match specified schema. # Missing: val Unexpected: v ``` **Python function not returning iterator:** ```Python df.mapInPandas(lambda it: 1, "id long").show() # was: TypeError: 'int' object is not iterable # now: TypeError: Return type of the user-defined function should be iterator of pandas.DataFrame, but is <class 'int'> ``` **Python function not returning iterator of pandas.DataFrame:** ```Python df.mapInPandas(lambda it: [1], "id long").show() # was: TypeError: Return type of the user-defined function should be Pandas.DataFrame, but is <class 'int'> # now: TypeError: Return type of the user-defined function should be iterator of pandas.DataFrame, but is iterator of <class 'int'> # sometimes: ValueError: A field of type StructType expects a pandas.DataFrame, but got: <class 'list'> # now: TypeError: Return type of the user-defined function should be iterator of pandas.DataFrame, but is iterator of <class 'list'> ``` **Mismatching types (ValueError and TypeError):** ```Python df.mapInPandas(lambda it: it, "id int, v string").show() # was: pyarrow.lib.ArrowTypeError: Expected a string or bytes dtype, got int64 # now: pyarrow.lib.ArrowTypeError: Expected a string or bytes dtype, got int64 # The above exception was the direct cause of the following exception: # TypeError: Exception thrown when converting pandas.Series (int64) with name 'v' to Arrow Array (string). df.mapInPandas(lambda it: [pdf.assign(v=pdf["v"].apply(str)) for pdf in it], "id int, v double").show() # was: pyarrow.lib.ArrowInvalid: Could not convert '0' with type str: tried to convert to double # now: pyarrow.lib.ArrowInvalid: Could not convert '0' with type str: tried to convert to double # The above exception was the direct cause of the following exception: # ValueError: Exception thrown when converting pandas.Series (object) with name 'v' to Arrow Array (double). with self.sql_conf({"spark.sql.execution.pandas.convertToArrowArraySafely": True}): df.mapInPandas(lambda it: [pdf.assign(v=pdf["v"].apply(str)) for pdf in it], "id int, v double").show() # was: ValueError: Exception thrown when converting pandas.Series (object) to Arrow Array (double). # It can be caused by overflows or other unsafe conversions warned by Arrow. Arrow safe type check can be disabled # by using SQL config `spark.sql.execution.pandas.convertToArrowArraySafely`. # now: ValueError: Exception thrown when converting pandas.Series (object) with name 'v' to Arrow Array (double). # It can be caused by overflows or other unsafe conversions warned by Arrow. Arrow safe type check can be disabled # by using SQL config `spark.sql.execution.pandas.convertToArrowArraySafely`. ``` ### Why are the changes needed? Existing errors are generic (`KeyError`) or meaningless (`'int' object is not iterable`). The errors should help users in spotting the mismatching columns by naming them. The schema of the returned Pandas DataFrames can only be checked during processing the DataFrame, so such errors are very expensive. Therefore, they should be expressive. ### Does this PR introduce _any_ user-facing change? This only changes error messages, not behaviour. ### How was this patch tested? Tests all cases of schema mismatch for `DataFrame.mapInPandas`. Closes #42316 from EnricoMi/branch-pyspark-map-in-pandas-schema-mismatch-3.5. Authored-by: Enrico Minack <github@enrico.minack.dev> Signed-off-by: Hyukjin Kwon <gurwls223@apache.org>

github-actions bot added CORE PYTHON SQL labels Oct 12, 2022

EnricoMi force-pushed the branch-pyspark-apply-in-pandas-schema-mismatch branch from 4af2a61 to 3d2ebbd Compare October 17, 2022 14:16

HyukjinKwon reviewed Oct 21, 2022

View reviewed changes

itholic approved these changes Oct 21, 2022

View reviewed changes

EnricoMi force-pushed the branch-pyspark-apply-in-pandas-schema-mismatch branch 3 times, most recently from 7c1da83 to 964d891 Compare October 21, 2022 09:57

EnricoMi force-pushed the branch-pyspark-apply-in-pandas-schema-mismatch branch 2 times, most recently from 3c761ec to 6465ca5 Compare October 21, 2022 13:38

HyukjinKwon reviewed Nov 1, 2022

View reviewed changes

ueshin reviewed Nov 3, 2022

View reviewed changes

viirya reviewed Nov 24, 2022

View reviewed changes

python/pyspark/sql/tests/pandas/test_pandas_cogrouped_map.py Outdated Show resolved Hide resolved

viirya reviewed Nov 24, 2022

View reviewed changes

python/pyspark/sql/tests/pandas/test_pandas_cogrouped_map.py Outdated Show resolved Hide resolved

viirya reviewed Nov 24, 2022

View reviewed changes

python/pyspark/worker.py Outdated Show resolved Hide resolved

viirya reviewed Nov 24, 2022

View reviewed changes

EnricoMi force-pushed the branch-pyspark-apply-in-pandas-schema-mismatch branch from b2c3fde to 5e57eb3 Compare November 24, 2022 10:26

github-actions bot removed STRUCTURED STREAMING SPARK SHELL DSTREAM PANDAS API ON SPARK AVRO MLLIB GRAPHX EXAMPLES CONNECT PROTOBUF WEB UI MESOS YARN INFRA ML DOCS labels Feb 8, 2023

EnricoMi added 2 commits February 8, 2023 09:43

Rename test class names to proper name

124cad0

Assert full expected result, not only subset

df27aa7

HyukjinKwon approved these changes Feb 9, 2023

View reviewed changes

HyukjinKwon closed this in bdf56c4 Feb 9, 2023

EnricoMi mentioned this pull request Feb 9, 2023

[SPARK-40770][PYTHON][FOLLOW-UP] Improved error messages for mapInPandas for schema mismatch #39952

Closed

EnricoMi mentioned this pull request Aug 3, 2023

[SPARK-40770][PYTHON][FOLLOW-UP][3.5] Improved error messages for mapInPandas for schema mismatch #42316

Closed

-                "Number of columns of the returned pandas.DataFrame "
-                "doesn't match specified schema.  "
-                "Expected: {}  Actual: {}".format(
-                    len(return_type),
-                    len(result.columns)
+                "Number of columns of the returned pandas.DataFrame "
+                "doesn't match specified schema.  "
+                "Expected: {}.  Actual: {}".format(
+                    len(return_type),
+                    len(result.columns)

[SPARK-40770][PYTHON] Improved error messages for applyInPandas for schema mismatch #38223

[SPARK-40770][PYTHON] Improved error messages for applyInPandas for schema mismatch #38223

Uh oh!

Conversation

EnricoMi commented Oct 12, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Uh oh!

EnricoMi commented Oct 12, 2022

Uh oh!

EnricoMi commented Oct 13, 2022

Uh oh!

zhengruifeng commented Oct 13, 2022

Uh oh!

AmplabJenkins commented Oct 14, 2022

Uh oh!

cloud-fan commented Oct 19, 2022

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

HyukjinKwon commented Oct 21, 2022

Uh oh!

itholic left a comment

Choose a reason for hiding this comment

Uh oh!

EnricoMi commented Oct 21, 2022

Uh oh!

EnricoMi commented Oct 31, 2022

Uh oh!

HyukjinKwon commented Nov 1, 2022

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

viirya left a comment

EnricoMi commented Oct 12, 2022 •

edited

Loading

HyukjinKwon commented Feb 9, 2023 •

edited

Loading

EnricoMi commented Mar 2, 2023 •

edited

Loading