You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
What went wrong?
When doing a query on a string indexed column, the Spark type UTF8String is not recognized by the Transformation method and it throws a Match error.
This is the result of filtering the e-commerce dataset indexed with qbeast by "brand == 'versace'"
versace (of class org.apache.spark.unsafe.types.UTF8String)
scala.MatchError: versace (of class org.apache.spark.unsafe.types.UTF8String)
at io.qbeast.core.transform.HashTransformation.transform(HashTransformation.scala:11)
at io.qbeast.core.model.QuerySpaceFromTo$.$anonfun$apply$1(QuerySpace.scala:68)
at scala.collection.immutable.List.map(List.scala:293)
at io.qbeast.core.model.QuerySpaceFromTo$.apply(QuerySpace.scala:67)
at io.qbeast.spark.index.query.QuerySpecBuilder.extractQuerySpace(QuerySpecBuilder.scala:107)
at io.qbeast.spark.index.query.QuerySpecBuilder.build(QuerySpecBuilder.scala:144)
at io.qbeast.spark.index.query.QueryExecutor.$anonfun$execute$1(QueryExecutor.scala:22)
The solution is to detect the spark type before calling core functions and parse it to the string representation.
How to reproduce?
Code that triggered the bug, or steps to reproduce:
What went wrong?
When doing a query on a string indexed column, the Spark type UTF8String is not recognized by the Transformation method and it throws a Match error.
This is the result of filtering the e-commerce dataset indexed with qbeast by
"brand == 'versace'"
The solution is to detect the spark type before calling core functions and parse it to the string representation.
How to reproduce?
main on c182980
3. Spark version:
On the spark shell run
spark.version
.3.1.2
On the spark shell run
org.apache.hadoop.util.VersionInfo.getVersion()
.3.2.0
On local computer
The text was updated successfully, but these errors were encountered: