-
Notifications
You must be signed in to change notification settings - Fork 345
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Bug report] Can not operate STRUCT column of iceberg table #1313
Comments
TEOTEO520
changed the title
[Bug report]
[Bug report] Can not operate STRUCT column of iceberg table
Jan 3, 2024
cc @mchades |
As you described in Additional context, this issue is related to the class |
OK,it may cost me one week to finish it |
FANNG1
pushed a commit
that referenced
this issue
Feb 5, 2024
…berg (#1721) ### What changes were proposed in this pull request? add struct column support for iceberg ### Why are the changes needed? Fix: #1313 Fix: #1471 ### Does this PR introduce _any_ user-facing change? no ### How was this patch tested? UT added --------- Co-authored-by: teo <teo@teodeMacBook-Pro.local>
github-actions bot
pushed a commit
that referenced
this issue
Feb 5, 2024
…berg (#1721) ### What changes were proposed in this pull request? add struct column support for iceberg ### Why are the changes needed? Fix: #1313 Fix: #1471 ### Does this PR introduce _any_ user-facing change? no ### How was this patch tested? UT added --------- Co-authored-by: teo <teo@teodeMacBook-Pro.local>
yuqi1129
pushed a commit
that referenced
this issue
Feb 6, 2024
### What changes were proposed in this pull request? add struct column support for iceberg ### Why are the changes needed? Fix: #1313 Fix: #1471 ### Does this PR introduce _any_ user-facing change? no ### How was this patch tested? UT added Co-authored-by: Tianhang Li <58762426+TEOTEO520@users.noreply.github.com> Co-authored-by: teo <teo@teodeMacBook-Pro.local>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Describe what's wrong
can not operate STRUCT column of iceberg table :
example: create iceberg
{
"name":"gravitino_table_create_iceberg_test_02",
"columns":[
{
"name":"id",
"type":"integer",
"nullable":true,
"comment":"Id of the user"
},
{
"name":"name",
"type":"string",
"nullable":true,
"comment":"Name of the user"
},
{
"name":"struct_type",
"type":{
"type": "struct",
"fields":[
{
"name":"street",
"type":"string",
"nullable":true,
"comment":"11"
},
{
"name":"state",
"type":"string",
"nullable":true,
"comment":"11"
}
]
},
"nullable":true,
"comment":"Name of the user"
}
],
"comment":"Create a new Table",
"properties":{
"format":"ORC"
}
}
Error message and/or stacktrace
{
"code": 1002,
"type": "RuntimeException",
"message": "Failed to operate object [gravitino_table_create_iceberg_test_02] operation [CREATE] under [test_iceberg_bdp], reason [ClassCastException]",
"stack": [
"java.lang.ClassCastException: com.datastrato.gravitino.rel.types.Types$StructType cannot be cast to com.datastrato.gravitino.rel.types.Type$PrimitiveType",
"\tat com.datastrato.gravitino.catalog.lakehouse.iceberg.converter.ToIcebergTypeVisitor.visit(ToIcebergTypeVisitor.java:56)",
"\tat com.datastrato.gravitino.catalog.lakehouse.iceberg.converter.ToIcebergTypeVisitor.visit(ToIcebergTypeVisitor.java:35)",
"\tat com.datastrato.gravitino.catalog.lakehouse.iceberg.converter.ConvertUtil.toIcebergSchema(ConvertUtil.java:23)",
"\tat com.datastrato.gravitino.catalog.lakehouse.iceberg.IcebergTable.toCreateTableRequest(IcebergTable.java:47)",
"\tat com.datastrato.gravitino.catalog.lakehouse.iceberg.IcebergCatalogOperations.createTable(IcebergCatalogOperations.java:526)",
"\tat com.datastrato.gravitino.catalog.CatalogOperationDispatcher.lambda$null$30(CatalogOperationDispatcher.java:431)",
"\tat com.datastrato.gravitino.catalog.CatalogManager$CatalogWrapper.lambda$doWithTableOps$1(CatalogManager.java:96)",
"\tat com.datastrato.gravitino.utils.IsolatedClassLoader.withClassLoader(IsolatedClassLoader.java:69)",
"\tat com.datastrato.gravitino.catalog.CatalogManager$CatalogWrapper.doWithTableOps(CatalogManager.java:91)",
"\tat com.datastrato.gravitino.catalog.CatalogOperationDispatcher.lambda$createTable$31(CatalogOperationDispatcher.java:429)",
"\tat com.datastrato.gravitino.catalog.CatalogOperationDispatcher.doWithCatalog(CatalogOperationDispatcher.java:698)",
"\tat com.datastrato.gravitino.catalog.CatalogOperationDispatcher.createTable(CatalogOperationDispatcher.java:426)",
"\tat com.datastrato.gravitino.server.web.rest.TableOperations.createTable(TableOperations.java:88)",
"\tat sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)",
"\tat sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)",
"\tat sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)",
"\tat java.lang.reflect.Method.invoke(Method.java:498)",
"\tat org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:52)",
"\tat org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:134)",
"\tat org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:177)",
"\tat org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$ResponseOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:176)",
"\tat org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:81)",
"\tat org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:478)",
"\tat org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:400)",
"\tat org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:81)",
"\tat org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:256)",
"\tat org.glassfish.jersey.internal.Errors$1.call(Errors.java:248)",
"\tat org.glassfish.jersey.internal.Errors$1.call(Errors.java:244)",
"\tat org.glassfish.jersey.internal.Errors.process(Errors.java:292)",
"\tat org.glassfish.jersey.internal.Errors.process(Errors.java:274)",
"\tat org.glassfish.jersey.internal.Errors.process(Errors.java:244)",
"\tat org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:265)",
"\tat org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:235)",
"\tat org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:684)",
"\tat org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:394)",
"\tat org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:346)",
"\tat org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:358)",
"\tat org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:311)",
"\tat org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:205)",
"\tat org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799)",
"\tat org.eclipse.jetty.servlet.ServletHandler$ChainEnd.doFilter(ServletHandler.java:1656)",
"\tat com.datastrato.gravitino.server.auth.AuthenticationFilter.doFilter(AuthenticationFilter.java:57)",
"\tat org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)",
"\tat org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626)",
"\tat com.datastrato.gravitino.server.web.VersioningFilter.doFilter(VersioningFilter.java:107)",
"\tat org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)",
"\tat org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626)",
"\tat org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:552)",
"\tat org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)",
"\tat org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:600)",
"\tat org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)",
"\tat org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)",
"\tat org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624)",
"\tat org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)",
"\tat org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1440)",
"\tat org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)",
"\tat org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:505)",
"\tat org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594)",
"\tat org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)",
"\tat org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1355)",
"\tat org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)",
"\tat org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146)",
"\tat org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)",
"\tat org.eclipse.jetty.server.Server.handle(Server.java:516)",
"\tat org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:487)",
"\tat org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:732)",
"\tat org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:479)",
"\tat org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277)",
"\tat org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)",
"\tat org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)",
"\tat org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)",
"\tat org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)",
"\tat org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)",
"\tat java.lang.Thread.run(Thread.java:748)"
]
}
How to reproduce
create table like the expamle
Additional context
ADD JUDGEMENT:
class:ToIcebergTypeVisitor
public static T visit(Type type, ToIcebergTypeVisitor visitor) {
if (type instanceof Types.MapType) {
Types.MapType map = (Types.MapType) type;
return visitor.map(map, visit(map.keyType(), visitor), visit(map.valueType(), visitor));
} else if (type instanceof Types.ListType) {
Types.ListType list = (Types.ListType) type;
return visitor.array(list, visit(list.elementType(), visitor));
} else {
return visitor.atomic((Type.PrimitiveType) type);
}
}
The text was updated successfully, but these errors were encountered: