Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Bug]: Fix File format is required when scanning file entries if no file format is specified in the file name #2441

Closed
2 tasks done
Tracked by #2448
wangtaohz opened this issue Dec 18, 2023 · 0 comments · Fixed by #2442
Closed
2 tasks done
Tracked by #2448
Labels
type:bug Something isn't working

Comments

@wangtaohz
Copy link
Contributor

wangtaohz commented Dec 18, 2023

What happened?

When there is no file format indicated in the file name(like xxxx.parquet, yyyy.avro), TableEntriesScan will throw an error File format is required.

Affects Versions

master

What engines are you seeing the problem on?

Core

How to reproduce

No response

Relevant log output

[com.netease.arctic.server.table.executor.SnapshotsExpiringExecutor] [] - unexpected expire error of table arcti
c_xxx.yyy.zzz(tableId=155)
java.lang.IllegalArgumentException: File format is required
        at org.apache.iceberg.relocated.com.google.common.base.Preconditions.checkArgument(Preconditions.java:145) ~[iceberg-bundled-guava-1.3.0.jar:?]
        at org.apache.iceberg.DataFiles$Builder.build(DataFiles.java:293) ~[iceberg-core-1.3.0.jar:?]
        at com.netease.arctic.scan.TableEntriesScan.buildDataFile(TableEntriesScan.java:325) ~[amoro-core-0.6.0-SNAPSHOT.jar:?]
        at com.netease.arctic.scan.TableEntriesScan.buildContentFile(TableEntriesScan.java:301) ~[amoro-core-0.6.0-SNAPSHOT.jar:?]
        at com.netease.arctic.scan.TableEntriesScan.lambda$entries$0(TableEntriesScan.java:240) ~[amoro-core-0.6.0-SNAPSHOT.jar:?]
        at org.apache.iceberg.io.CloseableIterable$7$1.next(CloseableIterable.java:202) ~[iceberg-api-1.3.0.jar:?]
        at org.apache.iceberg.io.FilterIterator.advance(FilterIterator.java:65) ~[iceberg-api-1.3.0.jar:?]
        at org.apache.iceberg.io.FilterIterator.hasNext(FilterIterator.java:49) ~[iceberg-api-1.3.0.jar:?]
        at com.netease.arctic.server.utils.IcebergTableUtil.getAllContentFilePath(IcebergTableUtil.java:155) ~[amoro-ams-server-0.6.0-SNAPSHOT.jar:?]
        at com.netease.arctic.server.optimizing.maintainer.MixedTableMaintainer.<init>(MixedTableMaintainer.java:91) ~[amoro-ams-server-0.6.0-SNAPSHOT.jar:?]
        at com.netease.arctic.server.optimizing.maintainer.TableMaintainer.ofTable(TableMaintainer.java:46) ~[amoro-ams-server-0.6.0-SNAPSHOT.jar:?]
        at com.netease.arctic.server.table.executor.SnapshotsExpiringExecutor.execute(SnapshotsExpiringExecutor.java:59) ~[amoro-ams-server-0.6.0-SNAPSHOT.jar:?]
        at com.netease.arctic.server.table.executor.BaseTableExecutor.executeTask(BaseTableExecutor.java:72) ~[amoro-ams-server-0.6.0-SNAPSHOT.jar:?]
        at com.netease.arctic.server.table.executor.BaseTableExecutor.lambda$scheduleIfNecessary$4(BaseTableExecutor.java:81) ~[amoro-ams-server-0.6.0-SNAPSHOT.jar:?]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_101]
        at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_101]
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_101]
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [?:1.8.0_101]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_101]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_101]
        at java.lang.Thread.run(Thread.java:745) [?:1.8.0_101]

Anything else

No response

Are you willing to submit a PR?

  • Yes I am willing to submit a PR!

Code of Conduct

  • I agree to follow this project's Code of Conduct
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
type:bug Something isn't working
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant