feat(storages): read_parquet uses OpenDAL for all IO operations.

[RinChanNOWWW]

I hereby agree to the terms of the CLA available at: https://databend.rs/dev/policies/cla/

Summary

• replace std::File with opendal for all IO operations. (TODO(Another PR): use opendal to do "glob").
• optimize prewhere logic. (do resort before filter).
• fix scan progress collection.
• TODO(Another PR): support remote location and async operations.

Closes #issue

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

1 Ignored Deployment

Name	Status	Preview	Comments	Updated
databend	⬜️ Ignored (Inspect)			Jan 28, 2023 at 5:47AM (UTC)

[Xuanwo]

only fs and hdfs implemented blocking api. Will we need to test s3 via blocking api too?

[RinChanNOWWW]

only fs and hdfs implemented blocking api. Will we need to test s3 via blocking api too?

I plan to check if the backend supports blocking api and choose which api to use.

But I don't have figured out how to achieve this yet
(reading file meta data is only in sync functions now, maybe we need to use blocking api for these operations?) .

[RinChanNOWWW]

Have no complete idea about the remote location. Let's implement it later.

[BohuTANG]

only fs and hdfs implemented blocking api. Will we need to test s3 via blocking api too?

I plan to check if the backend supports blocking api and choose which api to use.

But I don't have figured out how to achieve this yet (reading file meta data is only in sync functions now, maybe we need to use blocking api for these operations?) .

Like this?
https://github.com/datafuselabs/databend/blob/923f3fab8491411a81222917c17a55675f3aa860/src/query/storages/fuse/src/fuse_table.rs#L180

[RinChanNOWWW]

Yes. Or we block the file meta data reading?

[Xuanwo]

Let's implement it later.

LGTM, we can discussion the remote file support later.