feat: add framework of partitioned namespace

[wojiaodoubao]

Draft

This is based on lance-format/lance-namespace#279

Overview
PartitionedNamespace is an impl of LanceNamespace. It is a kind of DirectoryNamespace, containing a field member in type DirectoryNamespace.

Methods
There are 3 kinds of methods of PartitionedNamespace:

• Read and Write (for Engine)
  • plan_scan
  • resolve_or_create_partition_table
  • commit
• Partitioning
  • schema
  • tables
  • partitioning
  • update_partition_spec
  • add_column
  • drop_column
  • rename_column
  • type_promotion
• Create Partitioned Namespace
  • create_partitioned_namespace

[codecov]

Codecov Report

❌ Patch coverage is 0% with 138 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
rust/lance-namespace-impls/src/partition.rs	0.00%	138 Missing ⚠️

📢 Thoughts on this report? Let us know!

[wojiaodoubao]

Hi @jackye1995 , I think may be we can push forward partitioning by first discussing the behavior of PartitionedNamespace, using this pr.

After the PartitionedNamespace design is done, we can split partitioning into more sub-tasks and start integrate engines.

Looking forward to your thoughts!

[jackye1995]

nit: since it's a part of the namespace spec, I think we can add this as a part of the OpenAPI definition, so it is codegened and has all the serde aspects handled automatically.

[xloya]

Draft

This is based on lance-format/lance-namespace#279

Overview PartitionedNamespace is an impl of LanceNamespace. It is a kind of DirectoryNamespace, containing a field member in type DirectoryNamespace.

Methods There are 3 kinds of methods of PartitionedNamespace:

• Read and Write (for Engine)

  • plan_scan
  • resolve_or_create_partition_table
  • commit

• Partitioning

  • schema
  • tables
  • partitioning
  • update_partition_spec
  • add_column
  • drop_column
  • rename_column
  • type_promotion

• Create Partitioned Namespace

  • create_partitioned_namespace

Draft

This is based on lance-format/lance-namespace#279

Overview PartitionedNamespace is an impl of LanceNamespace. It is a kind of DirectoryNamespace, containing a field member in type DirectoryNamespace.

Methods There are 3 kinds of methods of PartitionedNamespace:

• Read and Write (for Engine)

  • plan_scan
  • resolve_or_create_partition_table
  • commit

• Partitioning

  • schema
  • tables
  • partitioning
  • update_partition_spec
  • add_column
  • drop_column
  • rename_column
  • type_promotion

• Create Partitioned Namespace

  • create_partitioned_namespace

Hi, I've only recently come across this discussion. Currently, it seems that a partition table is defined as a special type of Directory Namespace. My question is whether partition tables can be extended to also be applicable to REST Namespace Impl, thus adapting to a wider range of scenarios?

My question stems from our scenario having a centralized metadata service, where we'd prefer to manage both non-partitioned and partitioned tables uniformly using REST Namespaces.

Looking forward to your reply, thanks!

[wojiaodoubao]

My question stems from our scenario having a centralized metadata service, where we'd prefer to manage both non-partitioned and partitioned tables uniformly using REST Namespaces.

Hi @xloya , thanks your suggestion. One good feature of partitioned namespace is: it transforms a directory as a partitioned table. It means you can register a partitioned namespace uri to your centralized metadata service, just like normal lance table. Does is work for you?

[xloya]

My question stems from our scenario having a centralized metadata service, where we'd prefer to manage both non-partitioned and partitioned tables uniformly using REST Namespaces.

Hi @xloya , thanks your suggestion. One good feature of partitioned namespace is: it transforms a directory as a partitioned table. It means you can register a partitioned namespace uri to your centralized metadata service, just like normal lance table. Does is work for you?

If the following alignment can be achieved during registration, I think it will meet our needs, such as lance_catalog.schema.non_partitioned_table and lance_catalog.schema.partitioned_namespace, where partitioned_namespace represents a partitioned lance table.

[wojiaodoubao]

such as lance_catalog.schema.non_partitioned_table and lance_catalog.schema.partitioned_namespace, where partitioned_namespace represents a partitioned lance table.

Yes, partitioned namespace can work like this.

[xloya]

such as lance_catalog.schema.non_partitioned_table and lance_catalog.schema.partitioned_namespace, where partitioned_namespace represents a partitioned lance table.

Yes, partitioned namespace can work like this.

Cool! That's what we need, thanks!