Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: synchronous remote index dropping #414

Merged
merged 2 commits into from
Mar 13, 2024
Merged

Conversation

usamoi
Copy link
Collaborator

@usamoi usamoi commented Mar 11, 2024


  • Most code in service is moved to crates/index and crates/{algorithm name}. We can improve and test these algorithms independently without DB stuffs.
  • We run crates/{algorithm} in a thread pool while indexing. It's a copy of rayon and we support rayon::check to check cancellation point now.
  • Handle is refactored to take advantage of system identifier and the index oid. Based on it, we reimplemented drop logic: we drop remote index in a object_access_hook.
  • Old prefilter code is removed.
  • This PR introduces jemalloc. The new pgrx release makes it possible.
  • Added a toml format check to ci.

@usamoi usamoi requested a review from cutecutecat March 11, 2024 01:14
@usamoi usamoi force-pushed the main branch 3 times, most recently from 842abfa to 2f9a378 Compare March 11, 2024 01:49
@VoVAllen
Copy link
Member

Please update the details to be more specific about the change since this PR is huge

@VoVAllen
Copy link
Member

ZomboDB generates a UUID to use as the backing Elasticsearch index name, but also assigns an alias in the form of
database_name.schema_name.table_name.index_name-index_oid. "index_oid" is the Postgres catalog id for the index from
the "pg_class" system catalog table.

Shall we use similar approach?

Ref: https://github.com/zombodb/zombodb/blob/1416c99a4885f1cfe5e7bd86b935e89e6d6ba431/INDEX-MANAGEMENT.md?plain=1#L20C1-L23C1

@usamoi usamoi force-pushed the main branch 2 times, most recently from 3e007ab to 5016078 Compare March 11, 2024 16:48
Signed-off-by: usamoi <usamoi@outlook.com>
@usamoi
Copy link
Collaborator Author

usamoi commented Mar 12, 2024

ZomboDB generates a UUID to use as the backing Elasticsearch index name, but also assigns an alias in the form of
database_name.schema_name.table_name.index_name-index_oid. "index_oid" is the Postgres catalog id for the index from
the "pg_class" system catalog table.

Shall we use similar approach?

Ref: https://github.com/zombodb/zombodb/blob/1416c99a4885f1cfe5e7bd86b935e89e6d6ba431/INDEX-MANAGEMENT.md?plain=1#L20C1-L23C1

I think it's not necessary.

src/index/hook_access.rs Outdated Show resolved Hide resolved
src/index/hook_access.rs Outdated Show resolved Hide resolved
tests/transaction/test.sql Outdated Show resolved Hide resolved
src/index/hook_maintain.rs Show resolved Hide resolved
crates/service/src/worker.rs Outdated Show resolved Hide resolved
crates/service/src/worker.rs Show resolved Hide resolved
@usamoi usamoi force-pushed the main branch 5 times, most recently from 83d9170 to 426d1e3 Compare March 12, 2024 17:01
cutecutecat
cutecutecat previously approved these changes Mar 13, 2024
src/index/am.rs Show resolved Hide resolved
Signed-off-by: usamoi <usamoi@outlook.com>
@usamoi usamoi added this pull request to the merge queue Mar 13, 2024
Merged via the queue into tensorchord:main with commit afe2b65 Mar 13, 2024
13 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
3 participants