chore: use json_object field to merge doc index and code index, makin…

…g future index schema migration possible (#2229)

* chore: adapt json schema

* extract web builder

* migrate code index

* reorg

* migrate

* format id into different corpus

* migrate code search to united schema

* fix

* cleanup

* extract field_tokens

* extract field_chunk_tokens

* Update answer.rs

* update

* [autofix.ci] apply automated fixes

* restructure tabby_common::index

* cleanup again

* rename IndexSchema

* rename web -> doc

* rename web -> doc in tabby_common

* better naming for index

* better naming in index

* renaming

* renaming

---------

Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>

Cargo.toml

@@ -35,7 +35,7 @@ tracing = "0.1"
 tokio-cron-scheduler = "0.9.4"
 tracing-subscriber = "0.3"
 anyhow = "1.0.71"
-tantivy = "0.22"
+tantivy = { git = "https://github.com/quickwit-oss/tantivy", rev = "b806122" }
 async-trait = "0.1.72"
 reqwest = { version = "0.12" }
 derive_builder = "0.12.0"