diff --git a/diagrams.sh b/diagrams.sh index dfb8c285c..848766898 100755 --- a/diagrams.sh +++ b/diagrams.sh @@ -3,10 +3,10 @@ [[ -d ./build ]] || mkdir ./build # install-rr -[[ -f ./build/rr-2.0-java11.zip ]] || curl -L -o ./build/rr-2.0-java11.zip https://github.com/GuntherRademacher/rr/releases/download/v2.0/rr-2.0-java11.zip +[[ -f ./build/rr-2.1-java11.zip ]] || curl -L -o ./build/rr-2.1-java11.zip https://github.com/GuntherRademacher/rr/releases/download/v2.1/rr-2.1-java11.zip # extract-rr -[[ -f ./build/rr/rr.war ]] || unzip ./build/rr-2.0-java11.zip -d ./build/rr +[[ -f ./build/rr/rr.war ]] || unzip ./build/rr-2.1-java11.zip -d ./build/rr for file in n1ql dcl ddl dml dql tcl hints utility do diff --git a/modules/indexes/pages/index-replication.adoc b/modules/indexes/pages/index-replication.adoc index f249e3cac..a3e996716 100644 --- a/modules/indexes/pages/index-replication.adoc +++ b/modules/indexes/pages/index-replication.adoc @@ -40,7 +40,7 @@ This ensures: * _Availability_: If one Index-Service node is lost, the other continues to provide access to replicated indexes. * _High Performance_: If original and replica copies are available, incoming queries are load-balanced across them. -Index-replicas can be created with the {sqlpp} `CREATE INDEX` statement. +Index-replicas can be created with the {sqlpp} `CREATE INDEX` or `CREATE VECTOR INDEX` statement. Note that whenever a given number of index-replicas is specified for creation, the number must be less than the number of cluster-nodes currently running the {index-service}[Index Service]. If it is not, the index creation fails. Note also that if, following creation of the maximum number of copies, the number of nodes running the Index Service decreases, Couchbase Server progressively assigns replacement index-replicas to any and all Index-Service nodes subsequently be added to the cluster, until the required number of index-replicas again exists for each replicated index. @@ -69,10 +69,10 @@ include::example$replication-num.n1ql[] + Note that if `nodes` and `num_replica` are both specified in the `WITH` clause, the specified number of nodes must be _one greater_ than `num_replica`. -* Specifying a number of index-replicas to be created by the Index Service whenever `CREATE INDEX` is invoked. +* Specifying a number of index-replicas to be created by the Index Service whenever `CREATE INDEX` or `CREATE VECTOR INDEX` is invoked. The default is `0`. If the default is changed to, say, `2`, creation of a single index is henceforth accompanied by the creation of two replicas, which are automatically distributed across the nodes of the cluster running the Index Service. -No explicit specification within the `CREATE INDEX` statement is required. +No explicit specification within the `CREATE INDEX` or `CREATE VECTOR INDEX` statement is required. + With credentials that provide appropriate authorization, this default can be changed by means of the `curl` command, as follows: + @@ -83,10 +83,10 @@ curl -X POST -u 'Administrator:password' \ -d 'numReplica=2' ---- + -Here, `numReplica` is an integer that establishes the default number of replicas that must be created whenever `CREATE INDEX` is invoked. +Here, `numReplica` is an integer that establishes the default number of replicas that must be created whenever `CREATE INDEX` or `CREATE VECTOR INDEX` is invoked. Note that this call only succeeds if the cluster contains enough Index Service nodes to host each new index and its replicas: for example, if `2` is specified as the default number of replicas, the Index Service must have been established on at least 3 nodes. + -Note also that whenever explicit specification of replica-numbers is made within the `CREATE INDEX` statement, this explicit specification takes precedence over any established default. +Note also that whenever explicit specification of replica-numbers is made within the `CREATE INDEX` or `CREATE VECTOR INDEX` statement, this explicit specification takes precedence over any established default. You can change index replication settings via the {index-storage-mode}[UI] or the {index-settings-via-rest}[REST API]. For further information on using {sqlpp}, refer to {query}[Query Fundamentals]. diff --git a/modules/indexes/pages/indexing-and-query-perf.adoc b/modules/indexes/pages/indexing-and-query-perf.adoc index 3fe1cbe87..a5b7614ab 100644 --- a/modules/indexes/pages/indexing-and-query-perf.adoc +++ b/modules/indexes/pages/indexing-and-query-perf.adoc @@ -12,6 +12,8 @@ :bucket-analyzer: xref:tools:query-workbench.adoc#bucket-analyzer :storage-modes: xref:indexes:storage-modes.adoc :additional-storage-use: xref:learn:data/transactions.adoc#additional-storage-use +:hyperscale-vector-index: xref:vector-index:hyperscale-vector-index.adoc +:composite-vector-index: xref:vector-index:composite-vector-index.adoc Creating the right index -- with the right keys, in the right order, and using the right expressions -- is critical to query performance in any database system. This is true for Couchbase as well. @@ -20,7 +22,7 @@ This is true for Couchbase as well. .Examples on this Page **** The examples in this topic use the travel-sample dataset which is shipped with Couchbase Server. -For instructions on how to install the sample bucket, see {install-sample-buckets}[Sample Buckets]. +For instructions on how to install the sample bucket, see {install-sample-buckets}[]. include::ROOT:partial$query-context.adoc[tag=statement] **** @@ -79,7 +81,7 @@ Use the primary index for full keyspace scans (primary scans) when the query doe NOTE: A primary index does not index any {additional-storage-use}[transaction records] that may be stored in a keyspace. This means that if you are counting the number of documents in a keyspace, you may see slightly different results, depending on whether you are using a primary index or not. -Refer to {aggregatefun}[Aggregate Functions] and {bucket-analyzer}[Viewing the Data Insights]. +See {aggregatefun}[] and {bucket-analyzer}[Viewing the Data Insights]. .Metadata for Primary Index ==== @@ -479,7 +481,7 @@ CREATE INDEX travel_sched ON route ---- ==== -For further details and examples, refer to {indexing-arrays}[Array Indexing]. +For further details and examples, see {indexing-arrays}[]. [#partial-index] == Partial Index @@ -581,4 +583,21 @@ After the index selection is made, the query engine analyzes the query to see if If it does, the query engine skips retrieving the whole document from the data nodes. This is a performance optimization to keep in mind when designing your indexes. -For further details and examples, refer to {covering-indexes}[Covering Indexes]. +For further details and examples, see {covering-indexes}[]. + +[#hyperscale-vector-index] +== Hyperscale Vector Index + +Hyperscale vector indexes are a type of <> which contain a single vector field. +They excel at indexing huge datasets that can scale into the billions of documents. +They're optimized for pure vector searches, offering the highest performance of any index for your AI applications. + +For further details and examples, see {hyperscale-vector-index}[]. + +[#composite-vector-index] +== Composite Vector Index + +Composite vector indexes are a type of <> which contain a single vector field and one or more scalar fields. +Your AI applications can use the index's scalar fields to filter the dataset before performing a vector similarity search. + +For further details and examples, see {composite-vector-index}[]. diff --git a/modules/indexes/pages/indexing-overview.adoc b/modules/indexes/pages/indexing-overview.adoc index 6b21fd77d..7b9021492 100644 --- a/modules/indexes/pages/indexing-overview.adoc +++ b/modules/indexes/pages/indexing-overview.adoc @@ -35,10 +35,13 @@ endif::[] * xref:n1ql:n1ql-language-reference/advise.adoc[ADVISE] * xref:n1ql:n1ql-language-reference/createindex.adoc[CREATE INDEX] * xref:n1ql:n1ql-language-reference/createprimaryindex.adoc[CREATE PRIMARY INDEX] +* xref:n1ql:n1ql-language-reference/createvectorindex.adoc[CREATE VECTOR INDEX] * xref:n1ql:n1ql-language-reference/build-index.adoc[BUILD INDEX] * xref:n1ql:n1ql-language-reference/alterindex.adoc[ALTER INDEX] +* xref:n1ql:n1ql-language-reference/altervectorindex.adoc[ALTER VECTOR INDEX] * xref:n1ql:n1ql-language-reference/dropindex.adoc[DROP INDEX] * xref:n1ql:n1ql-language-reference/dropprimaryindex.adoc[DROP PRIMARY INDEX] +* xref:n1ql:n1ql-language-reference/dropvectorindex.adoc[DROP VECTOR INDEX] == Storage and Availability @@ -52,3 +55,4 @@ endif::[] * xref:learn:services-and-indexes/services/index-service.adoc[Index Service Architecture] * xref:manage:manage-indexes/manage-indexes.adoc[Manage Indexes] * xref:manage:monitor/monitoring-indexes.adoc[Monitor Indexes] +* xref:vector-index:vectors-and-indexes-overview.adoc[] diff --git a/modules/n1ql/assets/images/n1ql-language-reference/alter-vector-index.png b/modules/n1ql/assets/images/n1ql-language-reference/alter-vector-index.png new file mode 100644 index 000000000..129de15f8 Binary files /dev/null and b/modules/n1ql/assets/images/n1ql-language-reference/alter-vector-index.png differ diff --git a/modules/n1ql/assets/images/n1ql-language-reference/create-index.png b/modules/n1ql/assets/images/n1ql-language-reference/create-index.png index 2dc3ec5a0..3efe4d6c5 100644 Binary files a/modules/n1ql/assets/images/n1ql-language-reference/create-index.png and b/modules/n1ql/assets/images/n1ql-language-reference/create-index.png differ diff --git a/modules/n1ql/assets/images/n1ql-language-reference/create-primary-index.png b/modules/n1ql/assets/images/n1ql-language-reference/create-primary-index.png index be7e89910..924efc01f 100644 Binary files a/modules/n1ql/assets/images/n1ql-language-reference/create-primary-index.png and b/modules/n1ql/assets/images/n1ql-language-reference/create-primary-index.png differ diff --git a/modules/n1ql/assets/images/n1ql-language-reference/create-statement.png b/modules/n1ql/assets/images/n1ql-language-reference/create-statement.png index 9abbc3084..04e12d358 100644 Binary files a/modules/n1ql/assets/images/n1ql-language-reference/create-statement.png and b/modules/n1ql/assets/images/n1ql-language-reference/create-statement.png differ diff --git a/modules/n1ql/assets/images/n1ql-language-reference/create-vector-index.png b/modules/n1ql/assets/images/n1ql-language-reference/create-vector-index.png new file mode 100644 index 000000000..242e49a87 Binary files /dev/null and b/modules/n1ql/assets/images/n1ql-language-reference/create-vector-index.png differ diff --git a/modules/n1ql/assets/images/n1ql-language-reference/drop-index.png b/modules/n1ql/assets/images/n1ql-language-reference/drop-index.png index 2558a10c0..e56246bb3 100644 Binary files a/modules/n1ql/assets/images/n1ql-language-reference/drop-index.png and b/modules/n1ql/assets/images/n1ql-language-reference/drop-index.png differ diff --git a/modules/n1ql/assets/images/n1ql-language-reference/drop-primary-index.png b/modules/n1ql/assets/images/n1ql-language-reference/drop-primary-index.png index 4f0d73357..0728aca43 100644 Binary files a/modules/n1ql/assets/images/n1ql-language-reference/drop-primary-index.png and b/modules/n1ql/assets/images/n1ql-language-reference/drop-primary-index.png differ diff --git a/modules/n1ql/assets/images/n1ql-language-reference/drop-statement.png b/modules/n1ql/assets/images/n1ql-language-reference/drop-statement.png index ad8f3791c..6abe46dda 100644 Binary files a/modules/n1ql/assets/images/n1ql-language-reference/drop-statement.png and b/modules/n1ql/assets/images/n1ql-language-reference/drop-statement.png differ diff --git a/modules/n1ql/assets/images/n1ql-language-reference/drop-vector-index.png b/modules/n1ql/assets/images/n1ql-language-reference/drop-vector-index.png new file mode 100644 index 000000000..563e2edda Binary files /dev/null and b/modules/n1ql/assets/images/n1ql-language-reference/drop-vector-index.png differ diff --git a/modules/n1ql/assets/images/n1ql-language-reference/index-include.png b/modules/n1ql/assets/images/n1ql-language-reference/index-include.png new file mode 100644 index 000000000..76b1724d5 Binary files /dev/null and b/modules/n1ql/assets/images/n1ql-language-reference/index-include.png differ diff --git a/modules/n1ql/assets/images/n1ql-language-reference/index-key-and-attrib.png b/modules/n1ql/assets/images/n1ql-language-reference/index-key-and-attrib.png new file mode 100644 index 000000000..2ab3429cc Binary files /dev/null and b/modules/n1ql/assets/images/n1ql-language-reference/index-key-and-attrib.png differ diff --git a/modules/n1ql/assets/images/n1ql-language-reference/index-keys-and-attribs.png b/modules/n1ql/assets/images/n1ql-language-reference/index-keys-and-attribs.png new file mode 100644 index 000000000..c4a4cb185 Binary files /dev/null and b/modules/n1ql/assets/images/n1ql-language-reference/index-keys-and-attribs.png differ diff --git a/modules/n1ql/assets/images/n1ql-language-reference/index-name-on-keyspace.png b/modules/n1ql/assets/images/n1ql-language-reference/index-name-on-keyspace.png new file mode 100644 index 000000000..75a2bd0e8 Binary files /dev/null and b/modules/n1ql/assets/images/n1ql-language-reference/index-name-on-keyspace.png differ diff --git a/modules/n1ql/assets/images/n1ql-language-reference/index-path-and-name.png b/modules/n1ql/assets/images/n1ql-language-reference/index-path-and-name.png new file mode 100644 index 000000000..1f54e77c6 Binary files /dev/null and b/modules/n1ql/assets/images/n1ql-language-reference/index-path-and-name.png differ diff --git a/modules/n1ql/assets/images/n1ql-language-reference/index-vector.png b/modules/n1ql/assets/images/n1ql-language-reference/index-vector.png new file mode 100644 index 000000000..cdc92659e Binary files /dev/null and b/modules/n1ql/assets/images/n1ql-language-reference/index-vector.png differ diff --git a/modules/n1ql/assets/images/n1ql-language-reference/key-attribs.png b/modules/n1ql/assets/images/n1ql-language-reference/key-attribs.png index 91d65dbcc..aecfc15a8 100644 Binary files a/modules/n1ql/assets/images/n1ql-language-reference/key-attribs.png and b/modules/n1ql/assets/images/n1ql-language-reference/key-attribs.png differ diff --git a/modules/n1ql/assets/images/n1ql-language-reference/lead-key-attribs.png b/modules/n1ql/assets/images/n1ql-language-reference/lead-key-attribs.png index d88ddaab3..9e415c3c1 100644 Binary files a/modules/n1ql/assets/images/n1ql-language-reference/lead-key-attribs.png and b/modules/n1ql/assets/images/n1ql-language-reference/lead-key-attribs.png differ diff --git a/modules/n1ql/assets/images/n1ql-language-reference/other-statement.png b/modules/n1ql/assets/images/n1ql-language-reference/other-statement.png index 11e4e38d9..6fc9c571e 100644 Binary files a/modules/n1ql/assets/images/n1ql-language-reference/other-statement.png and b/modules/n1ql/assets/images/n1ql-language-reference/other-statement.png differ diff --git a/modules/n1ql/assets/images/n1ql-language-reference/rr-2.0.png b/modules/n1ql/assets/images/n1ql-language-reference/rr-2.1.png similarity index 100% rename from modules/n1ql/assets/images/n1ql-language-reference/rr-2.0.png rename to modules/n1ql/assets/images/n1ql-language-reference/rr-2.1.png diff --git a/modules/n1ql/pages/n1ql-language-reference/alterindex.adoc b/modules/n1ql/pages/n1ql-language-reference/alterindex.adoc index 59ecd83d0..ac8168b17 100644 --- a/modules/n1ql/pages/n1ql-language-reference/alterindex.adoc +++ b/modules/n1ql/pages/n1ql-language-reference/alterindex.adoc @@ -1,6 +1,8 @@ = ALTER INDEX :description: The ALTER INDEX statement moves the placement of an existing index or replica among different GSI nodes. :page-edition: Enterprise Edition +:page-partial: +:page-toclevels: 2 :imagesdir: ../../assets/images :rebalancing-the-index-service: xref:learn:clusters-and-availability/rebalance.adoc#rebalancing-the-index-service @@ -9,6 +11,7 @@ :identifiers: xref:n1ql-language-reference/identifiers.adoc :logical-hierarchy: xref:n1ql:n1ql-intro/queriesandresults.adoc#logical-hierarchy :querying-indexes: xref:n1ql-intro/sysinfo.adoc#querying-indexes +:alter-vector-index: xref:n1ql-language-reference/altervectorindex.adoc // TEMP include::partial$n1ql-language-reference/horizontal-style.adoc[] @@ -16,9 +19,13 @@ include::partial$n1ql-language-reference/horizontal-style.adoc[] [abstract] {description} +The {alter-vector-index}[ALTER VECTOR INDEX] statement is a synonym for the ALTER INDEX statement. +The two statements have the same functionality. + +//tag::purpose[] == Purpose -You can use the `ALTER INDEX` statement to change the placement of an existing index or replica among different GSI nodes, to increase or decrease the number of replicas, or to drop a specified index replica temporarily. +You can use the `{doctitle}` statement to change the placement of an existing index or replica among different GSI nodes, to increase or decrease the number of replicas, or to drop a specified index replica temporarily. You can also use it to perform the same alterations to a partitioned index and any replica partitions. You may use this statement when you encounter any of the following situations: @@ -28,27 +35,31 @@ You may use this statement when you encounter any of the following situations: * The automated process of rebalancing does not give the expected results. * Other types of scaling up or scaling down are needed. -For example, if a node fails, you can use the `ALTER INDEX` statement to move an index to another node. +For example, if a node fails, you can use the `{doctitle}` statement to move an index to another node. See <> below. -NOTE: The ALTER INDEX move operation is asynchronous. +NOTE: The {doctitle} move operation is asynchronous. As soon as the move alter index command is executed, the command returns. If there is no error in the input, the move operation can be tracked through the console UI and any error can be found in the Console logs and Indexer logs. -If a node goes down while an ALTER INDEX operation is in progress, then the index would rollback to its original node (not affecting queries) and a notification would appear. +If a node goes down while an {doctitle} operation is in progress, then the index would rollback to its original node (not affecting queries) and a notification would appear. IMPORTANT: It is not possible to move an index or index replica and change the number of index replicas at the same time. +//end::purpose[] + +//tag::prerequisites[] == Prerequisites -Only users with the RBAC role of `Administrator` are allowed to run the `ALTER INDEX` directive. +Only users with the RBAC role of `Administrator` are allowed to run the `{doctitle}` statement. -This directive is applicable only for Standard GSI (Plasma) and MOI Indexes; and hence supported only for Enterprise Edition nodes. +This statement is applicable only for Standard GSI (Plasma) and MOI Indexes; and hence supported only for Enterprise Edition nodes. (Not applicable for Forest DB.) -This statement is only supported only in Couchbase Server 5.5 and later; if the cluster is a mix of Couchbase Server 5.5 and previous versions, then this statement will not work. +Altering indexes is only supported in Couchbase Server 5.5 and later; if any nodes in the cluster are running a version of Couchbase Server earlier than 5.5, then this statement will not work. -In addition, altering the number of replicas or deleting an index replica is only supported in Couchbase Server 6.5 and later; if the cluster is a mix of Couchbase Server 6.5 and previous versions, then these operations will not work. +In addition, altering the number of replicas or deleting an index replica is only supported in Couchbase Server 6.5 and later; if any nodes in the cluster are running a version of Couchbase Server earlier than 6.5, then these operations will not work. +//end::prerequisites[] == Syntax @@ -59,25 +70,29 @@ include::partial$grammar/ddl.ebnf[tag=alter-index] image::n1ql-language-reference/alter-index.png["Syntax diagram: refer to source code listing", align=left] -The ALTER INDEX statement provides two possible syntaxes for specifying the index and the keyspace where the index is located. +//tag::syntax-arguments[] +The {doctitle} statement provides two possible syntaxes for specifying the index and the keyspace where the index is located. // TODO: Automatic links in EBNF. +// FIXME: Maybe try to make consistent with DROP INDEX [horizontal] index-name:: (Required) A unique name that identifies the index. index-path:: (Optional) One possible syntax for specifying the keyspace. -Refer to <> below. +See <>. keyspace-ref:: (Optional) The other possible syntax for specifying the keyspace. -Refer to <> below. +See <>. index-using:: (Optional) Specifies the index type. -Refer to <> below. +See <>. index-with:: (Required) Specifies options for the index. -Refer to <> below. +See <>. +//end::syntax-arguments[] +//tag::index-path[] [[index-path]] === Index Path @@ -93,7 +108,7 @@ This syntax provides compatibility with legacy versions of Couchbase Server. The index path may be a <>, a <>, or a <>. NOTE: If there is a hyphen (-) inside the index name or any part of the index path, you must wrap the index name or that part of the index path in backticks ({backtick}{nbsp}{backtick}). -Refer to the examples below. +See the examples on this page. [[keyspace-full-index,full keyspace path]] ==== Index Path: Full Keyspace @@ -172,7 +187,9 @@ collection:: ==== For example, `airline.{backtick}idx-name{backtick}` indicates the `idx-name` index on the `airline` collection, assuming that the query context is set. ==== +//end::index-path[] +//tag::keyspace-ref[] [[keyspace-ref]] === Index Name ON Keyspace Reference @@ -187,7 +204,7 @@ You can use the index name with the `ON` keyword and a keyspace reference to spe The keyspace reference may be a <> or a <>. NOTE: If there is a hyphen (-) inside the index name or any part of the keyspace reference, you must wrap the index name or that part of the keyspace reference in backticks ({backtick}{nbsp}{backtick}). -Refer to the examples below. +See the examples on this page. [[keyspace-path,keyspace path]] ==== Keyspace Reference: Keyspace Path @@ -245,7 +262,9 @@ collection:: ==== For example, `{backtick}idx-name{backtick} ON airline` indicates the `idx-name` index on the `airline` collection, assuming the query context is set. ==== +//end::keyspace-ref[] +//tag::index-using[] [[index-using]] === USING Clause @@ -258,7 +277,9 @@ image::n1ql-language-reference/index-using.png["Syntax diagram: refer to source The index type for a secondary index must be Global Secondary Index (GSI). The `USING GSI` keywords are optional and may be omitted. +//end::index-using[] +//tag::index-with[] [[index-with]] === WITH Clause @@ -330,12 +351,14 @@ __optional__ An integer, specifying a replica ID. | Integer |=== +//end::index-with[] +//tag::usage[] == Usage If you attempt to alter an index which is still scheduled for background creation, the request fails. -`ALTER INDEX` will not work while the cluster is undergoing a rebalance. +The statement will not work while the cluster is undergoing a rebalance. === Moving an Index or Index Replicas @@ -372,16 +395,24 @@ To find the ID of an index replica and see which node it is placed on, you can u When dropping a replica, it is possible to leave a server group with no replica. For a partitioned index, run a rebalance to move a replica into the vacant server group. +=== Index Redistribution + +Using this statement to move one index at a time may be cumbersome if there are a lot of indexes to be moved. +The _index redistribution_ setting enables you to specify how Couchbase Server redistributes indexes automatically on rebalance. +Refer to {rebalancing-the-index-service}[Rebalance] for more information. +//end::usage[] + +//tag::return-value[] == Return Value -If the `ALTER INDEX` succeeds, then: +If the statement succeeds, then: * The query returns an empty array. * The index progress is visible in the Query Workbench. * After the movement is complete, the new indexes begin to service query scans. * The command line displays the new index nodes. -If the `ALTER INDEX` fails, then: +If the statement fails, then: * The original indexes continue to service query scans. * The UI Log and Query Workbench has the appropriate error message. @@ -417,6 +448,7 @@ a| a| * Mistyped the `"action"` |=== +//end::return-value[] [[examples]] == Examples @@ -549,6 +581,14 @@ WITH {"action": "drop_replica", "replicaId": 2}; == Related Links -Using the ALTER INDEX command to move one index at a time may be cumbersome if there are a lot of indexes to be moved. -The _index redistribution_ setting enables you to specify how Couchbase Server redistributes indexes automatically on rebalance. -Refer to {rebalancing-the-index-service}[Rebalance] for more information. +* xref:indexes:indexing-overview.adoc[] +* xref:vector-index:composite-vector-index.adoc[] +* xref:vector-index:hyperscale-vector-index.adoc[] +* xref:n1ql:n1ql-language-reference/createprimaryindex.adoc[CREATE PRIMARY INDEX] +| xref:n1ql:n1ql-language-reference/createindex.adoc[CREATE INDEX] +| xref:n1ql:n1ql-language-reference/createvectorindex.adoc[CREATE VECTOR INDEX] +* xref:n1ql:n1ql-language-reference/build-index.adoc[BUILD INDEX] +* xref:n1ql:n1ql-language-reference/altervectorindex.adoc[ALTER VECTOR INDEX] +* xref:n1ql:n1ql-language-reference/dropprimaryindex.adoc[DROP PRIMARY INDEX] +| xref:n1ql:n1ql-language-reference/dropindex.adoc[DROP INDEX] +| xref:n1ql:n1ql-language-reference/dropvectorindex.adoc[DROP VECTOR INDEX] diff --git a/modules/n1ql/pages/n1ql-language-reference/altervectorindex.adoc b/modules/n1ql/pages/n1ql-language-reference/altervectorindex.adoc new file mode 100644 index 000000000..ae793da71 --- /dev/null +++ b/modules/n1ql/pages/n1ql-language-reference/altervectorindex.adoc @@ -0,0 +1,202 @@ += ALTER VECTOR INDEX +:description: The ALTER VECTOR INDEX statement moves the placement of an existing index or replica among different GSI nodes. +:page-edition: Enterprise Edition +:page-status: Couchbase Server 8.0 +:page-toclevels: 2 +:imagesdir: ../../assets/images + +:rebalancing-the-index-service: xref:learn:clusters-and-availability/rebalance.adoc#rebalancing-the-index-service +:console-indexes: xref:manage:manage-ui/manage-ui.adoc#console-indexes +:query-context: xref:n1ql:n1ql-intro/queriesandresults.adoc#query-context +:identifiers: xref:n1ql-language-reference/identifiers.adoc +:logical-hierarchy: xref:n1ql:n1ql-intro/queriesandresults.adoc#logical-hierarchy +:querying-indexes: xref:n1ql-intro/sysinfo.adoc#querying-indexes +:alter-index: xref:n1ql-language-reference/alterindex.adoc +:prerequisites: xref:vector-index:hyperscale-vector-index.adoc#prerequisites + +// TEMP +include::partial$n1ql-language-reference/horizontal-style.adoc[] + +[abstract] +{description} + +The {alter-index}[ALTER INDEX] statement is a synonym for the ALTER VECTOR INDEX statement. +The two statements have the same functionality. + +include::alterindex.adoc[tags=purpose] + +include::alterindex.adoc[tags=prerequisites] + +== Syntax + +[source,ebnf] +---- +include::partial$grammar/ddl.ebnf[tag=alter-vector-index] +---- + +image::n1ql-language-reference/alter-vector-index.png["Syntax diagram: refer to source code listing", align=left] + +include::alterindex.adoc[tags=syntax-arguments] + +include::alterindex.adoc[tags=index-path] + +include::alterindex.adoc[tags=keyspace-ref] + +include::alterindex.adoc[tags=index-using] + +include::alterindex.adoc[tags=index-with] + +include::alterindex.adoc[tags=usage] + +include::alterindex.adoc[tags=return-value] + +[[examples]] +== Examples + +To try the examples in this section, you must do the following: + +. Create a cluster of 3 nodes. +The examples in this section assume that the 3 nodes have the IP addresses 172.19.0.2, 172.19.0.3, and 172.19.0.4. +The nodes in your cluster may have different names or IP addresses. + +. Install the vector sample data as described in {prerequisites}[Prerequisites]. + +. Set the query context to the `color` scope in the vector sample dataset. +For more information, see xref:n1ql:n1ql-intro/queriesandresults.adoc#query-context[Query Context]. + +.Create and move an index from one node to another +==== +Create a hyperscale vector index on node 172.19.0.2. + +[source,sqlpp] +---- +CREATE VECTOR INDEX hyperscale_idx_move + ON rgb(embedding_vector_dot VECTOR) + WITH {"dimension": 1536, + "similarity": "L2", + "description": "IVF8,SQ4", + "nodes": "172.19.0.2:8091"} +---- + +Then move the index from its original node (172.19.0.*2* in this example) to a new node (172.19.0.*3* in this example). + +[source,sqlpp] +---- +ALTER VECTOR INDEX hyperscale_idx_move ON rgb +WITH {"action": "move", "nodes": ["172.19.0.3:8091"]}; +---- + +To check the node where the index is located, see xref:manage:manage-indexes/manage-indexes.adoc[]. +==== + +.Create and move an index replica from one node to another +==== +Create a hyperscale vector index on node 172.19.0.2 with a replica on node 172.19.0.3, then move its replica from node 172.19.0.*3* to 172.19.0.*4*. + +[source,sqlpp] +---- +CREATE VECTOR INDEX hyperscale_rep_move + ON rgb(embedding_vector_dot VECTOR) + WITH {"dimension": 1536, + "similarity": "L2", + "description": "IVF8,SQ4", + "nodes": ["172.19.0.2:8091", "172.19.0.3:8091"]}; + +ALTER VECTOR INDEX hyperscale_rep_move ON rgb +WITH {"action": "move", "nodes": ["172.19.0.2:8091", "172.19.0.4:8091"]}; +---- +==== + +.Increase the number of replicas +==== +Create a hyperscale vector index on node 172.19.0.2 with a replica on nodes 172.19.0.*3*, then increase the number of replicas to 2 and specify that new replicas may be placed on any available index nodes in the cluster. + +[source,sqlpp] +---- +CREATE VECTOR INDEX hyperscale_rep_multi + ON rgb(embedding_vector_dot VECTOR) + WITH {"dimension": 1536, + "similarity": "L2", + "description": "IVF8,SQ4", + "nodes": ["172.19.0.2:8091", "172.19.0.3:8091"]}; + +ALTER VECTOR INDEX hyperscale_rep_multi ON rgb +WITH {"action": "replica_count", "num_replica": 2}; +---- +==== + +.Increase the number of replicas and specify the nodes +==== +Create a hyperscale vector index on node 172.19.0.2 with a replica on node 172.19.0.3, then increase the number of replicas to 2, and specify that replicas may be placed on nodes 172.19.0.*3* and 172.19.0.*4*. + +[source,sqlpp] +---- +CREATE VECTOR INDEX hyperscale_rep_increase + ON rgb(embedding_vector_dot VECTOR) + WITH {"dimension": 1536, + "similarity": "L2", + "description": "IVF8,SQ4", + "nodes": ["172.19.0.2:8091", "172.19.0.3:8091"]}; + +ALTER VECTOR INDEX hyperscale_rep_increase ON rgb +WITH {"action": "replica_count", + "num_replica": 2, + "nodes": ["172.19.0.2:8091", + "172.19.0.3:8091", + "172.19.0.4:8091"]}; +---- +==== + +.Decrease the number of replicas +==== +Create a hyperscale vector index on node 172.19.0.2 with replicas on nodes 172.19.0.*3* and 172.19.0.*4*, then decrease the number of replicas to 1. + +[source,sqlpp] +---- +CREATE VECTOR INDEX hyperscale_rep_decrease + ON rgb(embedding_vector_dot VECTOR) + WITH {"dimension": 1536, + "similarity": "L2", + "description": "IVF8,SQ4", + "nodes": ["172.19.0.2:8091", "172.19.0.3:8091", "172.19.0.4:8091"]}; + +ALTER VECTOR INDEX hyperscale_rep_decrease ON rgb +WITH {"action": "replica_count", "num_replica": 1}; +---- +==== + +.Drop a specific replica +==== +Create a hyperscale vector index with two replicas, and specify that nodes 172.19.0.2, 172.19.0.3, and 172.19.0.4 should be available for index and replica placement. +Then delete replica 2. + +[source,sqlpp] +---- +CREATE VECTOR INDEX hyperscale_rep_drop + ON rgb(embedding_vector_dot VECTOR) + WITH {"dimension": 1536, + "similarity": "L2", + "description": "IVF8,SQ4", + "num_replica": 2, + "nodes": ["172.19.0.2:8091", + "172.19.0.3:8091", + "172.19.0.4:8091"]}; + +ALTER VECTOR INDEX hyperscale_rep_drop ON rgb +WITH {"action": "drop_replica", "replicaId": 2}; +---- +==== + +== Related Links + +* xref:indexes:indexing-overview.adoc[] +* xref:vector-index:composite-vector-index.adoc[] +* xref:vector-index:hyperscale-vector-index.adoc[] +* xref:n1ql:n1ql-language-reference/createprimaryindex.adoc[CREATE PRIMARY INDEX] +| xref:n1ql:n1ql-language-reference/createindex.adoc[CREATE INDEX] +| xref:n1ql:n1ql-language-reference/createvectorindex.adoc[CREATE VECTOR INDEX] +* xref:n1ql:n1ql-language-reference/build-index.adoc[BUILD INDEX] +* xref:n1ql:n1ql-language-reference/alterindex.adoc[ALTER INDEX] +* xref:n1ql:n1ql-language-reference/dropprimaryindex.adoc[DROP PRIMARY INDEX] +| xref:n1ql:n1ql-language-reference/dropindex.adoc[DROP INDEX] +| xref:n1ql:n1ql-language-reference/dropvectorindex.adoc[DROP VECTOR INDEX] diff --git a/modules/n1ql/pages/n1ql-language-reference/build-index.adoc b/modules/n1ql/pages/n1ql-language-reference/build-index.adoc index bc097c6ed..5da90b935 100644 --- a/modules/n1ql/pages/n1ql-language-reference/build-index.adoc +++ b/modules/n1ql/pages/n1ql-language-reference/build-index.adoc @@ -1,6 +1,7 @@ = BUILD INDEX :description: The BUILD INDEX statement enables you to build one or more GSI indexes that are marked for deferred building all at once. :page-topic-type: reference +:page-toclevels: 2 :imagesdir: ../../assets/images :authorization-overview: xref:learn:security/authorization-overview.adoc @@ -18,8 +19,8 @@ include::partial$n1ql-language-reference/horizontal-style.adoc[] [abstract] {description} -By default, CREATE INDEX starts building the created index after the creation stage is complete. -However for more efficient building of multiple indexes, CREATE INDEX can mark indexes for deferred building using the `defer_build:true` option. +By default, CREATE INDEX or CREATE VECTOR INDEX starts building the created index after the creation stage is complete. +However for more efficient building of multiple indexes, CREATE INDEX or CREATE VECTOR INDEX can mark indexes for deferred building using the `defer_build:true` option. BUILD INDEX is capable of building multiple indexes at once, and can utilize a single scan of documents in the keyspace to feed many index build operations. BUILD INDEX is an asynchronous operation. @@ -51,6 +52,12 @@ If any of the indexes specified by BUILD INDEX have already been built, BUILD IN When building an index which has automatic index replicas, all of the replicas are also built as part of the BUILD INDEX statement, without having to manually specify them. +Hyperscale vector indexes and composite vector indexes require a codebook for the vector field. +The codebook is the result of sampling the dataset and is saved as part of the index metadata. + +The codebook is created as part of the BUILD INDEX process, and is not incrementally updated. +If the data set changes dramatically, you must drop and rebuild the index to update the codebook. + == Prerequisites [discrete] @@ -296,3 +303,17 @@ include::example$n1ql-language-reference/check-idx-online.jsonc[] <1> Note that the index has now been created. ==== + +== Related Links + +* xref:indexes:indexing-overview.adoc[] +* xref:vector-index:composite-vector-index.adoc[] +* xref:vector-index:hyperscale-vector-index.adoc[] +* xref:n1ql:n1ql-language-reference/createprimaryindex.adoc[CREATE PRIMARY INDEX] +| xref:n1ql:n1ql-language-reference/createindex.adoc[CREATE INDEX] +| xref:n1ql:n1ql-language-reference/createvectorindex.adoc[CREATE VECTOR INDEX] +* xref:n1ql:n1ql-language-reference/alterindex.adoc[ALTER INDEX] +| xref:n1ql:n1ql-language-reference/altervectorindex.adoc[ALTER VECTOR INDEX] +* xref:n1ql:n1ql-language-reference/dropprimaryindex.adoc[DROP PRIMARY INDEX] +| xref:n1ql:n1ql-language-reference/dropindex.adoc[DROP INDEX] +| xref:n1ql:n1ql-language-reference/dropvectorindex.adoc[DROP VECTOR INDEX] diff --git a/modules/n1ql/pages/n1ql-language-reference/createindex.adoc b/modules/n1ql/pages/n1ql-language-reference/createindex.adoc index 3ab6c8885..81cbc98c8 100644 --- a/modules/n1ql/pages/n1ql-language-reference/createindex.adoc +++ b/modules/n1ql/pages/n1ql-language-reference/createindex.adoc @@ -1,9 +1,10 @@ = CREATE INDEX :page-topic-type: reference +:page-partial: +:page-toclevels: 2 :imagesdir: ../../assets/images :keywords: secondary, index, placement -:description: The CREATE INDEX statement allows you to create a secondary index. \ -Secondary indexes contain a filtered or a full set of keys in a given keyspace. +:description: The CREATE INDEX statement allows you to create secondary indexes and composite vector indexes. :authorization-overview: xref:learn:security/authorization-overview.adoc :index-replication: xref:indexes:index-replication.adoc#index-replication @@ -13,47 +14,69 @@ Secondary indexes contain a filtered or a full set of keys in a given keyspace. :identifiers: xref:n1ql-language-reference/identifiers.adoc :expression: xref:n1ql-language-reference/index.adoc :aggregatefun: xref:n1ql:n1ql-language-reference/aggregatefun.adoc +:flatten-keys: xref:n1ql:n1ql-language-reference/metafun.adoc#flatten_keys +:base64-decode: xref:n1ql:n1ql-language-reference/metafun.adoc#base64-decode :indexing-arrays: xref:n1ql-language-reference/indexing-arrays.adoc :index-partitioning: xref:n1ql-language-reference/index-partitioning.adoc :indexing-meta-info: xref:n1ql-language-reference/indexing-meta-info.adoc :operator-pushdowns: xref:indexes:index_pushdowns.adoc#operator-pushdowns :logical-hierarchy: xref:n1ql:n1ql-intro/queriesandresults.adoc#logical-hierarchy :querying-indexes: xref:n1ql-intro/sysinfo.adoc#querying-indexes +:create-vector-index: xref:n1ql-language-reference/createvectorindex.adoc +:index-training: xref:vector-index:vectors-and-indexes-overview.adoc#index-training +:composite-vector-index: xref:vector-index:composite-vector-index.adoc +:algo-settings: {composite-vector-index}#algo_settings +:prerequisites: {composite-vector-index}#prerequisites // TEMP include::partial$n1ql-language-reference/horizontal-style.adoc[] -The `CREATE INDEX` statement allows you to create a secondary index. +The `CREATE INDEX` statement allows you to create secondary indexes. Secondary indexes contain a filtered or a full set of keys in a given keyspace. Secondary indexes are optional but increase query efficiency on a keyspace. +In Couchbase Server 8.0 and later, the `CREATE INDEX` statement also allows you to create composite vector indexes. +To create hyperscale vector indexes, use the {create-vector-index}[CREATE VECTOR INDEX] statement. + +//tag::purpose[] == Purpose -`CREATE INDEX` allows you to make multiple concurrent index creation requests. +`{doctitle}` allows you to make multiple concurrent index creation requests. The command starts a task to create the index definition in the background. If there is an index creation task already running, the Index Service queues the incoming index creation request. -`CREATE INDEX` returns as soon as the index creation phase is complete. +`{doctitle}` returns as soon as the index creation phase is complete. By default, when the index creation phase is complete, the Index Service triggers the index build phase. If you lose connectivity, the index build operation continues in the background. You can defer the index build phase using the `defer_build` clause. -In deferred build mode, `CREATE INDEX` creates the index definition, but does not trigger the index build phase. +In deferred build mode, `{doctitle}` creates the index definition, but does not trigger the index build phase. You can then build the index using the {build-index}[BUILD INDEX] command. You can create multiple identical secondary indexes on a keyspace and place them on separate nodes for better index availability. In Couchbase Server Enterprise Edition, the recommended way to do this is using the `num_replica` option. In Couchbase Server Community Edition, you need to create multiple identical indexes and place them using the `nodes` option. -Refer to <> below for more details. +See <> for more details. + +Hyperscale vector indexes and composite vector indexes require a codebook for the vector field. +The codebook is the result of sampling the dataset and is saved as part of the index metadata. +The codebook is created as part of the {build-index}[BUILD INDEX] process, and is not incrementally updated. +If the data set changes dramatically, you must drop and rebuild the index to update the codebook. +//end::purpose[] + +//tag::prerequisites[] == Prerequisites [discrete] ===== RBAC Privileges -User executing the CREATE INDEX statement must have the _Query Manage Index_ privilege granted on the keyspace. +To execute the {doctitle} statement, you must have the _Query Manage Index_ privilege granted on the keyspace. For more details about user roles, see {authorization-overview}[Authorization]. +include::partial$n1ql-language-reference/vector-index-no-windows.adoc[] +//end::prerequisites[] + == Syntax [source,ebnf] @@ -73,29 +96,24 @@ The minimum length of an index name is 1 character and there is no maximum lengt When querying, if the index name contains a `#` or `_` character, you must enclose the index name within backticks. keyspace-ref:: (Required) Specifies the keyspace where the index is created. -Refer to <> below. - -index-key:: (Required) Specifies an index key. -Refer to <> below. - -lead-key-attribs:: (Optional) Specifies attributes for the leading index key. -Refer to <> below. +See <>. -key-attribs:: (Optional) Specifies attributes for a non-leading index key. -Refer to <> below. +index-keys-and-attribs:: (Required) Specifies the index keys and index key attributes. +See <>. index-partition:: (Optional) Specifies index partitions. -Refer to <> below. +See <>. where-clause:: (Optional) Specifies filters for a partial index. -Refer to <> below. +See <>. index-using:: (Optional) Specifies the index type. -Refer to <> below. +See <>. index-with:: (Optional) Specifies options for the index. -Refer to <> below. +See <>. +//tag::if-not-exists[] [[if-not-exists]] === IF NOT EXISTS Clause @@ -105,7 +123,9 @@ If an index with the same name already exists within the specified keyspace, the * If this clause is not present, an error is generated. * If this clause is present, the statement does nothing and completes without error. +//end::if-not-exists[] +//tag::keyspace-ref[] [[keyspace-ref]] === Keyspace Reference @@ -120,7 +140,7 @@ Specifies the keyspace for which the index needs to be created. The keyspace reference may be a <> or a <>. NOTE: If there is a hyphen (-) inside any part of the keyspace reference, you must wrap that part of the keyspace reference in backticks ({backtick}{nbsp}{backtick}). -Refer to the examples below. +See the examples on this page. [[keyspace-path,keyspace path]] ==== Keyspace Path @@ -178,9 +198,33 @@ collection:: ==== For example, `airline` indicates the `airline` collection, assuming the query context is set. ==== +//end::keyspace-ref[] + +[[index-keys-and-attribs]] +=== Index Keys and Attributes + +[source,ebnf] +---- +include::partial$grammar/ddl.ebnf[tag=index-keys-and-attribs] +---- + +image::n1ql-language-reference/index-keys-and-attribs.png["Syntax diagram: refer to source code listing", align=left] + +Secondary indexes and composite vector indexes can have many keys. +Each key may have index attributes, which define the behavior of the index key. + +[horizontal] +index-key:: (Required) Specifies an index key. +See <>. + +lead-key-attribs:: (Optional) Specifies attributes for the leading index key. +See <>. + +key-attribs:: (Optional) Specifies attributes for a non-leading index key. +See <>. [[index-key]] -=== Index Key +==== Index Key [source,ebnf] ---- @@ -189,21 +233,30 @@ include::partial$grammar/ddl.ebnf[tag=index-key] image::n1ql-language-reference/index-key.png["Syntax diagram: refer to source code listing", align=left] -Refers to an attribute name or a scalar function or an ARRAY expression on the attribute. -This constitutes an index-key for the index. +The index key is a {sqlpp} {expression}[expression] referring to a field in the document, or an ARRAY expression on the field. + +For a composite vector index, one index key must refer to a vector field in the document. +The index key that refers to a vector field may be the only index key. +If there are multiple index keys, the index key referring to the vector field may be any of the index keys, including the leading index key. [horizontal#index-key-args] expr:: -A {sqlpp} {expression}[expression] over any fields in the document. +A field name or a scalar function over any field in the document. This cannot use constant expressions, aggregate functions, or sub-queries. ++ +For a vector field, the expression may be the field name, or a {base64-decode}[BASE64_DECODE()] function on the vector field -- +this is necessary if the embedded vectors are stored as a base64-encoded string. array-expr:: An array expression. Array indexing enables you to create global indexes on array elements and optimize the execution of queries involving array elements. -For details, refer to {indexing-arrays}[Array Indexing]. ++ +For a vector field, only ALL ARRAY is supported. +The {flatten-keys}[FLATTEN_KEYS()] function is supported, but more than one key in {flatten-keys}[FLATTEN_KEYS()] is not permitted. +For details, see {indexing-arrays}[]. [[index-key-attrib]] -=== Index Key Attributes +==== Index Key Attributes [source,ebnf] ---- @@ -222,14 +275,17 @@ image::n1ql-language-reference/key-attribs.png["Syntax diagram: refer to source Specifies attributes for the index key. [horizontal] -index-order:: (Optional) All index keys may include an index order clause. -Refer to <> below. +index-order:: (Optional) Any index key on a non-vector field may include an index order clause. +See <>. + +include-missing:: (Optional) If the leading index key is a non-vector field, it may also include the `INCLUDE MISSING` clause. +See <>. -include-missing:: (Optional) The leading index key may also include `INCLUDE MISSING` clause. -Refer to <> below. +include-vector:: (Optional) In a composite vector index, one index key must include the `VECTOR` keyword. +See <>. [[index-order]] -==== Index Order +===== Index Order [source,ebnf] ---- @@ -239,6 +295,7 @@ include::partial$grammar/ddl.ebnf[tag=index-order] image::n1ql-language-reference/index-order.png["Syntax diagram: refer to source code listing", align=left] Specifies the sort order of the index key. +For non-vector fields only. [horizontal] `ASC`:: @@ -250,7 +307,7 @@ The index key is sorted in descending order. This clause is optional; if omitted, the default is `ASC`. [[include-missing]] -==== INCLUDE MISSING Clause +===== INCLUDE MISSING Clause [source,ebnf] ---- @@ -262,16 +319,35 @@ image::n1ql-language-reference/include-missing.png["Syntax diagram: refer to sou The optional `INCLUDE MISSING` clause ensures that documents which do not include the index key field are indexed regardless. If this clause is not present, then documents without the index key field are not indexed. -The `INCLUDE MISSING` clause can only be applied to the leading index key. +The `INCLUDE MISSING` clause can only be applied to the leading index key for non-vector fields. The `INCLUDE MISSING` clause may be included before or after the `ASC` or `DESC` keyword. +[[include-vector]] +===== VECTOR Keyword + +[source,ebnf] +---- +include::partial$grammar/ddl.ebnf[tag=index-vector] +---- + +image::n1ql-language-reference/index-vector.png["Syntax diagram: refer to source code listing", align=left] + +Indicates that the index key is a vector field. + +//tag::index-partition[] [[index-partition]] === PARTITION BY HASH Clause Used to partition the index. Index partitioning helps increase the query performance by dividing and spreading a large index of documents across multiple nodes, horizontally scaling out an index as needed. -For details, refer to {index-partitioning}[Index Partitioning]. +For details, see {index-partitioning}[Index Partitioning]. +With hyperscale vector indexes and composite vector indexes, training is done for each index node independently, and the codebook is provided to all partitions on that node. +If there are multiple partitions for an index on a node, training is only done once for all partitions. +See {index-training}[The Importance of Index Training]. +//end::index-partition[] + +//tag::where-clause[] [[where-clause]] === WHERE Clause @@ -285,7 +361,9 @@ image::n1ql-language-reference/where-clause.png["Syntax diagram: refer to source [horizontal#where-clause-args] cond:: Specifies WHERE clause predicates to qualify the subset of documents to include in the index. +//end::where-clause[] +//tag::index-using[] [[index-using]] === USING Clause @@ -298,7 +376,9 @@ image::n1ql-language-reference/index-using.png["Syntax diagram: refer to source The index type for a secondary index must be Global Secondary Index (GSI). The `USING GSI` keywords are optional and may be omitted. +//end::index-using[] +//tag::index-with[] [[index-with]] === WITH Clause @@ -334,7 +414,7 @@ The `nodes` property allows you to specify the node that the index is placed on. [.edition]#{enterprise}# In Couchbase Server Enterprise Edition, you can specify multiple nodes to distribute replicas of an index across nodes running the indexing service: for example, `WITH {"nodes": ["node1:8091", "node2:8091", "node3:8091"]}`. -For details and examples, refer to {index-replication}[Index Replication]. +For details and examples, see {index-replication}[Index Replication]. If specifying both [.var]`nodes` and [.var]`num_replica`, the number of nodes in the array must be one greater than the specified number of replicas otherwise the index creation will fail. @@ -376,12 +456,101 @@ If the value of this property is not less than the number of index nodes in the **Default:** `1` |Integer +//end::index-with[] +|=== + +Composite vector indexes support the following additional options. + +[options="header", cols="1a,4a,1a"] +|=== +|Name|Description|Schema +//tag::index-with[] + +|**dimension** + +__required__ +| The number of dimensions in the vector. +The embedded model you use to embed the vectors determines the number of dimensions in the vector. +|Integer + +|**similarity** + +__required__ +| Sets the distance metric to use when comparing vectors during index creation. + +Couchbase {product-name} uses the following strings to represent the distance metrics: + +[horizontal.compact] +COSINE:: xref:vector-index:vectors-and-indexes-overview.adoc#cosine[Cosine Similarity] +DOT:: xref:vector-index:vectors-and-indexes-overview.adoc#dot[Dot Product] +L2:: +EUCLIDEAN:: xref:vector-index:vectors-and-indexes-overview.adoc#euclidean[Euclidean Distance] +L2_SQUARED:: +EUCLIDEAN_SQUARED:: xref:vector-index:vectors-and-indexes-overview.adoc#euclidean-squared[Euclidean Squared Distance] + +For the greatest accuracy, use the distance metric you plan to use to query the data. +|String + +|**description** + +__required__ +| The settings for the quantization and index algorithms. +The string is made up of the following settings: + +[horizontal.compact] +IVF:: The number of centroids allocated for the index. +SQ:: For scalar quantization -- the number of bits used to store the centroid for each bin. +PQ:: For product quantization -- the number of subquantizers, and the number of bits in the centroid's index value. + +For more information, see {algo-settings}[Quantization and Centroid Settings]. + +**Pattern:** `^IVF[0-9]*,(SQ[468]|PQ[0-9]+x[0-9]+)$` +|String + +|**scan_nprobes** + +__optional__ +| The number of cells to search for each scan. + +**Default:** `1` +|Integer + +|**train_list** + +__optional__ +| The size of the sample set of vectors to be used for index training. + +If the index count is <{nbsp}10000, the default is to sample everything. + +Otherwise, the default value is 10% of the index count, or 10{nbsp}× the number of centroids, whichever is higher. + +**Maximum:** `1000000` +|Integer + +ifeval::["{docname}" == "createvectorindex"] +|**persist_full_vector** + +__optional__ +| If `true` (the default), the index stores the full vector value in addition to the quantized value. +This significantly increases the size of the index. + +Set to `false` if you do not plan to use reranking. +For more information, see {hyperscale-reranking}[]. + +**Default:** `true` +|Boolean +endif::[] |=== +Partitioned indexes support further options. +See {index-partitioning}[]. +//end::index-with[] + +//tag::usage[] == Usage -NOTE: It is not recommended to create (or drop) secondary indexes when any node with a secondary index role is down, as this may result in duplicate index names. +[IMPORTANT] +.Attention +==== +Do not create (or drop) secondary indexes, composite vector indexes, or hyperscale vector indexes when any Index service node is down, as this may result in duplicate index names. +==== +//end::usage[] +//tag::monitoring-indexes[] === Monitoring Indexes Index metadata provides a state field. @@ -402,19 +571,21 @@ include::example$n1ql-language-reference/build-idx-error.jsonc[] If the Index Service still cannot create the index after the maximum number of retries, the index state is marked as `offline`. You must drop the failed index using the `DROP INDEX` command. ==== +//end::monitoring-indexes[] [[indexing-metadata]] -=== Using the `meta().id` Function +=== Indexing Metadata -You can create indexes on metadata information. -For details, refer to {indexing-meta-info}[Indexing Meta Info]. +You can create indexes on metadata using the `META()` function. +For details, see {indexing-meta-info}[]. [[index-aggregates]] === Using Indexes for Aggregates If you have an index on a simple expression, such as `geo.alt`, you can use that index to satisfy a query on an {aggregatefun}[aggregate] of that expression, such as `MIN(geo.alt)` or `MAX(geo.alt)`. -For details and examples, refer to {operator-pushdowns}[Operator Pushdowns]. +For details and examples, see {operator-pushdowns}[Operator Pushdowns]. +//tag::index-replicas[] [[index-replicas]] === Index Replicas @@ -433,6 +604,11 @@ The system automatically load-balances an index scan across the index and all it Adding index replicas enables you to scale scan throughput, in addition to providing high availability. ==== +With hyperscale vector indexes and composite vector indexes, training is done by each replica index independently, and the codebook is stored as part of index metadata. +See {index-training}[The Importance of Index Training]. +//end::index-replicas[] + +//tag::defer-index-builds-by-default[] === Defer Index Builds by Default Usually, the default setting for the `defer_build` option is `false`. @@ -442,6 +618,7 @@ If you change the default setting for `defer_build` to `true`, index creation op To change the default setting for deferred builds, use the Index Settings REST API to set the `indexer.settings.defer_build` property. For an example, see xref:index-rest-settings:index.adoc#ex-defer-build[Defer Index Builds by Default]. +//end::defer-index-builds-by-default[] == Examples @@ -547,6 +724,48 @@ The following statement will index all airports, even if the `district` field is ---- include::example$n1ql-language-reference/create-idx-include.n1ql[] ---- -==== For more examples of indexes where the leading key may be missing, see xref:n1ql:n1ql-language-reference/selectintro.adoc#index-selection[Index Selection]. +==== + +[[eex-create-rgb-idx]] +.Create a composite vector index +==== +For this example, you must install the vector sample data as described in {prerequisites}[Prerequisites]. +The path to the required keyspace is specified by the query, so you do not need to set the query context. + +Create a composite vector index that indexes the vector field named `colorvect_l2`, as well as the scalar `color` and `brightness` fields. + +[source,sqlpp] +---- +include::vector-index:example$gsi-vector-idx-examples.sqlpp[tag=create-vectors-idx] +---- +==== + +[[ex-create-vectors-idx]] +.Create a composite vector index using embedded vectors +==== +For this example, you must install the vector sample data as described in {prerequisites}[Prerequisites]. +The path to the required keyspace is specified by the query, so you do not need to set the query context. + +Create a composite vector index that indexes the vector field named `embedding-vector-dot`, as well as the scalar `color` and `brightness` fields. + +[source,sqlpp] +---- +include::vector-index:example$gsi-vector-idx-examples.sqlpp[tag=create-vectors-idx] +---- +==== + +== Related Links + +* xref:indexes:indexing-overview.adoc[] +* xref:vector-index:composite-vector-index.adoc[] +* xref:vector-index:hyperscale-vector-index.adoc[] +* xref:n1ql:n1ql-language-reference/createprimaryindex.adoc[CREATE PRIMARY INDEX] +| xref:n1ql:n1ql-language-reference/createvectorindex.adoc[CREATE VECTOR INDEX] +* xref:n1ql:n1ql-language-reference/build-index.adoc[BUILD INDEX] +* xref:n1ql:n1ql-language-reference/alterindex.adoc[ALTER INDEX] +| xref:n1ql:n1ql-language-reference/altervectorindex.adoc[ALTER VECTOR INDEX] +* xref:n1ql:n1ql-language-reference/dropprimaryindex.adoc[DROP PRIMARY INDEX] +| xref:n1ql:n1ql-language-reference/dropindex.adoc[DROP INDEX] +| xref:n1ql:n1ql-language-reference/dropvectorindex.adoc[DROP VECTOR INDEX] diff --git a/modules/n1ql/pages/n1ql-language-reference/createprimaryindex.adoc b/modules/n1ql/pages/n1ql-language-reference/createprimaryindex.adoc index aa75fab1b..1b4236002 100644 --- a/modules/n1ql/pages/n1ql-language-reference/createprimaryindex.adoc +++ b/modules/n1ql/pages/n1ql-language-reference/createprimaryindex.adoc @@ -1,5 +1,6 @@ = CREATE PRIMARY INDEX :page-topic-type: reference +:page-toclevels: 2 :imagesdir: ../../assets/images :keywords: primary, index, placement :description: The CREATE PRIMARY INDEX statement allows you to create a primary index. \ @@ -19,7 +20,7 @@ include::partial$n1ql-language-reference/horizontal-style.adoc[] The `CREATE PRIMARY INDEX` statement allows you to create a primary index. Primary indexes contain a full set of keys in a given keyspace. -Primary indexes are optional and are only required for running ad hoc queries on a keyspace that is not supported by a secondary index. +Primary indexes are optional -- they enable you to run ad hoc queries on a keyspace that is not supported by a secondary index. == Purpose @@ -37,14 +38,14 @@ You can then build the index using the {build-index}[BUILD INDEX] command. You can create multiple identical primary indexes on a keyspace and place them on separate nodes for better index availability. In Couchbase Server Enterprise Edition, the recommended way to do this is using the `num_replicas` option. In Couchbase Server Community Edition, you need to create multiple identical indexes and place them using the `nodes` option. -Refer to <> below for more details. +See <> for more details. == Prerequisites [discrete] ===== RBAC Privileges -Users executing the `CREATE PRIMARY INDEX` statement must have the _Query Manage Index_ privilege granted on the keyspace. +To execute the `CREATE PRIMARY INDEX` statement, you must have the _Query Manage Index_ privilege granted on the keyspace. For more details about user roles, see {authorization-overview}[Authorization]. @@ -62,23 +63,20 @@ image::n1ql-language-reference/create-primary-index.png["Syntax diagram: refer t [horizontal] index-name:: (Optional) A unique name that identifies the index. -If a name is not specified, the default name of `#primary` is applied. [small]##[<>]## +If a name is not specified, the default name of `#primary` is applied. + Valid GSI index names can contain any of the following characters: `A-Z` `a-z` `0-9` `#` `_`, and must start with a letter, [`A-Z` `a-z`]. The minimum length of an index name is 1 character and there is no maximum length set for an index name. When querying, if the index name contains a `#` or `_` character, you must enclose the index name within backticks. keyspace-ref:: [Required] Specifies the keyspace where the index is created. -Refer to <> below. +See <>. index-using:: (Optional) Specifies the index type. -Refer to <> below. +See <>. index-with:: (Optional) Specifies options for the index. -Refer to <> below. - -[[note-index-name,note]] -NOTE: You can drop unnamed primary indexes using the `DROP PRIMARY INDEX` statement, and named primary indexes using the `DROP INDEX` statement. +See <>. [[if-not-exists]] === IF NOT EXISTS Clause @@ -104,7 +102,7 @@ Specifies the keyspace for which the index needs to be created. The keyspace reference may be a <> or a <>. NOTE: If there is a hyphen (-) inside any part of the keyspace reference, you must wrap that part of the keyspace reference in backticks ({backtick}{nbsp}{backtick}). -Refer to the examples below. +See the examples on this page. [[keyspace-path,keyspace path]] ==== Keyspace Path @@ -361,3 +359,17 @@ include::example$n1ql-language-reference/check-pri-online.n1ql[] The output from `system:indexes` shows that the index has now been created. ==== + +== Related Links + +* xref:indexes:indexing-overview.adoc[] +* xref:vector-index:composite-vector-index.adoc[] +* xref:vector-index:hyperscale-vector-index.adoc[] +* xref:n1ql:n1ql-language-reference/createindex.adoc[CREATE INDEX] +| xref:n1ql:n1ql-language-reference/createvectorindex.adoc[CREATE VECTOR INDEX] +* xref:n1ql:n1ql-language-reference/build-index.adoc[BUILD INDEX] +* xref:n1ql:n1ql-language-reference/alterindex.adoc[ALTER INDEX] +| xref:n1ql:n1ql-language-reference/altervectorindex.adoc[ALTER VECTOR INDEX] +* xref:n1ql:n1ql-language-reference/dropprimaryindex.adoc[DROP PRIMARY INDEX] +| xref:n1ql:n1ql-language-reference/dropindex.adoc[DROP INDEX] +| xref:n1ql:n1ql-language-reference/dropvectorindex.adoc[DROP VECTOR INDEX] diff --git a/modules/n1ql/pages/n1ql-language-reference/createvectorindex.adoc b/modules/n1ql/pages/n1ql-language-reference/createvectorindex.adoc new file mode 100644 index 000000000..6174e2db0 --- /dev/null +++ b/modules/n1ql/pages/n1ql-language-reference/createvectorindex.adoc @@ -0,0 +1,231 @@ += CREATE VECTOR INDEX +:page-topic-type: reference +:page-status: Couchbase Server 8.0 +:page-toclevels: 2 +:imagesdir: ../../assets/images +:keywords: secondary, index, placement +:description: The CREATE VECTOR INDEX statement allows you to create hyperscale vector indexes. + +:authorization-overview: xref:learn:security/authorization-overview.adoc +:index-replication: xref:indexes:index-replication.adoc#index-replication +:console-indexes: xref:manage:manage-ui/manage-ui.adoc#console-indexes +:query-context: xref:n1ql:n1ql-intro/queriesandresults.adoc#query-context +:build-index: xref:n1ql-language-reference/build-index.adoc +:identifiers: xref:n1ql-language-reference/identifiers.adoc +:expression: xref:n1ql-language-reference/index.adoc +:aggregatefun: xref:n1ql:n1ql-language-reference/aggregatefun.adoc +:flatten-keys: xref:n1ql:n1ql-language-reference/metafun.adoc#flatten_keys +:base64-decode: xref:n1ql:n1ql-language-reference/metafun.adoc#base64-decode +:indexing-arrays: xref:n1ql-language-reference/indexing-arrays.adoc +:index-partitioning: xref:n1ql-language-reference/index-partitioning.adoc +:indexing-meta-info: xref:n1ql-language-reference/indexing-meta-info.adoc +:operator-pushdowns: xref:indexes:index_pushdowns.adoc#operator-pushdowns +:logical-hierarchy: xref:n1ql:n1ql-intro/queriesandresults.adoc#logical-hierarchy +:querying-indexes: xref:n1ql-intro/sysinfo.adoc#querying-indexes +:create-index: xref:n1ql-language-reference/createindex.adoc +:index-training: xref:vector-index:vectors-and-indexes-overview.adoc#index-training +:hyperscale-vector-index: xref:vector-index:hyperscale-vector-index.adoc +:hyperscale-filter: xref:vector-index:hyperscale-filter.adoc +:hyperscale-reranking: xref:vector-index:hyperscale-reranking.adoc +:algo-settings: {hyperscale-vector-index}#algo_settings +:prerequisites: {hyperscale-vector-index}#prerequisites + +// TEMP +include::partial$n1ql-language-reference/horizontal-style.adoc[] + +The `CREATE VECTOR INDEX` statement allows you to create hyperscale vector indexes. + +To create secondary indexes or composite vector indexes, use the {create-index}[CREATE INDEX] statement. + +include::createindex.adoc[tags=purpose] + +include::createindex.adoc[tags=prerequisites] + +== Syntax + +[source,ebnf] +---- +include::partial$grammar/ddl.ebnf[tag=create-vector-index] +---- + +image::n1ql-language-reference/create-vector-index.png["Syntax diagram: refer to source code listing", align=left] + +// TODO: Automatic links in EBNF. + +[horizontal#index-name, reftext="index-name"] +index-name:: (Required) A unique name that identifies the index. ++ +Valid GSI index names can contain any of the following characters: `A-Z` `a-z` `0-9` `#` `_`, and must start with a letter, [`A-Z` `a-z`]. +The minimum length of an index name is 1 character and there is no maximum length set for an index name. +When querying, if the index name contains a `#` or `_` character, you must enclose the index name within backticks. + +keyspace-ref:: (Required) Specifies the keyspace where the index is created. +See <>. + +index-key-and-attrib:: (Required) Specifies the index key and index key attribute. +See <>. + +index-include:: (Optional) Specifies non-key fields to include in the index. +See <>. + +index-partition:: (Optional) Specifies index partitions. +See <>. + +where-clause:: (Optional) Specifies filters for a partial index. +See <>. + +index-using:: (Optional) Specifies the index type. +See <>. + +index-with:: (Optional) Specifies options for the index. +See <>. + +include::createindex.adoc[tags=if-not-exists] + +include::createindex.adoc[tags=keyspace-ref] + +[[index-key-and-attrib]] +=== Index Key and Attribute + +[source,ebnf] +---- +include::partial$grammar/ddl.ebnf[tags=index-key-and-attrib] +---- + +image::n1ql-language-reference/index-key-and-attrib.png["Syntax diagram: refer to source code listing", align=left] + +Hyperscale vector indexes only have one key, which must be a vector field. +The index key takes one attribute, the VECTOR keyword. + +[horizontal] +index-key:: (Required) Specifies an index key. +See <>. + +index-vector:: (Required) Specifies an attribute for the index key. +See <>. + +[[index-key]] +==== Index Key + +[source,ebnf] +---- +include::partial$grammar/ddl.ebnf[tag=index-key] +---- + +image::n1ql-language-reference/index-key.png["Syntax diagram: refer to source code listing", align=left] + +The index key is a {sqlpp} {expression}[expression] referring to a vector field, or an ARRAY expression on the vector field. + +[horizontal#index-key-args] +expr:: +The name of a vector field in the document, or a {base64-decode}[BASE64_DECODE()] function on the vector field -- +this is necessary if the embedded vectors are stored as a base64-encoded string. + +array-expr:: +An array expression on a vector field in the document. +Only ALL ARRAY is supported. +The {flatten-keys}[FLATTEN_KEYS()] function is supported, but more than one key in {flatten-keys}[FLATTEN_KEYS()] is not permitted. +For details, see {indexing-arrays}[]. + +[[include-vector]] +==== VECTOR Keyword + +[source,ebnf] +---- +include::partial$grammar/ddl.ebnf[tag=index-vector] +---- + +image::n1ql-language-reference/index-vector.png["Syntax diagram: refer to source code listing", align=left] + +Indicates that the index key is a vector field. + +[[index-include]] +=== INCLUDE Clause + +[source,ebnf] +---- +include::partial$grammar/ddl.ebnf[tag=index-include] +---- + +image::n1ql-language-reference/index-include.png["Syntax diagram: refer to source code listing", align=left] + +Used to include scalar fields in the index, which you can use to filter the vector search. +The INCLUDE clause cannot include a vector field. +For details, see {hyperscale-filter}[]. + +[horizontal#index-include-args] +expr:: +A {sqlpp} {expression}[expression] referring to any scalar field in the document. + +include::createindex.adoc[tags=index-partition] + +include::createindex.adoc[tags=where-clause] + +include::createindex.adoc[tags=index-using] + +include::createindex.adoc[tags=index-with] + +include::createindex.adoc[tags=usage] + +include::createindex.adoc[tags=monitoring-indexes] + +include::createindex.adoc[tags=index-replicas] + +include::createindex.adoc[tags=defer-index-builds-by-default] + +== Examples + +To try the examples in this section, you must install the vector sample data as described in {prerequisites}[Prerequisites]. + +[[ex-create-rgb-idx]] +.Create a hyperscale vector index +==== +For this example, the path to the required keyspace is specified by the query, so you do not need to set the query context. + +Create a hyperscale vector index for the vector column named `embedding-vector-dot`. + +[source,sqlpp] +---- +include::vector-index:example$hyperscale-idx-examples.sqlpp[tag=create-rgb-idx] +---- +==== + +[[ex-create-idx-brightness]] +.Create a hyperscale vector index with included scalar values +==== +For this example, the path to the required keyspace is specified by the query, so you do not need to set the query context. + +Create a hyperscale vector index for the vector column named `embedding-vector-dot`, including the scalar `brightness` field. + +[source,sqlpp] +---- +include::vector-index:example$hyperscale-idx-examples.sqlpp[tag=create-idx-brightness] +---- +==== + +[[ex-create-rgb-no-persist]] +.Create a hyperscale vector index with no reranking +==== +For this example, the path to the required keyspace is specified by the query, so you do not need to set the query context. + +Create a hyperscale vector index from the example RGB dataset that does not persist the full vector value. + +[source,sqlpp] +---- +include::vector-index:example$hyperscale-idx-examples.sqlpp[tag=create-rgb-no-persist] +---- +==== + +== Related Links + +* xref:indexes:indexing-overview.adoc[] +* xref:vector-index:composite-vector-index.adoc[] +* xref:vector-index:hyperscale-vector-index.adoc[] +* xref:n1ql:n1ql-language-reference/createprimaryindex.adoc[CREATE PRIMARY INDEX] +| xref:n1ql:n1ql-language-reference/createindex.adoc[CREATE INDEX] +* xref:n1ql:n1ql-language-reference/build-index.adoc[BUILD INDEX] +* xref:n1ql:n1ql-language-reference/alterindex.adoc[ALTER INDEX] +| xref:n1ql:n1ql-language-reference/altervectorindex.adoc[ALTER VECTOR INDEX] +* xref:n1ql:n1ql-language-reference/dropprimaryindex.adoc[DROP PRIMARY INDEX] +| xref:n1ql:n1ql-language-reference/dropindex.adoc[DROP INDEX] +| xref:n1ql:n1ql-language-reference/dropvectorindex.adoc[DROP VECTOR INDEX] diff --git a/modules/n1ql/pages/n1ql-language-reference/dropindex.adoc b/modules/n1ql/pages/n1ql-language-reference/dropindex.adoc index f0032a4bd..c8ccc5379 100644 --- a/modules/n1ql/pages/n1ql-language-reference/dropindex.adoc +++ b/modules/n1ql/pages/n1ql-language-reference/dropindex.adoc @@ -1,6 +1,8 @@ = DROP INDEX -:description: The DROP INDEX statement allows you to drop a named primary index or a secondary index. +:description: The DROP INDEX statement allows you to drop a secondary index, a composite vector index, or a hyperscale vector index. :page-topic-type: reference +:page-partial: +:page-toclevels: 2 :imagesdir: ../../assets/images :authorization-overview: xref:learn:security/authorization-overview.adoc @@ -8,22 +10,31 @@ :logical-hierarchy: xref:n1ql-intro/queriesandresults.adoc#logical-hierarchy :identifiers: xref:n1ql-language-reference/identifiers.adoc :drop-primary-index: xref:n1ql-language-reference/dropprimaryindex.adoc +:drop-vector-index: xref:n1ql-language-reference/dropvectorindex.adoc // TEMP include::partial$n1ql-language-reference/horizontal-style.adoc[] {description} -Dropping an index that has replicas will also drop all of the replica indexes too. -You can drop unnamed primary indexes using the {drop-primary-index}[DROP PRIMARY INDEX] statement. +Dropping an index that has replicas will drop all of the replica indexes too. +The {drop-vector-index}[DROP VECTOR INDEX] statement is a synonym for the DROP INDEX statement. +The two statements have the same functionality. + +[NOTE] +To drop a primary index, use the {drop-primary-index}[DROP PRIMARY INDEX] statement. +For compatibility with legacy versions of Couchbase Server, you can also use DROP INDEX or DROP VECTOR INDEX to drop a named primary index. + +// tag::prerequisites[] == Prerequisites [discrete] ===== RBAC Privileges -User executing the DROP INDEX statement must have the _Query Manage Index_ privilege granted on the keyspace/bucket. +To use the {doctitle} statement, you must have the _Query Manage Index_ privilege on the keyspace or bucket. For more details about user roles, see {authorization-overview}[Authorization]. +// end::prerequisites == Syntax @@ -34,34 +45,46 @@ include::partial$grammar/ddl.ebnf[tag=drop-index] image::n1ql-language-reference/drop-index.png["Syntax diagram: refer to source code listing", align=left] -The DROP INDEX statement provides two possible syntaxes for specifying the index and the keyspace where the index is located. +//tag::syntax-arguments[] +The {doctitle} statement provides two possible syntaxes for specifying the index and the keyspace where the index is located. // TODO: Automatic links in EBNF. [horizontal] -index-name:: (Required) A unique name that identifies the index. +index-path-and-name:: (Optional) One possible syntax for specifying the index and keyspace. +See <>. -index-path:: (Optional) One possible syntax for specifying the the keyspace. -Refer to <> below. +index-name-on-keyspace:: (Optional) The other possible syntax for specifying the index and keyspace. +See <>. -keyspace-ref:: (Optional) The other possible syntax for specifying the keyspace. -Refer to <> below. +index-using:: (Optional) Specifies the index type. +See <>. +//end::syntax-arguments[] -index-using:: Specifies the index type. -(Optional) Refer to <> below. +//tag::index-path-and-name[] +[[index-path-and-name]] +=== Index Path and Name -[[if-exists]] -=== IF EXISTS Clause +[source,ebnf] +---- +include::partial$grammar/ddl.ebnf[tag=index-path-and-name] +---- -The optional `IF EXISTS` clause enables the statement to complete successfully when the specified index doesn't exist. -If the index does not exist within the specified keyspace, then: +image::n1ql-language-reference/index-path-and-name.png["Syntax diagram: refer to source code listing", align=left] -* If this clause is not present, an error is generated. +You can use a dotted notation to specify the index and the keyspace on which the index is built. +This syntax provides compatibility with legacy versions of Couchbase Server. -* If this clause is present, the statement does nothing and completes without error. +[horizontal] +index-name:: (Required) A unique name that identifies the index. + +index-path:: (Required) See <>. + +NOTE: If there is a hyphen (-) inside the index name or any part of the index path, you must wrap the index name or that part of the index path in backticks ({backtick}{nbsp}{backtick}). +See the examples on this page. [[index-path]] -=== Index Path +==== Index Path [source,ebnf] ---- @@ -70,15 +93,10 @@ include::partial$grammar/ddl.ebnf[tag=index-path] image::n1ql-language-reference/index-path.png["Syntax diagram: refer to source code listing", align=left] -You can use a dotted notation to specify the index and the keyspace on which the index is built. -This syntax provides compatibility with legacy versions of Couchbase Server. The index path may be a <>, a <>, or a <>. -NOTE: If there is a hyphen (-) inside the index name or any part of the index path, you must wrap the index name or that part of the index path in backticks ({backtick}{nbsp}{backtick}). -Refer to the examples below. - [[keyspace-full-index,full keyspace path]] -==== Index Path: Full Keyspace +===== Index Path: Full Keyspace [source,ebnf] ---- @@ -109,7 +127,7 @@ For example, `default:{backtick}travel-sample{backtick}.inventory.airline.{backt ==== [[keyspace-prefix-index,keyspace prefix]] -==== Index Path: Keyspace Prefix +===== Index Path: Keyspace Prefix [source,ebnf] ---- @@ -135,7 +153,7 @@ For example, `default:{backtick}travel-sample{backtick}.def_type` indicates the ==== [[keyspace-partial-index,keyspace partial]] -==== Index Path: Keyspace Partial +===== Index Path: Keyspace Partial [source,ebnf] ---- @@ -155,9 +173,29 @@ collection:: For example, `airline.{backtick}idx-name{backtick}` indicates the `idx-name` index on the `airline` collection, assuming that the query context is set. ==== -[[keyspace-ref]] +[[index-name-on-keyspace]] === Index Name ON Keyspace Reference +[source,ebnf] +---- +include::partial$grammar/ddl.ebnf[tags=index-name-on-keyspace] +---- + +image::n1ql-language-reference/index-name-on-keyspace.png["Syntax diagram: refer to source code listing", align=left] + +You can use the index name with the `ON` keyword and a keyspace reference to specify the index and the keyspace on which the index is built. + +[horizontal] +index-name:: (Required) A unique name that identifies the index. + +keyspace-ref:: (Required) See <>. + +NOTE: If there is a hyphen (-) inside the index name or any part of the keyspace reference, you must wrap the index name or that part of the keyspace reference in backticks ({backtick}{nbsp}{backtick}). +See the examples on this page. + +[[keyspace-ref]] +==== Keyspace Reference + [source,ebnf] ---- include::partial$grammar/dql.ebnf[tag=keyspace-ref] @@ -165,14 +203,10 @@ include::partial$grammar/dql.ebnf[tag=keyspace-ref] image::n1ql-language-reference/keyspace-ref.png["Syntax diagram: refer to source code listing", align=left] -You can use the index name with the `ON` keyword and a keyspace reference to specify the keyspace on which the index is built. The keyspace reference may be a <> or a <>. -NOTE: If there is a hyphen (-) inside the index name or any part of the keyspace reference, you must wrap the index name or that part of the keyspace reference in backticks ({backtick}{nbsp}{backtick}). -Refer to the examples below. - [[keyspace-path,keyspace path]] -==== Keyspace Reference: Keyspace Path +===== Keyspace Reference: Keyspace Path [source,ebnf] ---- @@ -208,7 +242,7 @@ Similarly, `{backtick}idx-name{backtick} ON default:{backtick}travel-sample{back ==== [[keyspace-partial,keyspace partial]] -==== Keyspace Reference: Keyspace Partial +===== Keyspace Reference: Keyspace Partial [source,ebnf] ---- @@ -227,7 +261,21 @@ collection:: ==== For example, `{backtick}idx-name{backtick} ON airline` indicates the `idx-name` index on the `airline` collection, assuming the query context is set. ==== +//end::index-path-and-name[] +//tag::if-exists[] +[[if-exists]] +=== IF EXISTS Clause + +The optional `IF EXISTS` clause enables the statement to complete successfully when the specified index doesn't exist. +If the index does not exist within the specified keyspace, then: + +* If this clause is not present, an error is generated. + +* If this clause is present, the statement does nothing and completes without error. +//end::if-exists[] + +//tag::index-using[] [[index-using]] === USING Clause @@ -240,10 +288,12 @@ image::n1ql-language-reference/index-using.png["Syntax diagram: refer to source The index type for a secondary index must be Global Secondary Index (GSI). The `USING GSI` keywords are optional and may be omitted. +//end::index-using[] +//tag::usage[] == Usage -When using memory-optimized indexes, DROP INDEX is an expensive operation and may take a few minutes to complete. +When using memory-optimized indexes, {doctitle} is an expensive operation and may take a few minutes to complete. If you drop an index with replicas while one of the index nodes is failed over, then only the replicas in the active index nodes are dropped. If the failed-over index node is recovered, then the orphan replica will be dropped when this failed-over indexer is added back to cluster. @@ -252,8 +302,12 @@ If you drop an index with replicas when one of the index nodes is unavailable bu If you drop an index which is scheduled for background creation, a warning message is generated, but the drop index operation succeeds. -[caption=Attention] -IMPORTANT: We recommend that you do not drop (or create) secondary indexes when any node with a secondary index role is down as this may result in duplicate index names. +[IMPORTANT] +.Attention +==== +Do not drop (or create) secondary indexes, composite vector indexes, or hyperscale vector indexes when any Index service node is down, as this may result in duplicate index names. +==== +//end::usage[] == Examples @@ -335,3 +389,17 @@ The following command would drop the index in exactly the same way, but uses alt include::example$n1ql-language-reference/drop-idx-alt.n1ql[] ---- ==== + +== Related Links + +* xref:indexes:indexing-overview.adoc[] +* xref:vector-index:composite-vector-index.adoc[] +* xref:vector-index:hyperscale-vector-index.adoc[] +* xref:n1ql:n1ql-language-reference/createprimaryindex.adoc[CREATE PRIMARY INDEX] +| xref:n1ql:n1ql-language-reference/createindex.adoc[CREATE INDEX] +| xref:n1ql:n1ql-language-reference/createvectorindex.adoc[CREATE VECTOR INDEX] +* xref:n1ql:n1ql-language-reference/build-index.adoc[BUILD INDEX] +* xref:n1ql:n1ql-language-reference/alterindex.adoc[ALTER INDEX] +| xref:n1ql:n1ql-language-reference/altervectorindex.adoc[ALTER VECTOR INDEX] +* xref:n1ql:n1ql-language-reference/dropprimaryindex.adoc[DROP PRIMARY INDEX] +| xref:n1ql:n1ql-language-reference/dropvectorindex.adoc[DROP VECTOR INDEX] diff --git a/modules/n1ql/pages/n1ql-language-reference/dropprimaryindex.adoc b/modules/n1ql/pages/n1ql-language-reference/dropprimaryindex.adoc index ea8becba6..c4f759c3b 100644 --- a/modules/n1ql/pages/n1ql-language-reference/dropprimaryindex.adoc +++ b/modules/n1ql/pages/n1ql-language-reference/dropprimaryindex.adoc @@ -1,6 +1,7 @@ = DROP PRIMARY INDEX -:description: The DROP PRIMARY INDEX statement allows you to drop an unnamed primary index. +:description: The DROP PRIMARY INDEX statement allows you to drop a primary index. :page-topic-type: reference +:page-toclevels: 2 :imagesdir: ../../assets/images :roles: xref:learn:security/roles.adoc @@ -13,14 +14,14 @@ include::partial$n1ql-language-reference/horizontal-style.adoc[] {description} -IMPORTANT: Named primary indexes that are created using CREATE PRIMARY INDEX can only be dropped using the DROP INDEX command. +NOTE: For compatibility with legacy versions of Couchbase Server, you can also use the DROP INDEX or DROP VECTOR INDEX statement to drop a named primary index. == Prerequisites [discrete] ===== RBAC Privileges -User executing the DROP PRIMARY INDEX statement must have the _Query Manage Index_ privilege granted on the keyspace. +To execute the DROP PRIMARY INDEX statement, you must have the _Query Manage Index_ privilege granted on the keyspace. For more details about user roles, see {roles}[Roles]. == Syntax @@ -35,11 +36,14 @@ image::n1ql-language-reference/drop-primary-index.png["Syntax diagram: refer to // TODO: Automatic links in EBNF. [horizontal] +index-name:: (Optional) A unique name that identifies the index. +If you don't specify a name, the index with the default name of `#primary` is deleted. + keyspace-ref:: (Required) Specifies the keyspace where the index is located. -Refer to <> below. +See <>. index-using:: (Optional) Specifies the index type. -Refer to <> below. +See <>. [[if-exists]] === IF EXISTS Clause @@ -65,7 +69,7 @@ Specifies the keyspace for the primary index to drop. The keyspace reference may be a <> or a <>. NOTE: If there is a hyphen (-) inside any part of the keyspace reference, you must wrap that part of the keyspace reference in backticks ({backtick}{nbsp}{backtick}). -Refer to the examples below. +See the examples on this page. [[keyspace-path,keyspace path]] ==== Keyspace Path @@ -157,4 +161,18 @@ Subsequently, drop the unnamed primary index with the following statement so tha ---- include::example$n1ql-language-reference/drop-pri-nameless.n1ql[tags=**] ---- -==== \ No newline at end of file +==== + +== Related Links + +* xref:indexes:indexing-overview.adoc[] +* xref:vector-index:composite-vector-index.adoc[] +* xref:vector-index:hyperscale-vector-index.adoc[] +* xref:n1ql:n1ql-language-reference/createprimaryindex.adoc[CREATE PRIMARY INDEX] +| xref:n1ql:n1ql-language-reference/createindex.adoc[CREATE INDEX] +| xref:n1ql:n1ql-language-reference/createvectorindex.adoc[CREATE VECTOR INDEX] +* xref:n1ql:n1ql-language-reference/build-index.adoc[BUILD INDEX] +* xref:n1ql:n1ql-language-reference/alterindex.adoc[ALTER INDEX] +| xref:n1ql:n1ql-language-reference/altervectorindex.adoc[ALTER VECTOR INDEX] +* xref:n1ql:n1ql-language-reference/dropindex.adoc[DROP INDEX] +| xref:n1ql:n1ql-language-reference/dropvectorindex.adoc[DROP VECTOR INDEX] diff --git a/modules/n1ql/pages/n1ql-language-reference/dropvectorindex.adoc b/modules/n1ql/pages/n1ql-language-reference/dropvectorindex.adoc new file mode 100644 index 000000000..ba3442aca --- /dev/null +++ b/modules/n1ql/pages/n1ql-language-reference/dropvectorindex.adoc @@ -0,0 +1,89 @@ += DROP VECTOR INDEX +:description: The DROP VECTOR INDEX statement allows you to drop a hyperscale vector index, a composite vector index, or a secondary index. +:page-topic-type: reference +:page-status: Couchbase Server 8.0 +:page-toclevels: 2 +:imagesdir: ../../assets/images + +:authorization-overview: xref:learn:security/authorization-overview.adoc +:query-context: xref:n1ql:n1ql-intro/queriesandresults.adoc#query-context +:logical-hierarchy: xref:n1ql-intro/queriesandresults.adoc#logical-hierarchy +:identifiers: xref:n1ql-language-reference/identifiers.adoc +:drop-primary-index: xref:n1ql-language-reference/dropprimaryindex.adoc +:drop-index: xref:n1ql-language-reference/dropindex.adoc +:prerequisites: xref:vector-index:hyperscale-vector-index.adoc#prerequisites + +// TEMP +include::partial$n1ql-language-reference/horizontal-style.adoc[] + +{description} +Dropping an index that has replicas will drop all of the replica indexes too. + +The {drop-index}[DROP INDEX] statement is a synonym for the DROP VECTOR INDEX statement. +The two statements have the same functionality. + +[NOTE] +To drop a primary index, use the {drop-primary-index}[DROP PRIMARY INDEX] statement. +For compatibility with legacy versions of Couchbase Server, you can also use DROP INDEX or DROP VECTOR INDEX to drop a named primary index. + +include::dropindex.adoc[tags=prerequisites] + +== Syntax + +[source,ebnf] +---- +include::partial$grammar/ddl.ebnf[tag=drop-vector-index] +---- + +image::n1ql-language-reference/drop-vector-index.png["Syntax diagram: refer to source code listing", align=left] + +include::dropindex.adoc[tags=syntax-arguments] + +include::dropindex.adoc[tags=index-path-and-name] + +include::dropindex.adoc[tags=if-exists] + +include::dropindex.adoc[tags=index-using] + +include::dropindex.adoc[tags=usage] + +== Examples + +To try the examples in this section, you must install the vector sample data as described in {prerequisites}[Prerequisites]. + +[[ex-1]] +.Drop a hyperscale vector index +==== +For this example, the path to the required keyspace is specified by the query, so you do not need to set the query context. + +Drop the hyperscale vector index called `color_desc_hyperscale`, if it exists. + +[source,sqlpp] +---- +DROP VECTOR INDEX `color_desc_hyperscale` + IF EXISTS + ON `vector-sample`.color.rgb +---- + +The following command would drop the index in exactly the same way, but uses alternative syntax. + +[source,sqlpp] +---- +DROP VECTOR INDEX IF EXISTS + default:`vector-sample`.color.rgb.`color_desc_hyperscale` +---- +==== + +== Related Links + +* xref:indexes:indexing-overview.adoc[] +* xref:vector-index:composite-vector-index.adoc[] +* xref:vector-index:hyperscale-vector-index.adoc[] +* xref:n1ql:n1ql-language-reference/createprimaryindex.adoc[CREATE PRIMARY INDEX] +| xref:n1ql:n1ql-language-reference/createindex.adoc[CREATE INDEX] +| xref:n1ql:n1ql-language-reference/createvectorindex.adoc[CREATE VECTOR INDEX] +* xref:n1ql:n1ql-language-reference/build-index.adoc[BUILD INDEX] +* xref:n1ql:n1ql-language-reference/alterindex.adoc[ALTER INDEX] +| xref:n1ql:n1ql-language-reference/altervectorindex.adoc[ALTER VECTOR INDEX] +* xref:n1ql:n1ql-language-reference/dropprimaryindex.adoc[DROP PRIMARY INDEX] +| xref:n1ql:n1ql-language-reference/dropindex.adoc[DROP INDEX] diff --git a/modules/n1ql/partials/grammar/ddl.ebnf b/modules/n1ql/partials/grammar/ddl.ebnf index 827efc4b9..ef9d7569c 100644 --- a/modules/n1ql/partials/grammar/ddl.ebnf +++ b/modules/n1ql/partials/grammar/ddl.ebnf @@ -8,6 +8,7 @@ create-statement ::= create-scope | create-collection | create-primary-index | create-index + | create-vector-index | create-function | create-sequence @@ -15,10 +16,12 @@ drop-statement ::= drop-scope | drop-collection | drop-primary-index | drop-index + | drop-vector-index | drop-function | drop-sequence other-statement ::= alter-index + | alter-vector-index | alter-sequence | build-index | execute-function @@ -52,7 +55,8 @@ drop-collection ::= 'DROP' 'COLLECTION' ( ( namespace ':' )? bucket '.' scope '. ***********/ /* tag::create-primary-index[] */ -create-primary-index ::= 'CREATE' 'PRIMARY' 'INDEX' index-name? ( 'IF' 'NOT' 'EXISTS' )? +create-primary-index ::= 'CREATE' 'PRIMARY' 'INDEX' ( index-name? ( 'IF' 'NOT' 'EXISTS' )? | + 'IF' 'NOT' 'EXISTS' index-name ) 'ON' keyspace-ref index-using? index-with? /* end::create-primary-index[] */ @@ -65,21 +69,39 @@ index-with ::= 'WITH' expr /* end::index-with[] */ /* tag::create-index[] */ -create-index ::= 'CREATE' 'INDEX' index-name ( 'IF' 'NOT' 'EXISTS' )? 'ON' keyspace-ref - '(' index-key lead-key-attribs? ( ( ',' index-key key-attribs? )+ )? ')' +create-index ::= 'CREATE' 'INDEX' ( index-name ( 'IF' 'NOT' 'EXISTS' )? | + 'IF' 'NOT' 'EXISTS' index-name ) 'ON' keyspace-ref + '(' index-keys-and-attribs ')' index-partition? where-clause? index-using? index-with? /* end::create-index[] */ +/* tag::create-vector-index[] */ +create-vector-index ::= 'CREATE' 'VECTOR' 'INDEX' ( index-name ( 'IF' 'NOT' 'EXISTS' )? | + 'IF' 'NOT' 'EXISTS' index-name ) 'ON' keyspace-ref + '(' index-key-and-attrib ')' + index-include? index-partition? where-clause? index-using? index-with? +/* end::create-vector-index[] */ + +/* tag::index-keys-and-attribs[] */ +index-keys-and-attribs ::= index-key lead-key-attribs? ( ( ',' index-key key-attribs? )+ )? +/* end::index-keys-and-attribs[] */ + +/* tag::index-key-and-attrib[] */ +index-key-and-attrib ::= index-key index-vector +/* end::index-key-and-attrib[] */ + /* tag::index-key[] */ index-key ::= expr | array-expr /* end::index-key[] */ /* tag::lead-key-attribs[] */ -lead-key-attribs ::= index-order include-missing? | include-missing index-order? +lead-key-attribs ::= index-order include-missing? | + include-missing index-order? | + index-vector /* end::lead-key-attribs[] */ /* tag::key-attribs[] */ -key-attribs ::= index-order +key-attribs ::= index-order | index-vector /* end::key-attribs[] */ /* tag::include-missing[] */ @@ -90,6 +112,10 @@ include-missing ::= 'INCLUDE' 'MISSING' index-order ::= 'ASC' | 'DESC' /* end::index-order[] */ +/* tag::index-vector[] */ +index-vector ::= 'VECTOR' +/* end::index-vector[] */ + /* tag::array-expr[] */ array-expr ::= full-array-expr | simple-array-expr /* end::array-expr[] */ @@ -112,6 +138,10 @@ pairs-function ::= 'PAIRS' '(' ( 'SELF' | index-key-object ) ')' index-key-object ::= expr /* end::index-key-object[] */ +/* tag::index-include[] */ +index-include ::= 'INCLUDE' '(' expr ( ',' expr )* ')' +/* end::index-include[] */ + /* tag::index-partition[] */ index-partition ::= 'PARTITION' 'BY' 'HASH' '(' partition-key-expr ( ',' partition-key-expr )* ')' @@ -122,10 +152,15 @@ partition-key-expr ::= expr /* end::partition-key-expr[] */ /* tag::alter-index[] */ -alter-index ::= 'ALTER' 'INDEX' ( index-path '.' index-name | index-name 'ON' keyspace-ref ) - index-using? index-with +alter-index ::= 'ALTER' 'INDEX' ( index-path '.' index-name | + index-name 'ON' keyspace-ref ) index-using? index-with /* end::alter-index[] */ +/* tag::alter-vector-index[] */ +alter-vector-index ::= 'ALTER' 'VECTOR' 'INDEX' ( index-path '.' index-name | + index-name 'ON' keyspace-ref ) index-using? index-with +/* end::alter-vector-index[] */ + /* tag::build-index[] */ build-index ::= 'BUILD' 'INDEX' 'ON' keyspace-ref '(' index-term (',' index-term)* ')' index-using? @@ -140,15 +175,30 @@ index-expr ::= string | array /* end::index-expr[] */ /* tag::drop-primary-index[] */ -drop-primary-index ::= 'DROP' 'PRIMARY' 'INDEX' ( 'IF' 'EXISTS' )? 'ON' keyspace-ref - index-using? +drop-primary-index ::= 'DROP' 'PRIMARY' 'INDEX' ( index-name? ( 'IF' 'EXISTS' )? | + 'IF' 'EXISTS' index-name ) 'ON' keyspace-ref index-using? /* end::drop-primary-index[] */ /* tag::drop-index[] */ -drop-index ::= 'DROP' 'INDEX' ( index-path '.' index-name ( 'IF' 'EXISTS' )? | - index-name ( 'IF' 'EXISTS' )? 'ON' keyspace-ref ) index-using? +drop-index ::= 'DROP' 'INDEX' ( index-path-and-name | index-name-on-keyspace ) + index-using? /* end::drop-index[] */ +/* tag::drop-vector-index[] */ +drop-vector-index ::= 'DROP' 'VECTOR' 'INDEX' ( index-path-and-name | index-name-on-keyspace ) + index-using? +/* end::drop-vector-index[] */ + +/* tag::index-path-and-name[] */ +index-path-and-name ::= index-path '.' index-name ( 'IF' 'EXISTS' )? | + 'IF' 'EXISTS' index-path '.' index-name +/* end::index-path-and-name[] */ + +/* tag::index-name-on-keyspace[] */ +index-name-on-keyspace ::= ( index-name ( 'IF' 'EXISTS' )? | 'IF' 'EXISTS' index-name ) + 'ON' keyspace-ref +/* end::index-name-on-keyspace[] */ + /* tag::index-path[] */ index-path ::= keyspace-full | keyspace-prefix | keyspace-partial /* end::index-path[] */ diff --git a/modules/n1ql/partials/n1ql-language-reference/vector-index-no-windows.adoc b/modules/n1ql/partials/n1ql-language-reference/vector-index-no-windows.adoc new file mode 100644 index 000000000..bf7ee376e --- /dev/null +++ b/modules/n1ql/partials/n1ql-language-reference/vector-index-no-windows.adoc @@ -0,0 +1,5 @@ +[IMPORTANT] +-- +You cannot use Hyperscale Vector Indexes or Composite Vector Indexes on Windows platforms. +You can use Hyperscale Vector Indexes and Composite Vector Indexes in Couchbase Server version 8.0 and later on Linux and macOS. +-- \ No newline at end of file diff --git a/modules/n1ql/partials/nav.adoc b/modules/n1ql/partials/nav.adoc index aa43f279e..56f7531c4 100644 --- a/modules/n1ql/partials/nav.adoc +++ b/modules/n1ql/partials/nav.adoc @@ -90,6 +90,7 @@ **** xref:n1ql:n1ql-language-reference/advise.adoc[] **** xref:n1ql:n1ql-language-reference/alterindex.adoc[] **** xref:n1ql:n1ql-language-reference/altersequence.adoc[] + **** xref:n1ql:n1ql-language-reference/altervectorindex.adoc[] **** xref:n1ql:n1ql-language-reference/begin-transaction.adoc[] **** xref:n1ql:n1ql-language-reference/build-index.adoc[] **** xref:n1ql:n1ql-language-reference/commit-transaction.adoc[] @@ -103,12 +104,14 @@ **** xref:n1ql:n1ql-language-reference/createprimaryindex.adoc[] **** xref:n1ql:n1ql-language-reference/createsequence.adoc[] **** xref:n1ql:n1ql-language-reference/createscope.adoc[] + **** xref:n1ql:n1ql-language-reference/createvectorindex.adoc[] **** xref:n1ql:n1ql-language-reference/delete.adoc[] **** xref:n1ql:n1ql-language-reference/dropcollection.adoc[] **** xref:n1ql:n1ql-language-reference/dropfunction.adoc[] **** xref:n1ql:n1ql-language-reference/dropindex.adoc[] **** xref:n1ql:n1ql-language-reference/dropprimaryindex.adoc[] **** xref:n1ql:n1ql-language-reference/dropsequence.adoc[] + **** xref:n1ql:n1ql-language-reference/dropvectorindex.adoc[] **** xref:n1ql:n1ql-language-reference/dropscope.adoc[] **** xref:n1ql:n1ql-language-reference/execute.adoc[] **** xref:n1ql:n1ql-language-reference/execfunction.adoc[]