DOC-12434: DDL for GSI Indexes with a Vector field (#275)

* Add vector index syntax to EBNF
* Update script to use Railroad Diagram Generator 2.1
* Update syntax diagrams
* Update primary index functionality
* Update nav
* Update indexing overview
* Update index types
* Add ALTER VECTOR INDEX
* Add CREATE VECTOR INDEX
* Add DROP VECTOR INDEX
* Update ALTER INDEX
* Update BUILD INDEX
* Update CREATE INDEX
* Update CREATE PRIMARY INDEX
* Update DROP INDEX
* UPDATE DROP PRIMARY INDEX
* Update toc depth
* Adjusting related links
* Add composite vector index example
* Add examples to ALTER VECTOR INDEX
* Add clarification about base64 decoding
* Update CREATE INDEX examples

* DOC-12940: GSI Vector Index not supported on Windows (#406)

Author: simon-dew

diagrams.sh

@@ -3,10 +3,10 @@
 [[ -d ./build ]] || mkdir ./build
 
 # install-rr
-[[ -f ./build/rr-2.0-java11.zip ]] || curl -L -o ./build/rr-2.0-java11.zip https://github.com/GuntherRademacher/rr/releases/download/v2.0/rr-2.0-java11.zip
+[[ -f ./build/rr-2.1-java11.zip ]] || curl -L -o ./build/rr-2.1-java11.zip https://github.com/GuntherRademacher/rr/releases/download/v2.1/rr-2.1-java11.zip
 
 # extract-rr
-[[ -f ./build/rr/rr.war ]] || unzip ./build/rr-2.0-java11.zip -d ./build/rr
+[[ -f ./build/rr/rr.war ]] || unzip ./build/rr-2.1-java11.zip -d ./build/rr
 
 for file in n1ql dcl ddl dml dql tcl hints utility
 do

modules/indexes/pages/index-replication.adoc

@@ -40,7 +40,7 @@ This ensures:
 * _Availability_: If one Index-Service node is lost, the other continues to provide access to replicated indexes.
 * _High Performance_: If original and replica copies are available, incoming queries are load-balanced across them.
 
-Index-replicas can be created with the {sqlpp} `CREATE INDEX` statement.
+Index-replicas can be created with the {sqlpp} `CREATE INDEX` or `CREATE VECTOR INDEX` statement.
 Note that whenever a given number of index-replicas is specified for creation, the number must be less than the number of cluster-nodes currently running the {index-service}[Index Service].
 If it is not, the index creation fails.
 Note also that if, following creation of the maximum number of copies, the number of nodes running the Index Service decreases, Couchbase Server progressively assigns replacement index-replicas to any and all Index-Service nodes subsequently be added to the cluster, until the required number of index-replicas again exists for each replicated index.
@@ -69,10 +69,10 @@ include::example$replication-num.n1ql[]
 +
 Note that if `nodes` and `num_replica` are both specified in the `WITH` clause, the specified number of nodes must be _one greater_ than `num_replica`.
 
-* Specifying a number of index-replicas to be created by the Index Service whenever `CREATE INDEX` is invoked.
+* Specifying a number of index-replicas to be created by the Index Service whenever `CREATE INDEX` or `CREATE VECTOR INDEX` is invoked.
 The default is `0`.
 If the default is changed to, say, `2`, creation of a single index is henceforth accompanied by the creation of two replicas, which are automatically distributed across the nodes of the cluster running the Index Service.
-No explicit specification within the `CREATE INDEX` statement is required.
+No explicit specification within the `CREATE INDEX` or `CREATE VECTOR INDEX` statement is required.
 +
 With credentials that provide appropriate authorization, this default can be changed by means of the `curl` command, as follows:
 +
@@ -83,10 +83,10 @@ curl -X POST -u 'Administrator:password' \
 -d 'numReplica=2'
 ----
 +
-Here, `numReplica` is an integer that establishes the default number of replicas that must be created whenever `CREATE INDEX` is invoked.
+Here, `numReplica` is an integer that establishes the default number of replicas that must be created whenever `CREATE INDEX` or `CREATE VECTOR INDEX` is invoked.
 Note that this call only succeeds if the cluster contains enough Index Service nodes to host each new index and its replicas: for example, if `2` is specified as the default number of replicas, the Index Service must have been established on at least 3 nodes.
 +
-Note also that whenever explicit specification of replica-numbers is made within the `CREATE INDEX` statement, this explicit specification takes precedence over any established default.
+Note also that whenever explicit specification of replica-numbers is made within the `CREATE INDEX` or `CREATE VECTOR INDEX` statement, this explicit specification takes precedence over any established default.
 
 You can change index replication settings via the {index-storage-mode}[UI] or the {index-settings-via-rest}[REST API].
 For further information on using {sqlpp}, refer to {query}[Query Fundamentals].

modules/indexes/pages/indexing-and-query-perf.adoc

@@ -12,6 +12,8 @@
 :bucket-analyzer: xref:tools:query-workbench.adoc#bucket-analyzer
 :storage-modes: xref:indexes:storage-modes.adoc
 :additional-storage-use: xref:learn:data/transactions.adoc#additional-storage-use
+:hyperscale-vector-index: xref:vector-index:hyperscale-vector-index.adoc
+:composite-vector-index: xref:vector-index:composite-vector-index.adoc
 
 Creating the right index -- with the right keys, in the right order, and using the right expressions -- is critical to query performance in any database system.
 This is true for Couchbase as well.
@@ -20,7 +22,7 @@ This is true for Couchbase as well.
 .Examples on this Page
 ****
 The examples in this topic use the travel-sample dataset which is shipped with Couchbase Server.
-For instructions on how to install the sample bucket, see {install-sample-buckets}[Sample Buckets].
+For instructions on how to install the sample bucket, see {install-sample-buckets}[].
 
 include::ROOT:partial$query-context.adoc[tag=statement]
 ****
@@ -79,7 +81,7 @@ Use the primary index for full keyspace scans (primary scans) when the query doe
 
 NOTE: A primary index does not index any {additional-storage-use}[transaction records] that may be stored in a keyspace.
 This means that if you are counting the number of documents in a keyspace, you may see slightly different results, depending on whether you are using a primary index or not.
-Refer to {aggregatefun}[Aggregate Functions] and {bucket-analyzer}[Viewing the Data Insights].
+See {aggregatefun}[] and {bucket-analyzer}[Viewing the Data Insights].
 
 .Metadata for Primary Index
 ====
@@ -479,7 +481,7 @@ CREATE INDEX travel_sched ON route
 ----
 ====
 
-For further details and examples, refer to {indexing-arrays}[Array Indexing].
+For further details and examples, see {indexing-arrays}[].
 
 [#partial-index]
 == Partial Index
@@ -581,4 +583,21 @@ After the index selection is made, the query engine analyzes the query to see if
 If it does, the query engine skips retrieving the whole document from the data nodes.
 This is a performance optimization to keep in mind when designing your indexes.
 
-For further details and examples, refer to {covering-indexes}[Covering Indexes].
+For further details and examples, see {covering-indexes}[].
+
+[#hyperscale-vector-index]
+== Hyperscale Vector Index
+
+Hyperscale vector indexes are a type of <<secondary-index>> which contain a single vector field.
+They excel at indexing huge datasets that can scale into the billions of documents.
+They're optimized for pure vector searches, offering the highest performance of any index for your AI applications.
+
+For further details and examples, see {hyperscale-vector-index}[].
+
+[#composite-vector-index]
+== Composite Vector Index
+
+Composite vector indexes are a type of <<composite-secondary-index>> which contain a single vector field and one or more scalar fields.
+Your AI applications can use the index's scalar fields to filter the dataset before performing a vector similarity search.
+
+For further details and examples, see {composite-vector-index}[].

modules/indexes/pages/indexing-overview.adoc

@@ -35,10 +35,13 @@ endif::[]
 * xref:n1ql:n1ql-language-reference/advise.adoc[ADVISE]
 * xref:n1ql:n1ql-language-reference/createindex.adoc[CREATE INDEX]
 * xref:n1ql:n1ql-language-reference/createprimaryindex.adoc[CREATE PRIMARY INDEX]
+* xref:n1ql:n1ql-language-reference/createvectorindex.adoc[CREATE VECTOR INDEX]
 * xref:n1ql:n1ql-language-reference/build-index.adoc[BUILD INDEX]
 * xref:n1ql:n1ql-language-reference/alterindex.adoc[ALTER INDEX]
+* xref:n1ql:n1ql-language-reference/altervectorindex.adoc[ALTER VECTOR INDEX]
 * xref:n1ql:n1ql-language-reference/dropindex.adoc[DROP INDEX]
 * xref:n1ql:n1ql-language-reference/dropprimaryindex.adoc[DROP PRIMARY INDEX]
+* xref:n1ql:n1ql-language-reference/dropvectorindex.adoc[DROP VECTOR INDEX]
 
 == Storage and Availability
 
@@ -52,3 +55,4 @@ endif::[]
 * xref:learn:services-and-indexes/services/index-service.adoc[Index Service Architecture]
 * xref:manage:manage-indexes/manage-indexes.adoc[Manage Indexes]
 * xref:manage:monitor/monitoring-indexes.adoc[Monitor Indexes]
+* xref:vector-index:vectors-and-indexes-overview.adoc[]