Skip to content

Commit 22beaaa

Browse files
dwdoughertymich-elle-luna
dwdougherty
and
mich-elle-luna
authored
DEV: add new vector set data type and command pages (#1334)
* Initial commit of vector set docs * Add'l commit of vector set docs * Add'l commit of vector set docs * Adding data type and command pages * Added RESP2 and RESP3 return information * Fix some atrocious grammar on the vs index page. * A few more quick fixes * Apply suggestions from code review Co-authored-by: mich-elle-luna <153109578+mich-elle-luna@users.noreply.github.com> * Apply review comments Co-authored-by: mich-elle-luna <153109578+mich-elle-luna@users.noreply.github.com> * Apply preview banner + misc. fixes --------- Co-authored-by: mich-elle-luna <153109578+mich-elle-luna@users.noreply.github.com>
1 parent 64525dd commit 22beaaa

File tree

20 files changed

+1415
-0
lines changed

20 files changed

+1415
-0
lines changed

‎content/commands/vadd/index.md

Lines changed: 119 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,119 @@
1+
---
2+
categories:
3+
- docs
4+
- develop
5+
- stack
6+
- oss
7+
- rs
8+
- rc
9+
- oss
10+
- kubernetes
11+
- clients
12+
complexity: O(log(N)) for each element added, where N is the number of elements in the vector set.
13+
description: Add a new element to a vector set, or update its vector if it already exists.
14+
group: vector_set
15+
hidden: false
16+
linkTitle: VADD
17+
since: 8.0.0
18+
summary: Add a new element to a vector set, or update its vector if it already exists.
19+
syntax_fmt: "VADD key [REDUCE dim] (FP32 | VALUES num) vector element [CAS] [NOQUANT | Q8 | BIN]\n [EF build-exploration-factor] [SETATTR attributes] [M numlinks]"
20+
title: VADD
21+
bannerText: Vector set is a new data type that is currently in preview and may be subject to change.
22+
---
23+
24+
Add a new element into the vector set specified by `key`. The vector can be provided as 32-bit floating point (`FP32`) blob of values, or as floating point numbers as strings, prefixed by the number of elements (3 in the example below):
25+
26+
```
27+
VADD mykey VALUES 3 0.1 1.2 0.5 my-element
28+
```
29+
30+
## Required arguments
31+
32+
<details open>
33+
<summary><code>key</code></summary>
34+
35+
is the name of the key that will hold the vector set data.
36+
</details>
37+
38+
<details open>
39+
<summary><code>FP32 vector or VALUES num vector</code></summary>
40+
41+
either a 32-bit floating point (FP32) blob of values or `num` floating point numbers as strings.
42+
</details>
43+
44+
<details open>
45+
<summary><code>element</code></summary>
46+
47+
is the name of the element that is being added to the vector set.
48+
</details>
49+
50+
## Optional arguments
51+
52+
<details open>
53+
<summary><code>REDUCE dim</code></summary>
54+
55+
implements random projection to reduce the dimensionality of the vector. The projection matrix is saved and reloaded along with the vector set. Please note that the REDUCE option must be passed immediately before the vector. For example,
56+
57+
```
58+
VADD mykey REDUCE 50 VALUES ...
59+
```
60+
</details>
61+
62+
<details open>
63+
<summary><code>CAS</code></summary>
64+
65+
performs the operation partially using threads, in a check-and-set style. The neighbor candidates collection, which is slow, is performed in the background, while the command is executed in the main thread.
66+
</details>
67+
68+
<details open>
69+
<summary><code>NOQUANT</code></summary>
70+
71+
in the first VADD call for a given key, NOQUANT forces the vector to be created without int8 quantization, which is otherwise the default.
72+
</details>
73+
74+
<details open>
75+
<summary><code>BIN</code></summary>
76+
77+
forces the vector to use binary quantization instead of int8. This is much faster and uses less memory, but impacts the recall quality.
78+
</details>
79+
80+
<details open>
81+
<summary><code>Q8</code></summary>
82+
83+
forces the vector to use signed 8-bit quantization. This is the default, and the option only exists to make sure to check at insertion time that the vector set is of the same format.
84+
</details>
85+
86+
{{< note >}}
87+
`NOQUANT`, `Q8`, and `BIN` are mutually exclusive.
88+
89+
{{< /note >}}
90+
91+
<details open>
92+
<summary><code>EF build-exploration-factor</code></summary>
93+
94+
plays a role in the effort made to find good candidates when connecting the new node to the existing Hierarchical Navigable Small World (HNSW) graph. The default is 200. Using a larger value may help in achieving a better recall. To improve the recall it is also possible to increase EF during VSIM searches.
95+
</details>
96+
97+
<details open>
98+
<summary><code>SETATTR attributes</code></summary>
99+
100+
associates attributes in the form of a JavaScript object to the newly created entry or updates the attributes (if they already exist).
101+
It is the same as calling the VSETATTR command separately.
102+
</details>
103+
104+
<details open>
105+
<summary><code>M numlinks</code></summary>
106+
107+
is the maximum number of connections that each node of the graph will have with other nodes. The default is 16. More connections means more memory, but provides for more efficient graph exploration. Nodes at layer zero (every node exists at least at layer zero) have `M * 2` connections, while the other layers only have `M` connections. For example, setting `M` to `64` will use at least 1024 bytes of memory for layer zero. That's `M * 2` connections times 8 bytes (pointers), or `128 * 8 = 1024`. For higher layers, consider the following:
108+
109+
- Each node appears in ~1.33 layers on average (empirical observation from HNSW papers), which works out to be 0.33 higher layers per node.
110+
- Each of those higher layers has `M = 64` connections.
111+
112+
So, the additional amount of memory is approximately `0.33 × 64 × 8 ≈ 169.6` bytes per node, bringing the total memory to ~1193 bytes.
113+
114+
If you don't have a recall quality problem, the default is acceptable, and uses a minimal amount of memory.
115+
</details>
116+
117+
## Related topics
118+
119+
- [Vector sets]({{< relref "/develop/data-types/vector-sets" >}})

‎content/commands/vcard/index.md

Lines changed: 41 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,41 @@
1+
---
2+
categories:
3+
- docs
4+
- develop
5+
- stack
6+
- oss
7+
- rs
8+
- rc
9+
- oss
10+
- kubernetes
11+
- clients
12+
complexity: O(1)
13+
description: Return the number of elements in a vector set.
14+
group: vector_set
15+
hidden: false
16+
linkTitle: VCARD
17+
since: 8.0.0
18+
summary: Return the number of elements in a vector set.
19+
syntax_fmt: "VCARD key"
20+
title: VCARD
21+
bannerText: Vector set is a new data type that is currently in preview and may be subject to change.
22+
---
23+
24+
Return the number of elements in the specified vector set.
25+
26+
```shell
27+
VCARD word_embeddings
28+
(integer) 3000000
29+
```
30+
31+
## Required arguments
32+
33+
<details open>
34+
<summary><code>key</code></summary>
35+
36+
is the name of the key that holds the vector set.
37+
</details>
38+
39+
## Related topics
40+
41+
- [Vector sets]({{< relref "/develop/data-types/vector-sets" >}})

‎content/commands/vdim/index.md

Lines changed: 43 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,43 @@
1+
---
2+
categories:
3+
- docs
4+
- develop
5+
- stack
6+
- oss
7+
- rs
8+
- rc
9+
- oss
10+
- kubernetes
11+
- clients
12+
complexity: O(1)
13+
description: Return the dimension of vectors in the vector set.
14+
group: vector_set
15+
hidden: false
16+
linkTitle: VDIM
17+
since: 8.0.0
18+
summary: Return the dimension of vectors in the vector set.
19+
syntax_fmt: "VDIM key"
20+
title: VDIM
21+
bannerText: Vector set is a new data type that is currently in preview and may be subject to change.
22+
---
23+
24+
Return the number of dimensions of the vectors in the specified vector set.
25+
26+
```shell
27+
VDIM word_embeddings
28+
(integer) 300
29+
```
30+
31+
If the vector set was created using the `REDUCE` option for dimensionality reduction, this command reports the reduced dimension. However, you must still use full-size vectors when performing queries with the `VSIM` command.
32+
33+
## Required arguments
34+
35+
<details open>
36+
<summary><code>key</code></summary>
37+
38+
is the name of the key that holds the vector set.
39+
</details>
40+
41+
## Related topics
42+
43+
- [Vector sets]({{< relref "/develop/data-types/vector-sets" >}})

‎content/commands/vemb/index.md

Lines changed: 72 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,72 @@
1+
---
2+
categories:
3+
- docs
4+
- develop
5+
- stack
6+
- oss
7+
- rs
8+
- rc
9+
- oss
10+
- kubernetes
11+
- clients
12+
complexity: O(1)
13+
description: Return the vector associated with an element.
14+
group: vector_set
15+
hidden: false
16+
linkTitle: VEMB
17+
since: 8.0.0
18+
summary: Return the vector associated with an element.
19+
syntax_fmt: "VEMB key element [RAW]"
20+
title: VEMB
21+
bannerText: Vector set is a new data type that is currently in preview and may be subject to change.
22+
---
23+
24+
Return the approximate vector associated with a given element in the vector set.
25+
26+
```shell
27+
VEMB word_embeddings SQL
28+
1) "0.18208661675453186"
29+
2) "0.08535309880971909"
30+
3) "0.1365649551153183"
31+
4) "-0.16501599550247192"
32+
5) "0.14225517213344574"
33+
... 295 more elements ...
34+
```
35+
36+
Vector sets normalize and may quantize vectors on insertion. `VEMB` reverses this process to approximate the original vector by de-normalizing and de-quantizing it.
37+
38+
To retrieve the raw internal representation, use the `RAW` option:
39+
40+
```shell
41+
VEMB word_embeddings apple RAW
42+
1) int8
43+
2) "\xf1\xdc\xfd\x1e\xcc%E...\xde\x1f\xfbN" # artificially shortened for this example
44+
3) "3.1426539421081543"
45+
4) "0.17898885905742645"
46+
```
47+
48+
## Required arguments
49+
50+
<details open>
51+
<summary><code>key</code></summary>
52+
53+
is the name of the key that holds the vector set.
54+
</details>
55+
56+
<details open>
57+
<summary><code>element</code></summary>
58+
59+
is the name of the element whose vector you want to retrieve.
60+
</details>
61+
62+
## Optional arguments
63+
64+
<details open>
65+
<summary><code>RAW</code></summary>
66+
67+
returns the raw vector data, its quantization type, and metadata such as norm and range.
68+
</details>
69+
70+
## Related topics
71+
72+
- [Vector sets]({{< relref "/develop/data-types/vector-sets" >}})

‎content/commands/vgetattr/index.md

Lines changed: 46 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,46 @@
1+
---
2+
categories:
3+
- docs
4+
- develop
5+
- stack
6+
- oss
7+
- rs
8+
- rc
9+
- oss
10+
- kubernetes
11+
- clients
12+
complexity: O(1)
13+
description: Retrieve the JSON attributes of elements.
14+
group: vector_set
15+
hidden: false
16+
linkTitle: VGETATTR
17+
since: 8.0.0
18+
summary: Retrieve the JSON attributes of elements.
19+
syntax_fmt: "VGETATTR key element"
20+
title: VGETATTR
21+
bannerText: Vector set is a new data type that is currently in preview and may be subject to change.
22+
---
23+
24+
Return the JSON attributes associated with an element in a vector set.
25+
26+
```shell
27+
VGETATTR key element
28+
```
29+
30+
## Required arguments
31+
32+
<details open>
33+
<summary><code>key</code></summary>
34+
35+
is the name of the key that holds the vector set.
36+
</details>
37+
38+
<details open>
39+
<summary><code>element</code></summary>
40+
41+
is the name of the element whose attributes you want to retrieve.
42+
</details>
43+
44+
## Related topics
45+
46+
- [Vector sets]({{< relref "/develop/data-types/vector-sets" >}})

‎content/commands/vinfo/index.md

Lines changed: 52 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,52 @@
1+
---
2+
categories:
3+
- docs
4+
- develop
5+
- stack
6+
- oss
7+
- rs
8+
- rc
9+
- oss
10+
- kubernetes
11+
- clients
12+
complexity: O(1)
13+
description: Return information about a vector set.
14+
group: vector_set
15+
hidden: false
16+
linkTitle: VINFO
17+
since: 8.0.0
18+
summary: Return information about a vector set.
19+
syntax_fmt: "VINFO key"
20+
title: VINFO
21+
bannerText: Vector set is a new data type that is currently in preview and may be subject to change.
22+
---
23+
24+
Return metadata and internal details about a vector set, including size, dimensions, quantization type, and graph structure.
25+
26+
```shell
27+
VINFO word_embeddings
28+
1) quant-type
29+
2) int8
30+
3) vector-dim
31+
4) (integer) 300
32+
5) size
33+
6) (integer) 3000000
34+
7) max-level
35+
8) (integer) 12
36+
9) vset-uid
37+
10) (integer) 1
38+
11) hnsw-max-node-uid
39+
12) (integer) 3000000
40+
```
41+
42+
## Required arguments
43+
44+
<details open>
45+
<summary><code>key</code></summary>
46+
47+
is the name of the key that holds the vector set.
48+
</details>
49+
50+
## Related topics
51+
52+
- [Vector sets]({{< relref "/develop/data-types/vector-sets" >}})

0 commit comments

Comments
 (0)