Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

updating streams data type page with bike story for CLI commands and … #2516

Merged
merged 11 commits into from
Sep 18, 2023
Merged

updating streams data type page with bike story for CLI commands and … #2516

Changes from 1 commit
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
d72c985
updating streams data type page with bike story for CLI commands and …
Aug 14, 2023
2ed6566
updating wordlist for geo and streams
Aug 14, 2023
62dd113
wordlist update with commands
Aug 15, 2023
20fa652
Merge branch 'master' into master
sav-norem Aug 15, 2023
51d8fa5
streams_tutorial -> stream_tutorial
Aug 15, 2023
e627742
wordlist fix
Aug 15, 2023
9f1aeb3
Merge branch 'master' of https://github.com/sav-norem/redis-doc
Aug 15, 2023
96614a8
hopefully the last wordlist fix
sav-norem Aug 15, 2023
caa1987
update stream IDs to be more consistent, add caveat about various imp…
sav-norem Aug 21, 2023
4ee5a0c
address feedback from PR + spelling tutorial correctly
sav-norem Aug 22, 2023
cd2e477
Update docs/data-types/streams.md
sav-norem Aug 22, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
60 changes: 30 additions & 30 deletions docs/data-types/streams.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -118,7 +118,7 @@ The format of such IDs may look strange at first, and the gentle reader may wond

If for some reason the user needs incremental IDs that are not related to time but are actually associated to another external system ID, as previously mentioned, the `XADD` command can take an explicit ID instead of the `*` wildcard ID that triggers auto-generation, like in the following examples:

{{< clients-example stream_toturial xadd_id >}}
{{< clients-example stream_tutorial xadd_id >}}
> XADD race:usa 0-1 racer Castilla
0-1
> XADD race:usa 0-2 racer Norem
Expand All @@ -127,14 +127,14 @@ If for some reason the user needs incremental IDs that are not related to time b

Note that in this case, the minimum ID is 0-1 and that the command will not accept an ID equal or smaller than a previous one:

{{< clients-example stream_toturial xadd_bad_id >}}
{{< clients-example stream_tutorial xadd_bad_id >}}
> XADD race:usa 0-1 racer Prickett
(error) ERR The ID specified in XADD is equal or smaller than the target stream top item
{{< /clients-example >}}

If you're running Redis 7 or later, you can also provide an explicit ID consisting of the milliseconds part only. In this case, the sequence portion of the ID will be automatically generated. To do this, use the syntax below:

{{< clients-example stream_toturial xadd_7 >}}
{{< clients-example stream_tutorial xadd_7 >}}
> XADD race:usa 0-* racer Prickett
0-3
{{< /clients-example >}}
Expand All @@ -153,7 +153,7 @@ Redis Streams support all three of the query modes described above via different

To query the stream by range we are only required to specify two IDs, *start* and *end*. The range returned will include the elements having start or end as ID, so the range is inclusive. The two special IDs `-` and `+` respectively mean the smallest and the greatest ID possible.

{{< clients-example stream_toturial xrange_all >}}
{{< clients-example stream_tutorial xrange_all >}}
> XRANGE race:france - +
1) 1) "1692632086370-0"
2) 1) "rider"
Expand Down Expand Up @@ -195,7 +195,7 @@ To query the stream by range we are only required to specify two IDs, *start* an

Each entry returned is an array of two items: the ID and the list of field-value pairs. We already said that the entry IDs have a relation with the time, because the part at the left of the `-` character is the Unix time in milliseconds of the local node that created the stream entry, at the moment the entry was created (however note that streams are replicated with fully specified `XADD` commands, so the replicas will have identical IDs to the master). This means that I could query a range of time using `XRANGE`. In order to do so, however, I may want to omit the sequence part of the ID: if omitted, in the start of the range it will be assumed to be 0, while in the end part it will be assumed to be the maximum sequence number available. This way, querying using just two milliseconds Unix times, we get all the entries that were generated in that range of time, in an inclusive way. For instance, if I want to query a two milliseconds period I could use:

{{< clients-example stream_toturial xrange_time >}}
{{< clients-example stream_tutorial xrange_time >}}
> XRANGE race:france 1692632086369 1692632086371
1) 1) "1692632086370-0"
2) 1) "rider"
Expand All @@ -208,9 +208,9 @@ Each entry returned is an array of two items: the ID and the list of field-value
8) "1"
{{< /clients-example >}}

I have only a single entry in this range, however in real data sets, I could query for ranges of hours, or there could be many items in just two milliseconds, and the result returned could be huge. For this reason, `XRANGE` supports an optional **COUNT** option at the end. By specifying a count, I can just get the first *N* items. If I want more, I can get the last ID returned, increment the sequence part by one, and query again. Let's see this in the following example. Let's assume that the stream `race:france` was populated with 4 items. To start my iteration, getting 2 items per command, I start with the full range, but with a count of 2.
I have only a single entry in this range. However in real data sets, I could query for ranges of hours, or there could be many items in just two milliseconds, and the result returned could be huge. For this reason, `XRANGE` supports an optional **COUNT** option at the end. By specifying a count, I can just get the first *N* items. If I want more, I can get the last ID returned, increment the sequence part by one, and query again. Let's see this in the following example. Let's assume that the stream `race:france` was populated with 4 items. To start my iteration, getting 2 items per command, I start with the full range, but with a count of 2.

{{< clients-example stream_toturial xrange_step_1 >}}
{{< clients-example stream_tutorial xrange_step_1 >}}
> XRANGE race:france - + COUNT 2
1) 1) "1692632086370-0"
2) 1) "rider"
Expand All @@ -232,9 +232,9 @@ I have only a single entry in this range, however in real data sets, I could que
8) "1"
{{< /clients-example >}}

In order to continue the iteration with the next two items, I have to pick the last ID returned, that is `1692632094485-0` and add the prefix `(` to it. The resulting exclusive range interval, that is `(1692632094485-0` in this case, can now be used as the new *start* argument for the next `XRANGE` call:
To continue the iteration with the next two items, I have to pick the last ID returned, that is `1692632094485-0`, and add the prefix `(` to it. The resulting exclusive range interval, that is `(1692632094485-0` in this case, can now be used as the new *start* argument for the next `XRANGE` call:

{{< clients-example stream_toturial xrange_step_2 >}}
{{< clients-example stream_tutorial xrange_step_2 >}}
> XRANGE race:france (1692632094485-0 + COUNT 2
1) 1) "1692632102976-0"
2) 1) "rider"
Expand All @@ -256,9 +256,9 @@ In order to continue the iteration with the next two items, I have to pick the l
8) "2"
{{< /clients-example >}}

Now that we've gotten 4 items out of a stream that only had 4 things in it, if we try to get more items, we'll get an empty array:
Now that we've retrieved 4 items out of a stream that only had 4 entries in it, if we try to retrieve more items, we'll get an empty array:

{{< clients-example stream_toturial xrange_empty >}}
{{< clients-example stream_tutorial xrange_empty >}}
> XRANGE race:france (1692632147973-0 + COUNT 2
(empty array)
{{< /clients-example >}}
Expand All @@ -267,7 +267,7 @@ Since `XRANGE` complexity is *O(log(N))* to seek, and then *O(M)* to return M el

The command `XREVRANGE` is the equivalent of `XRANGE` but returning the elements in inverted order, so a practical use for `XREVRANGE` is to check what is the last item in a Stream:

{{< clients-example stream_toturial xrevrange >}}
{{< clients-example stream_tutorial xrevrange >}}
> XREVRANGE race:france + - COUNT 1
1) 1) "1692632147973-0"
2) 1) "rider"
Expand All @@ -292,7 +292,7 @@ When we do not want to access items by a range in a stream, usually what we want

The command that provides the ability to listen for new messages arriving into a stream is called `XREAD`. It's a bit more complex than `XRANGE`, so we'll start showing simple forms, and later the whole command layout will be provided.

{{< clients-example stream_toturial xread >}}
{{< clients-example stream_tutorial xread >}}
> XREAD COUNT 2 STREAMS race:france 0
1) 1) "race:france"
2) 1) 1) "1692632086370-0"
Expand Down Expand Up @@ -391,7 +391,7 @@ Now it's time to zoom in to see the fundamental consumer group commands. They ar

Assuming I have a key `race:france` of type stream already existing, in order to create a consumer group I just need to do the following:

{{< clients-example stream_toturial xgroup_create >}}
{{< clients-example stream_tutorial xgroup_create >}}
> XGROUP CREATE race:france france_riders $
OK
{{< /clients-example >}}
Expand All @@ -400,7 +400,7 @@ As you can see in the command above when creating the consumer group we have to

`XGROUP CREATE` also supports creating the stream automatically, if it doesn't exist, using the optional `MKSTREAM` subcommand as the last argument:

{{< clients-example stream_toturial xgroup_create_mkstream >}}
{{< clients-example stream_tutorial xgroup_create_mkstream >}}
> XGROUP CREATE race:italy italy_riders $ MKSTREAM
OK
{{< /clients-example >}}
Expand All @@ -410,9 +410,9 @@ Now that the consumer group is created we can immediately try to read messages v
`XREADGROUP` is very similar to `XREAD` and provides the same **BLOCK** option, otherwise it is a synchronous command. However there is a *mandatory* option that must be always specified, which is **GROUP** and has two arguments: the name of the consumer group, and the name of the consumer that is attempting to read. The option **COUNT** is also supported and is identical to the one in `XREAD`.

We'll add riders to the race:italy stream and try reading something using the consumer group:
Note: *here rider is the field name, and the name is the associated value, remember that stream items are small dictionaries.*
Note: *here rider is the field name, and the name is the associated value. Remember that stream items are small dictionaries.*

{{< clients-example stream_toturial xgroup_read >}}
{{< clients-example stream_tutorial xgroup_read >}}
> XADD race:italy * rider Castilla
"1692632639151-0"
> XADD race:italy * rider Royce
Expand Down Expand Up @@ -441,7 +441,7 @@ This is almost always what you want, however it is also possible to specify a re

We can test this behavior immediately specifying an ID of 0, without any **COUNT** option: we'll just see the only pending message, that is, the one about apples:

{{< clients-example stream_toturial xgroup_read_id >}}
{{< clients-example stream_tutorial xgroup_read_id >}}
> XREADGROUP GROUP italy_riders Alice STREAMS race:italy 0
1) 1) "race:italy"
2) 1) 1) "1692632639151-0"
Expand All @@ -451,7 +451,7 @@ We can test this behavior immediately specifying an ID of 0, without any **COUNT

However, if we acknowledge the message as processed, it will no longer be part of the pending messages history, so the system will no longer report anything:

{{< clients-example stream_toturial xack >}}
{{< clients-example stream_tutorial xack >}}
> XACK race:italy italy_riders 1692632639151-0
(integer) 1
> XREADGROUP GROUP italy_riders Alice STREAMS race:italy 0
Expand All @@ -463,7 +463,7 @@ Don't worry if you yet don't know how `XACK` works, the idea is just that proces

Now it's Bob's turn to read something:

{{< clients-example stream_toturial xgroup_read_bob >}}
{{< clients-example stream_tutorial xgroup_read_bob >}}
> XREADGROUP GROUP italy_riders Bob COUNT 2 STREAMS race:italy >
1) 1) "race:italy"
2) 1) 1) "1692632647899-0"
Expand Down Expand Up @@ -555,7 +555,7 @@ The first step of this process is just a command that provides observability of
This is a read-only command which is always safe to call and will not change ownership of any message.
In its simplest form, the command is called with two arguments, which are the name of the stream and the name of the consumer group.

{{< clients-example stream_toturial xpending >}}
{{< clients-example stream_tutorial xpending >}}
> XPENDING race:italy italy_riders
1) (integer) 2
2) "1692632647899-0"
Expand All @@ -575,7 +575,7 @@ XPENDING <key> <groupname> [[IDLE <min-idle-time>] <start-id> <end-id> <count> [

By providing a start and end ID (that can be just `-` and `+` as in `XRANGE`) and a count to control the amount of information returned by the command, we are able to know more about the pending messages. The optional final argument, the consumer name, is used if we want to limit the output to just messages pending for a given consumer, but won't use this feature in the following example.

{{< clients-example stream_toturial xpending_plus_minus >}}
{{< clients-example stream_tutorial xpending_plus_minus >}}
> XPENDING race:italy italy_riders - + 10
1) 1) "1692632647899-0"
2) "Bob"
Expand All @@ -592,7 +592,7 @@ We have two messages from Bob, and they are idle for 60000+ milliseconds, about

Note that nobody prevents us from checking what the first message content was by just using `XRANGE`.

{{< clients-example stream_toturial xrange_pending >}}
{{< clients-example stream_tutorial xrange_pending >}}
> XRANGE race:italy 1692632647899-0 1692632647899-0
1) 1) "1692632647899-0"
2) 1) "rider"
Expand All @@ -618,7 +618,7 @@ However, as a side effect, claiming a message will reset its idle time and will

This is the result of the command execution:

{{< clients-example stream_toturial xclaim >}}
{{< clients-example stream_tutorial xclaim >}}
> XCLAIM race:italy italy_riders Alice 60000 1692632647899-0
1) 1) "1692632647899-0"
2) 1) "rider"
Expand Down Expand Up @@ -646,7 +646,7 @@ XAUTOCLAIM <key> <group> <consumer> <min-idle-time> <start> [COUNT count] [JUSTI

So, in the example above, I could have used automatic claiming to claim a single message like this:

{{< clients-example stream_toturial xautoclaim >}}
{{< clients-example stream_tutorial xautoclaim >}}
> XAUTOCLAIM race:italy italy_riders Alice 60000 0-0 COUNT 1
1) "0-0"
2) 1) 1) "1692632662819-0"
Expand All @@ -657,7 +657,7 @@ So, in the example above, I could have used automatic claiming to claim a single
Like `XCLAIM`, the command replies with an array of the claimed messages, but it also returns a stream ID that allows iterating the pending entries.
The stream ID is a cursor, and I can use it in my next call to continue in claiming idle pending messages:

{{< clients-example stream_toturial xautoclaim_cursor >}}
{{< clients-example stream_tutorial xautoclaim_cursor >}}
> XAUTOCLAIM race:italy italy_riders Lora 60000 (1692632662819-0 COUNT 1
1) "1692632662819-0"
2) 1) 1) "1692632647899-0"
Expand All @@ -682,7 +682,7 @@ However we may want to do more than that, and the `XINFO` command is an observab

This command uses subcommands in order to show different information about the status of the stream and its consumer groups. For instance **XINFO STREAM <key>** reports information about the stream itself.

{{< clients-example stream_toturial xinfo >}}
{{< clients-example stream_tutorial xinfo >}}
> XINFO STREAM race:italy
1) "length"
2) (integer) 5
Expand All @@ -706,7 +706,7 @@ This command uses subcommands in order to show different information about the s

The output shows information about how the stream is encoded internally, and also shows the first and last message in the stream. Another piece of information available is the number of consumer groups associated with this stream. We can dig further asking for more information about the consumer groups.

{{< clients-example stream_toturial xinfo_groups >}}
{{< clients-example stream_tutorial xinfo_groups >}}
> XINFO GROUPS race:italy
1) 1) "name"
2) "italy_riders"
Expand All @@ -722,7 +722,7 @@ As you can see in this and in the previous output, the `XINFO` command outputs a

The output of the example above, where the **GROUPS** subcommand is used, should be clear observing the field names. We can check in more detail the state of a specific consumer group by checking the consumers that are registered in the group.

{{< clients-example stream_toturial xinfo_consumers >}}
{{< clients-example stream_tutorial xinfo_consumers >}}
> XINFO CONSUMERS race:italy italy_riders
1) 1) "name"
2) "Alice"
Expand Down Expand Up @@ -805,7 +805,7 @@ However trimming with **MAXLEN** can be expensive: streams are represented by ma
XADD race:italy MAXLEN ~ 1000 * ... entry fields here ...
```

The `~` argument between the **MAXLEN** option and the actual count means, I don't really need this to be exactly 1000 items. It can be 1000 or 1010 or 1030, just make sure to save at least 1000 items. With this argument, the trimming is performed only when we can remove a whole node. This makes it much more efficient, and it is usually what you want. You'll note here that the client libraries have various implementations of this; for example the Python client defaults to approximate and has to explicitly be set to a true length.
The `~` argument between the **MAXLEN** option and the actual count means, I don't really need this to be exactly 1000 items. It can be 1000 or 1010 or 1030, just make sure to save at least 1000 items. With this argument, the trimming is performed only when we can remove a whole node. This makes it much more efficient, and it is usually what you want. You'll note here that the client libraries have various implementations of this, For example, the Python client defaults to approximate and has to be explicitly set to a true length.

There is also the `XTRIM` command, which performs something very similar to what the **MAXLEN** option does above, except that it can be run by itself:

Expand Down