feat: add zset #98

SilverRainZ · 2021-10-12T08:06:38Z

Intro

Package zset provides a concurrent-safety sorted set, can be used as a local replacement of Redis' zset (https://redis.com/ebook/part-2-core-concepts/chapter-3-commands-in-redis/3-5-sorted-sets/).

The main different to other sets is, every value of set is associated with a score, that is used in order to take the sorted set ordered, from the smallest to the greatest score.

The sorted set has O(log(N)) time complexity when doing Add(ZADD) and Remove(ZREM) operations and O(1) time complexity when doing Contains operations.

Package zset is a go implmentation of https://github.com/redis/redis/blob/unstable/src/t_zset.c

Comparison

Redis command	Go function
ZADD	Add
ZINCRBY	IncrBy
ZREM	Remove
ZREMRANGEBYSCORE	RemoveRangeByScore
ZREMRANGEBYRANK	RemoveRangeByRank
ZUNION	Union
ZINTER	Inter
ZINTERCARD	TODO
ZDIFF	TODO
ZRANGE	Range
ZRANGEBYSCORE	IncrBy
ZREVRANGEBYSCORE	RevRangeByScore
ZCOUNT	Count
ZREVRANGE	RevRange
ZCARD	Len
ZSCORE	Score
ZRANK	Rank
ZREVRANK	RevRank
ZPOPMIN	TODO
ZPOPMAX	TODO
ZRANDMEMBER	TODO

List of redis commands are generated from the following command:

cat redis/src/server.c | grep -o '"z.*",z.*Command' | grep -o '".*"' | cut -d '"' -f2

You may find that not all redis commands have corresponding go implementations,
the reason is as follows:

Unsupported Commands

Redis' zset can operates elements in lexicographic order, which is not commonly
used function, so zset does not support commands like ZREMRANGEBYLEX, ZLEXCOUNT
and so on.

Redis command
ZREMRANGEBYLEX
ZRANGEBYLEX
ZREVRANGEBYLEX
ZLEXCOUNT

In redis, user accesses zset via a string key. We do not need such string key
because we have variable. so the following commands are not implemented:

Redis command
ZUNIONSTORE
ZINTERSTORE
ZDIFFSTORE
ZRANGESTORE
ZMSCORE
ZSCAN

Tests

Unittests

coverage: 94.0% of statements

Benchmark

go test -v -cpuprofile cpu.pprof -cpu=1 -run=NOTEST -bench=. -benchmem -benchtime=100000x -count=10 -timeout=60m > y.txt && benchstat y.txt
name                                  time/op
Contains100Hits/sortedset             39.5ns ± 1%
Contains50Hits/sortedset              43.4ns ± 4%
ContainsNoHits/sortedset              21.9ns ± 5%
Add/sortedset                         53.6ns ± 1%
1Add99Contains/sortedset              38.1ns ±12%
10Add90Contains/sortedset             47.6ns ± 5%
50Add50Contains/sortedset             56.9ns ± 3%
1Add3Incr6Remove90Contains/sortedset  49.3ns ±11%

name                                  alloc/op
Contains100Hits/sortedset              0.00B
Contains50Hits/sortedset               0.00B
ContainsNoHits/sortedset               0.00B
Add/sortedset                          2.00B ± 0%
1Add99Contains/sortedset               0.00B
10Add90Contains/sortedset              0.00B
50Add50Contains/sortedset              1.00B ± 0%
1Add3Incr6Remove90Contains/sortedset  0.50B ±100%

name                                  allocs/op
Contains100Hits/sortedset               0.00
Contains50Hits/sortedset                0.00
ContainsNoHits/sortedset                0.00
Add/sortedset                           0.00
1Add99Contains/sortedset                0.00
10Add90Contains/sortedset               0.00
50Add50Contains/sortedset               0.00
1Add3Incr6Remove90Contains/sortedset    0.00

TODOs

luchsh · 2021-10-13T16:22:09Z

/benchdiff

nothing happened; guess this issue should depend on #95

zhangyunhao116

Maybe we could support NewInt in this pull request? (In next PR is also ok)

collection/zset/zset.go

SilverRainZ · 2021-10-18T03:25:10Z

Maybe we could support NewInt in this pull request? (In next PR is also ok)

I prefer in next PR, just a code generateion stuff, we focus on the core implementaion in this PR.

zhangyunhao116

一些风格问题，统一一下就好啦，已经加了 suggestion

collection/zset/oparry.go

collection/zset/zset.go

SilverRainZ · 2021-10-28T03:19:59Z

Any progress？

luchsh · 2021-10-29T05:53:52Z

collection/zset/readme.md

+| ZREMRANGEBYLEX        | NOT SUPPORTED       |
+| ZUNIONSTORE           | UnionStore          |
+| ZINTERSTORE           | InterStore          |
+| ZDIFFSTORE            |                     |


这些空着的command是什么状态？

在内部的需求文档里没覆盖到，因此这次先不实现，以后可能会实现。

<NOT SUPPORTED>, <TODO>

Done, 用斜体区分了，<> 在 markdown 里不友好。

collection/zset/skiplist.go

luchsh · 2021-10-29T12:38:41Z

collection/zset/skiplist.go

+
+const (
+	maxLevel = 32   // same to ZSKIPLIST_MAXLEVEL
+	p        = 0.25 // same to ZSKIPLIST_P


作为常量名字稍微多些清晰的意义吧

这里是保持和 redis 的常量名一致，也许后面加点注释会好一点？

改成 probability 了。

collection/zset/oparry.go

luchsh · 2021-10-29T12:51:57Z

collection/zset/skiplist.go

+		value: value,
+		level: level,
+	}
+	if level > op1 {


如果写成 node.oparr.init(level) 会不会更模块化一些？

是的，之前想这么写，但处于保持一致的原因没有这么做…

collection/zset/zset.go

luchsh · 2021-10-29T13:25:05Z

collection/zset/readme.md

+## Features
+
+- Concurrent safe API
+- Value is sorted with score


Values are always sorted by their score ? 但这句话听起来像是实现上的做法，那用户能感知的特点是啥呢？比如快速的range、rank等api？

我理解这就是元素按分数排序就是对 zset 的主要需求了？

实现上应该说用 skiplist blahblah……

只改了语法……

collection/zset/readme.md

collection/zset/skiplist.go

luchsh · 2021-10-29T13:34:55Z

Any progress？

pls @ the reviewers or shout at them in Feishu :)

luchsh · 2021-11-02T07:02:02Z

collection/zset/readme.md

+| ZREMRANGEBYRANK       | RemoveRangeByRank   |
+| ZREMRANGEBYLEX        | *NOT SUPPORTED*     |
+| ZUNIONSTORE           | *NOT SUITABLE*      |
+| ZINTERSTORE           | *NOT SUITABLE*      |


unsuitable?

重新整理了格式，把不打算支持的命令分开放了。

collection/zset/readme.md

@zhangyunhao116

* release: 20210913 (#83) (#88) * feat: circuitbreaker.panel use skipmap (#78) * fix(metainfo): fix misuse of append (#79) * fix(lscq): add write barrier for LSCQ Pointer (#80) * feat(metainfo): improve backward APIs (#81) * release: 20210913 (#83) (#93) * feat: circuitbreaker.panel use skipmap (#78) * fix(metainfo): fix misuse of append (#79) * fix(lscq): add write barrier for LSCQ Pointer (#80) * feat(metainfo): improve backward APIs (#81) * chore(skipmap,skipset): remove duplicated code generation declaration (#87) * fix(lscq): cas2 use runtime.noescape (#94) * feat: add fastrand.Read() (#90) Co-authored-by: liyichao <liyichao@bytedance.com> * doc: add badge to link to godoc (#99) * ci: skip golang related workflows when there are only doc changes (#101) * ci: add workflow for feishu/lark notification (#100) * ci: add workflow for feishu/lark notification * ci: fix typo * ci: also send feishu notification on issue opened Co-authored-by: Jonathan Lu <luchsh@users.noreply.github.com> * feat(metainfo): define standard for backward prefix (#102) * feat(fastrand): support Read,Shuffle,Perm (#103) * feat(fastrand): Read remove temp buffer (#104) * ci: add github workflow for performance regression check (#95) * ci: add github workflow for performance regression check Comment "/benchdiff" to trigger this check * ci: add comments to pr-benchdiff.yml * ci: report benchdiff result to PR_HEAD's check run * ci: various changes - support "pull_reqeust" event - filter deleted go packages - post a comment on job started - post a comment on job failed * ci: let xargs ignore empty line * ci: skip benchdiff when there are only doc changes * ci: set -x for debugging * ci: make sure we are using github default branch as baseline * ci: set benchtime=1s for benchdiff * chore: add github issue forms (#105) * test(lang/syncx): add benchmark for RWMutex (#89) * fix(xxhash3): add fallback to fix panic occurs on non avx2, non sse2 machine (#108) Co-authored-by: Ye Li <liye.e@bytedance.com> * feat: add zset (#98) * feat: add zset * chore(zset): update comment * docs(zset): add readme * chore: some code style fixes * chore(zset): rename Float64RangeOpt -> RangeOpt * chore(zset): comment style fixes * chore(zset): add a todo about maxLevel * chore(zset): another comment fix * chore(zset): move skiplist impl to another file * chore(zset): add license header for opt.go * chore: add myself as zset's CODEOWNER * zset: don't use z.Range(0, z.Len()-1) * doc: remove redundant section * fix(zset): break when key is not exist * chore(zset): simplify func name * chore(zset): update cheatsheet * chore(zset): meaningful const * refactor(zset): optionalArray.init() * docs: update zset readme * docs(zset): also add @zhangyunhao116 as code owner * docs(zset): some grammar fixes * docs: fix docs typo (#109) * feat: auto tuning gc (#112) * feat: auto tuning gc * doc: gctuner * fix: gctuner tests data race (#113) * fix: gctuner tests data race * chore: add owner * chore(ci): use self-hosted runner (#114) * chore(ci): use self-hosted runner * fix lint Co-authored-by: Shengyu Zhang <reg@silverrainz.me> Co-authored-by: ziposcar <499581494@qq.com> Co-authored-by: liyichao <liyichao@bytedance.com> Co-authored-by: Shengyu Zhang <zhangshengyu.0@bytedance.com> Co-authored-by: Jonathan Lu <luchsh@users.noreply.github.com> Co-authored-by: Pure White <wudi.daniel@bytedance.com> Co-authored-by: lyeeeeee <awesomeyeli@gmail.com> Co-authored-by: Ye Li <liye.e@bytedance.com> Co-authored-by: chyroc <chyroc@qq.com> Co-authored-by: Joway <joway.w@gmail.com> Co-authored-by: Joway <wangzhuowei@bytedance.com>

SilverRainZ requested a review from zhangyunhao116 October 13, 2021 03:03

SilverRainZ marked this pull request as ready for review October 13, 2021 03:05

SilverRainZ requested a review from PureWhiteWu as a code owner October 13, 2021 03:05

zhangyunhao116 reviewed Oct 15, 2021

View reviewed changes

zhangyunhao116 reviewed Oct 18, 2021

View reviewed changes

luchsh self-requested a review October 19, 2021 05:27

SilverRainZ added the enhancement New feature or request label Oct 29, 2021

luchsh reviewed Oct 29, 2021

View reviewed changes

luchsh reviewed Nov 2, 2021

View reviewed changes

SilverRainZ force-pushed the feat/zset branch from 0072fb2 to 7c09d20 Compare November 2, 2021 08:41

luchsh previously approved these changes Nov 3, 2021

View reviewed changes

zhangyunhao116 reviewed Nov 3, 2021

View reviewed changes

collection/zset/readme.md Outdated Show resolved Hide resolved

collection/zset/readme.md Outdated Show resolved Hide resolved

SilverRainZ dismissed luchsh’s stale review via da8d7e7 November 3, 2021 08:22

zhangyunhao116 previously approved these changes Nov 3, 2021

View reviewed changes

luchsh previously approved these changes Nov 3, 2021

View reviewed changes

SilverRainZ added 11 commits November 3, 2021 16:38

feat: add zset

46dff14

chore(zset): update comment

4bffc12

docs(zset): add readme

e73098a

chore: some code style fixes

391f976

chore(zset): rename Float64RangeOpt -> RangeOpt

a2151c5

chore(zset): comment style fixes

f10e7b9

chore(zset): add a todo about maxLevel

ce93486

chore(zset): another comment fix

cfe3082

chore(zset): move skiplist impl to another file

f5126db

chore(zset): add license header for opt.go

1ab8eb4

chore: add myself as zset's CODEOWNER

7e5fbcb

SilverRainZ added 10 commits November 3, 2021 16:38

zset: don't use z.Range(0, z.Len()-1)

84f0e95

doc: remove redundant section

dc377d8

fix(zset): break when key is not exist

aed0dc8

chore(zset): simplify func name

2b6fb51

chore(zset): update cheatsheet

33fe348

chore(zset): meaningful const

6417eba

refactor(zset): optionalArray.init()

5ad25da

docs: update zset readme

63455cd

docs(zset): also add @zhangyunhao116 as code owner

a93c5ca

docs(zset): some grammar fixes

820432c

SilverRainZ dismissed stale reviews from luchsh and zhangyunhao116 via 820432c November 3, 2021 08:38

SilverRainZ force-pushed the feat/zset branch from da8d7e7 to 820432c Compare November 3, 2021 08:38

luchsh approved these changes Nov 3, 2021

View reviewed changes

zhangyunhao116 approved these changes Nov 3, 2021

View reviewed changes

SilverRainZ merged commit d4719f7 into develop Nov 3, 2021

SilverRainZ deleted the feat/zset branch November 3, 2021 09:05

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: add zset #98

feat: add zset #98

SilverRainZ commented Oct 12, 2021 •

edited

Loading

luchsh commented Oct 13, 2021 •

edited

Loading

zhangyunhao116 left a comment

SilverRainZ commented Oct 18, 2021

zhangyunhao116 left a comment

SilverRainZ commented Oct 28, 2021

luchsh Oct 29, 2021

SilverRainZ Nov 1, 2021

luchsh Nov 1, 2021

SilverRainZ Nov 2, 2021

luchsh Oct 29, 2021

SilverRainZ Nov 1, 2021

SilverRainZ Nov 2, 2021

luchsh Oct 29, 2021

SilverRainZ Nov 1, 2021

SilverRainZ Nov 2, 2021

luchsh Oct 29, 2021

SilverRainZ Nov 1, 2021

SilverRainZ Nov 2, 2021

luchsh commented Oct 29, 2021 •

edited

Loading

luchsh Nov 2, 2021

SilverRainZ Nov 2, 2021

feat: add zset #98

feat: add zset #98

Conversation

SilverRainZ commented Oct 12, 2021 • edited Loading

Intro

Comparison

Unsupported Commands

Tests

Unittests

Benchmark

TODOs

luchsh commented Oct 13, 2021 • edited Loading

zhangyunhao116 left a comment

Choose a reason for hiding this comment

SilverRainZ commented Oct 18, 2021

zhangyunhao116 left a comment

Choose a reason for hiding this comment

SilverRainZ commented Oct 28, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

luchsh commented Oct 29, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

SilverRainZ commented Oct 12, 2021 •

edited

Loading

luchsh commented Oct 13, 2021 •

edited

Loading

luchsh commented Oct 29, 2021 •

edited

Loading