Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add TiKV support for cache persistence #69

Open
tstromberg opened this issue May 7, 2020 · 9 comments · Fixed by #73
Open

Add TiKV support for cache persistence #69

tstromberg opened this issue May 7, 2020 · 9 comments · Fixed by #73
Labels
enhancement New feature or request

Comments

@tstromberg
Copy link
Collaborator

It seems to be a popular Kubernetes option.

@tstromberg tstromberg added good first issue Good for newcomers help wanted Extra attention is needed labels May 7, 2020
@tstromberg
Copy link
Collaborator Author

If someone wants to take this on, it's pretty simple:

masihtehrani added a commit to masihtehrani/triage-party that referenced this issue May 8, 2020
@masihtehrani
Copy link
Contributor

masihtehrani commented May 8, 2020

If someone wants to take this on, it's pretty simple:

I started to implement Tikv same Postgres but it has different concept because it just has GET/PUT/DELETE/SCAN operation and got a problem for implement for methods loadItems() & cleanup() because I and find operation for select all and delete all keys values, I think it must be implement the same mem

@tstromberg
Copy link
Collaborator Author

tstromberg commented May 9, 2020

Thanks!

  • cleanup() is more of a bad hack than an important feature. With TiKV, you can simply set an expiration time for keys to keep them from filling up the database. For instance, a 30-day expiration should be plenty of time.

  • For loadItems(), here's an idea: have Save() store a key (named something like saved-keys) with a list of keys that it has saved. Then your loadItems() implementation can call fetch the value for saved-keys, iterate over the values, and call setMem() on each item.

What do you think?

@masihtehrani
Copy link
Contributor

masihtehrani commented May 10, 2020

Thanks!

  • cleanup() is more of a bad hack than an important feature. With TiKV, you can simply set an expiration time for keys to keep them from filling up the database. For instance, a 30-day expiration should be plenty of time.
  • For loadItems(), here's an idea: have Save() store a key (named something like saved-keys) with a list of keys that it has saved. Then your loadItems() implementation can call fetch the value for saved-keys, iterate over the values, and call setMem() on each item.

What do you think?

Tikv is the tiny project doesn't have TTL and select all it just has Scan(startKey, endKey []byte, limit int) maybe we can do something with scan method. I have some suggestions.

  1. I think about we can get all persist data in one key-value with storage with JSON data for example key: persis, value: {"key1":"value1"} and handle in Golang but I don't have an idea about implement TTL and this function m.cleanup(start.Add(-1 * time.Hour)) now.
  2. We can use TIDB it has SQL and we can implement same Postgres
  3. Chose Another KV store same Redis does have TTL, select all and...

@tstromberg
Copy link
Collaborator Author

OK. Let's skip cleanup(), as it's a private function and not necessary. TTL's aren't necessary either, as they are not part of the API.

For an initial iteration, I say let's just ignore TTL's altogether. If you can figure out how to make Scan work, then great. If not, you can fall back on saving a list of keys to load at a specific key.

If you go the Scan route, in a second iteration, we can implement TTL's in a round-about way by refusing to load values into memory if the creation time is older than a set time (30-days?).

masihtehrani added a commit to masihtehrani/triage-party that referenced this issue May 11, 2020
@masihtehrani
Copy link
Contributor

OK. Let's skip cleanup(), as it's a private function and not necessary. TTL's aren't necessary either, as they are not part of the API.

For an initial iteration, I say let's just ignore TTL's altogether. If you can figure out how to make Scan work, then great. If not, you can fall back on saving a list of keys to load at a specific key.

If you go the Scan route, in a second iteration, we can implement TTL's in a round-about way by refusing to load values into memory if the creation time is older than a set time (30-days?).

I try to do this with a simple key, Please review my PR

@tstromberg tstromberg added enhancement New feature or request and removed good first issue Good for newcomers help wanted Extra attention is needed labels May 12, 2020
@tstromberg tstromberg reopened this May 12, 2020
@tstromberg
Copy link
Collaborator Author

I'm temporarily going to back the PR out because I can't seem to get it to compile:

 ~/s/triage-party   tikv-cleanup $…  go run cmd/server/main.go                                                                                                                                             Mon May 11 21:57:38 2020
go: finding module for package google.golang.org/grpc/codes
go: finding module for package google.golang.org/grpc/keepalive
go: finding module for package github.com/pingcap/pd/pd-client
go: finding module for package google.golang.org/grpc/grpclog
go: finding module for package google.golang.org/grpc/encoding
go: finding module for package google.golang.org/grpc/internal
go: finding module for package google.golang.org/grpc/encoding/proto
go: finding module for package google.golang.org/grpc/status
go: finding module for package google.golang.org/grpc/peer
go: finding module for package google.golang.org/grpc/balancer
go: finding module for package google.golang.org/grpc/attributes
go: finding module for package google.golang.org/grpc/serviceconfig
go: finding module for package google.golang.org/grpc/internal/resolver/passthrough
go: finding module for package google.golang.org/grpc/connectivity
go: finding module for package google.golang.org/grpc/internal/resolver/dns
go: found google.golang.org/grpc/codes in google.golang.org/grpc v1.29.1
go: finding module for package github.com/pingcap/pd/pd-client
../../go/pkg/mod/github.com/pingcap/tidb@v2.0.11+incompatible/store/tikv/kv.go:28:2: module github.com/pingcap/pd@latest found (v2.1.19+incompatible), but does not contain package github.com/pingcap/pd/pd-client

I spent an hour trying to debug the go.mod madness, and came to the conclusion that:

  • pingcap did bad things
  • grpc did bad things
  • tikv and cloudsql-proxy want different versions of grpc, ultimately.

@masihtehrani
Copy link
Contributor

I'm temporarily going to back the PR out because I can't seem to get it to compile:

 ~/s/triage-party   tikv-cleanup $…  go run cmd/server/main.go                                                                                                                                             Mon May 11 21:57:38 2020
go: finding module for package google.golang.org/grpc/codes
go: finding module for package google.golang.org/grpc/keepalive
go: finding module for package github.com/pingcap/pd/pd-client
go: finding module for package google.golang.org/grpc/grpclog
go: finding module for package google.golang.org/grpc/encoding
go: finding module for package google.golang.org/grpc/internal
go: finding module for package google.golang.org/grpc/encoding/proto
go: finding module for package google.golang.org/grpc/status
go: finding module for package google.golang.org/grpc/peer
go: finding module for package google.golang.org/grpc/balancer
go: finding module for package google.golang.org/grpc/attributes
go: finding module for package google.golang.org/grpc/serviceconfig
go: finding module for package google.golang.org/grpc/internal/resolver/passthrough
go: finding module for package google.golang.org/grpc/connectivity
go: finding module for package google.golang.org/grpc/internal/resolver/dns
go: found google.golang.org/grpc/codes in google.golang.org/grpc v1.29.1
go: finding module for package github.com/pingcap/pd/pd-client
../../go/pkg/mod/github.com/pingcap/tidb@v2.0.11+incompatible/store/tikv/kv.go:28:2: module github.com/pingcap/pd@latest found (v2.1.19+incompatible), but does not contain package github.com/pingcap/pd/pd-client

I spent an hour trying to debug the go.mod madness, and came to the conclusion that:

  • pingcap did bad things
  • grpc did bad things
  • tikv and cloudsql-proxy want different versions of grpc, ultimately.

I try to fix it

@tstromberg
Copy link
Collaborator Author

tstromberg commented May 12, 2020

Once etcd-io/etcd#11881 is merged, this may just work.

Alternatively, selecting a version of cloudsqlproxy and tikv of the same age will likely work as well, but I couldn't get it on my first attempt.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants