Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement in-memory DB to run Yorkie without MongoDB #276

Merged
merged 1 commit into from
Nov 30, 2021
Merged

Conversation

hackerwins
Copy link
Member

@hackerwins hackerwins commented Nov 25, 2021

What this PR does / why we need it:

If mongo-connection-uri flag is set when the Agent is running, MongoDB is used, otherwise MemDB is used.
When using MemDB, Agent stores its data in memory, so stopping the server will lose all data.

Run Yorkie without MongoDB:

$ ./bin/yorkie agent
2021-11-28T00:24:13.931+0900	INFO	backend created: id: c6h4r7d94810ghl74dhg, rpc: localhost:11101: db: memory
2021-11-28T00:24:13.932+0900	INFO	serving profiling on 11102
2021-11-28T00:24:13.932+0900	INFO	serving RPC on 11101

Run Yorkie with MongoDB:

$ yorkie agent --mongo-connection-uri mongodb://localhost:27017
2021-11-28T00:23:46.842+0900	INFO	MongoDB connected, URI: mongodb://localhost:27017, DB: yorkie-meta
2021-11-28T00:23:46.842+0900	INFO	backend created: id: c6h4r0l94810ggn6obgg, rpc: localhost:11101: db: mongodb://localhost:27017
2021-11-28T00:23:46.843+0900	INFO	serving profiling on 11102
2021-11-28T00:23:46.843+0900	INFO	serving RPC on 11101
  • I used github.com/hashicorp/go-memdb for implementation.
  • Hexagonal, which I worked with @dc7303 last winter, works well this time as well.

Which issue(s) this PR fixes:

Fixes #275

Special notes for your reviewer:

This PR starts the Memory DB implementation. If we can't finish all in this PR, we can just merge this and wait for help from other contributors.

Tasks:

  • ActivateClient
  • DeactivateClient
  • FindClientInfoByID
  • UpdateClientInfoAfterPushPull
  • FindDocInfoByKey
  • StoreChangeInfos
  • FindChangesBetweenServerSeqs
  • FindChangeInfosBetweenServerSeqs
  • CreateSnapshotInfo
  • FindLastSnapshotInfo
  • UpdateAndFindMinSyncedTicket

Finally, implemented all DB interfaces in the memory version.

Does this PR introduce a user-facing change?:


Additional documentation:


Checklist:

  • Added relevant tests or not required
  • Didn't break anything

@codecov
Copy link

codecov bot commented Nov 25, 2021

Codecov Report

Merging #276 (d8f7a9d) into main (e544cdb) will decrease coverage by 0.27%.
The diff coverage is 37.46%.

Impacted file tree graph

@@            Coverage Diff             @@
##             main     #276      +/-   ##
==========================================
- Coverage   50.21%   49.94%   -0.28%     
==========================================
  Files          53       54       +1     
  Lines        4375     4655     +280     
==========================================
+ Hits         2197     2325     +128     
- Misses       1966     2087     +121     
- Partials      212      243      +31     
Impacted Files Coverage Δ
pkg/document/document.go 50.72% <0.00%> (-0.75%) ⬇️
yorkie/backend/backend.go 0.00% <0.00%> (ø)
yorkie/backend/db/change_info.go 26.92% <ø> (ø)
yorkie/backend/db/client_info.go 39.62% <0.00%> (-14.23%) ⬇️
yorkie/backend/db/doc_info.go 0.00% <0.00%> (ø)
yorkie/backend/db/mongo/client.go 0.00% <0.00%> (ø)
yorkie/backend/db/mongo/indexes.go 0.00% <0.00%> (ø)
yorkie/backend/db/memory/db.go 48.51% <48.51%> (ø)
yorkie/backend/db/db.go 66.66% <50.00%> (ø)
yorkie/rpc/yorkie_server.go 42.85% <50.00%> (-0.23%) ⬇️
... and 1 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update e544cdb...d8f7a9d. Read the comment docs.

@hackerwins hackerwins force-pushed the memdb branch 5 times, most recently from 66bb781 to 0f24aad Compare November 27, 2021 15:05
@hackerwins hackerwins requested a review from a team November 27, 2021 15:27
@hackerwins hackerwins force-pushed the memdb branch 5 times, most recently from 9652e6d to 0d84941 Compare November 28, 2021 02:27
@hackerwins
Copy link
Member Author

I used MemoryDB for Rust SDK integration test. Response times were shorter than when using MongoDB.

$ yorkie agent --mongo-connection-uri mongodb://localhost:27017
2021-11-28T21:11:02.915+0900	INFO	MongoDB connected, URI: mongodb://localhost:27017, DB: yorkie-meta
2021-11-28T21:11:02.915+0900	INFO	backend created: id: c6hn3lnblarmnt1sq8f0, rpc: localhost:11101: db: mongodb://localhost:27017
2021-11-28T21:11:02.915+0900	INFO	serving profiling on 11102
2021-11-28T21:11:02.915+0900	INFO	serving RPC on 11101
2021-11-28T21:11:11.959+0900	INFO	RPC : "/api.Yorkie/ActivateClient" 5.954118ms
2021-11-28T21:11:11.962+0900	INFO	RPC : "/api.Yorkie/ActivateClient" 8.416765ms
2021-11-28T21:11:11.963+0900	INFO	RPC : "/api.Yorkie/ActivateClient" 10.306341ms
2021-11-28T21:11:11.966+0900	INFO	RPC : "/api.Yorkie/DeactivateClient" 2.281372ms
^C2021-11-28T21:11:17.952+0900	INFO	Caught signal: interrupt
2021-11-28T21:11:17.956+0900	INFO	backend stoped: id: c6hn3lnblarmnt1sq8f0, rpc: localhost:11101
$ yorkie agent
2021-11-28T21:11:22.569+0900	INFO	backend created: id: c6hn3qnblarmo8ro91d0, rpc: localhost:11101: db: memory
2021-11-28T21:11:22.570+0900	INFO	serving profiling on 11102
2021-11-28T21:11:22.570+0900	INFO	serving RPC on 11101
2021-11-28T21:11:24.439+0900	INFO	RPC : "/api.Yorkie/ActivateClient" 32.881µs
2021-11-28T21:11:24.439+0900	INFO	RPC : "/api.Yorkie/ActivateClient" 91.922µs
2021-11-28T21:11:24.440+0900	INFO	RPC : "/api.Yorkie/ActivateClient" 145.309µs
2021-11-28T21:11:24.442+0900	INFO	RPC : "/api.Yorkie/DeactivateClient" 23.304µs

Copy link
Member

@dc7303 dc7303 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for sharing your awesome work! 👍
I have some questions and suggestions. Please check. 🙇

pkg/document/document.go Show resolved Hide resolved
yorkie/backend/db/doc_info.go Outdated Show resolved Hide resolved
@dc7303 dc7303 assigned hackerwins and unassigned dc7303 Nov 28, 2021
@hackerwins hackerwins assigned dc7303 and unassigned hackerwins Nov 29, 2021
@hackerwins hackerwins requested review from dc7303 and ppeeou November 29, 2021 01:20
@hackerwins hackerwins force-pushed the memdb branch 2 times, most recently from 65104f5 to 4300cd9 Compare November 29, 2021 04:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Implement an in-memory DB of db interface
4 participants