Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Obsidian Version 8.0 #77

Merged
merged 58 commits into from
Apr 12, 2023
Merged

Obsidian Version 8.0 #77

merged 58 commits into from
Apr 12, 2023

Conversation

DavidMNorman
Copy link
Contributor

@DavidMNorman DavidMNorman commented Apr 12, 2023 •
edited
Loading

Checklist

  • Bugfix
  • New feature
  • Refactor

Related Issue

  • LFU/LRU caching is not as performant as other client caching options
  • Algorithms other than LFU would crash apps using the Obsidian Wrapper
  • Large queries are expensive to send to the server on client cache misses
  • ROOT_QUERY could not provide hashes to individual item queries after broad queries and would miss the cache even though the item was present in it
  • Client was not set up to communicate with rebuilt devtool
  • Server side caching was nonfunctional

Solution

  • Implemented W-TinyLFU to provide better hit-ratios with minimal memory overhead
  • Refactored ObsidianWrapper.jsx to initialize state once based on the selected algorithm rather than updating state with a new cache twice.
  • Implemented persistent queries to allow only the sending of a query hash between client and server
  • Implemented searchTerms option to allow developers to specify what search categories their app uses, which saves extra entries to ROOT_QUERY that can later be found by individual queries, reducing the need for endpoint queries when the item is in the cache.
  • Added messaging from the Obsidian Wrapper to send necessary information to our developer tool.
  • Completely overhauled server-side caching using a denormalized approach that restores basic read and write functionality.

Additional Info

  • Included Rhum tests for WTinyLFU algorithm
  • Cleared out vestigial files no longer being used by any active part of the codebase, including documentation folder, which contained old, deprecated versions of the codebase with comments. Did leave old test files intact, however.
  • Updated the Obsidian Logo

DavidMNorman and others added 30 commits March 22, 2023 10:08
@DavidMNorman @davidtoyoukim
@exlxxn @JoanManto
Initial w-tinyLFU structure
16188f5 
Co-authored-by: David Norman <120237773+DavidMNorman@users.noreply.github.com>
Co-authored-by: David Kim <david.toyou.kim@gmail.com>
Co-authored-by: Eileen Cho <109772371+exlxxn@users.noreply.github.com>
Co-authored-by: Joan Manto <120276510+JoanManto@users.noreply.github.com>
@DavidMNorman @davidtoyoukim
@exlxxn @JoanManto
Add Peek/Delete metehods to LRU
0bb0eec 
Create WLRU and SLRU structures / methods
Create Frequency Sketch frame.

Co-authored-by: David Norman <120237773+DavidMNorman@users.noreply.github.com>
Co-authored-by: David Kim <david.toyou.kim@gmail.com>
Co-authored-by: Eileen Cho <109772371+exlxxn@users.noreply.github.com>
Co-authored-by: Joan Manto <120276510+JoanManto@users.noreply.github.com>
@DavidMNorman @davidtoyoukim
@exlxxn @JoanManto
Build out frequency Sketch,
a14733b 
Fix SLRU/LRU methods for protected eviction retention,
Create WTLFU frame

Co-authored-by: David Kim <david.toyou.kim@gmail.com>
Co-authored-by: Eileen Cho <109772371+exlxxn@users.noreply.github.com>
Co-authored-by: Joan Manto <120276510+JoanManto@users.noreply.github.com>
@DavidMNorman
Merge pull request #1 from oslabs-beta/david/w-tinylfu
e506973 
David/w-tinylfu
@DavidMNorman @davidtoyoukim
@exlxxn @JoanManto
w-tinyLFU and frequency sketch updates
9254ac8 
Update read/write for w-tinyLFU
update file structure for the w-tinyLFU cache
Finish first build of Frequency Sketch

Co-authored-by: David Norman <120237773+DavidMNorman@users.noreply.github.com>
Co-authored-by: David Kim <david.toyou.kim@gmail.com>
Co-authored-by: Eileen Cho <109772371+exlxxn@users.noreply.github.com>
Co-authored-by: Joan Manto <120276510+JoanManto@users.noreply.github.com>
@DavidMNorman @davidtoyoukim
@exlxxn @JoanManto
Complete basic WTLFU functionality w/ Freq Sketch,
3bc7c3e 
refactor Frequency Sketch,
begin TS refactor for WTLFU

Co-authored-by: David Norman <120237773+DavidMNorman@users.noreply.github.com>
Co-authored-by: David Kim <david.toyou.kim@gmail.com>
Co-authored-by: Eileen Cho <109772371+exlxxn@users.noreply.github.com>
Co-authored-by: Joan Manto <120276510+JoanManto@users.noreply.github.com>
@DavidMNorman
Merge pull request #2 from oslabs-beta/david/w-tinylfu
5e72271 
David/w tinylfu
@DavidMNorman @davidtoyoukim
@exlxxn @JoanManto
added code to Obsidian Wrapper to send messages to DevTool
3453d77 
Co-authored-by: David Norman <120237773+DavidMNorman@users.noreply.github.com>
Co-authored-by: David Kim <david.toyou.kim@gmail.com>
Co-authored-by: Eileen Cho <109772371+exlxxn@users.noreply.github.com>
Co-authored-by: Joan Manto <120276510+JoanManto@users.noreply.github.com>
@JoanManto
Merge pull request #3 from oslabs-beta/ECJM/wrapper
a9eec4a 
added and fixed code in Obsidian Wrapper
@DavidMNorman @davidtoyoukim
@exlxxn @JoanManto
WTinyLFU bugfixes, begin hillclimber optimization
89d1a33 
Co-authored-by: David Norman <120237773+DavidMNorman@users.noreply.github.com>
Co-authored-by: David Kim <david.toyou.kim@gmail.com>
Co-authored-by: Eileen Cho <109772371+exlxxn@users.noreply.github.com>
Co-authored-by: Joan Manto <120276510+JoanManto@users.noreply.github.com>
@DavidMNorman
Merge branch 'dev' into david/w-tinylfu
a63df68
@exlxxn @DavidMNorman
@davidtoyoukim @JoanManto
added logic to respond to request for algo/capacity from content.js
1054717 
Co-authored-by: David Norman <120237773+DavidMNorman@users.noreply.github.com>
Co-authored-by: David Kim <david.toyou.kim@gmail.com>
Co-authored-by: Eileen Cho <109772371+exlxxn@users.noreply.github.com>
Co-authored-by: Joan Manto <120276510+JoanManto@users.noreply.github.com>
@DavidMNorman @davidtoyoukim
@exlxxn @JoanManto
Debug cache partitions,
6de73ed 
Debug cache evictions,
Debug cache put methods
debug frequency sketch incrementation, initilization, freq checking.

Co-authored-by: David Norman <120237773+DavidMNorman@users.noreply.github.com>
Co-authored-by: David Kim <david.toyou.kim@gmail.com>
Co-authored-by: Eileen Cho <109772371+exlxxn@users.noreply.github.com>
Co-authored-by: Joan Manto <120276510+JoanManto@users.noreply.github.com>
@DavidMNorman
Merge pull request #5 from oslabs-beta/david/w-tinylfu
045d298 
David/w tinylfu
@exlxxn
Merge pull request #4 from oslabs-beta/ECJM/algoCap
2cc8304 
added logic to respond to request for algo/capacity from content.js
@DavidMNorman @davidtoyoukim
@exlxxn @JoanManto
Fix bug with SLRU methods
c415812 
Co-authored-by: David Norman <120237773+DavidMNorman@users.noreply.github.com>
Co-authored-by: David Kim <david.toyou.kim@gmail.com>
Co-authored-by: Eileen Cho <109772371+exlxxn@users.noreply.github.com>
Co-authored-by: Joan Manto <120276510+JoanManto@users.noreply.github.com>
@DavidMNorman @davidtoyoukim
@exlxxn @JoanManto
wTinyLFU Bugfixes
23d1249 
Co-authored-by: David Norman <120237773+DavidMNorman@users.noreply.github.com>
Co-authored-by: David Kim <david.toyou.kim@gmail.com>
Co-authored-by: Eileen Cho <109772371+exlxxn@users.noreply.github.com>
Co-authored-by: Joan Manto <120276510+JoanManto@users.noreply.github.com>
@DavidMNorman
Merge pull request #6 from oslabs-beta/david/w-tinylfu
bfe42b2 
wTinyLFU Bugfixes
@DavidMNorman @davidtoyoukim
@exlxxn @JoanManto
Add searchTerms options to w-tinyLFU
ace5c0c 
Co-authored-by: David Norman <120237773+DavidMNorman@users.noreply.github.com>
Co-authored-by: David Kim <david.toyou.kim@gmail.com>
Co-authored-by: Eileen Cho <109772371+exlxxn@users.noreply.github.com>
Co-authored-by: Joan Manto <120276510+JoanManto@users.noreply.github.com>
@DavidMNorman @davidtoyoukim
@exlxxn @JoanManto
Add searchTerms to LFU and LRU
c9e9721 
Co-authored-by: David Norman <120237773+DavidMNorman@users.noreply.github.com>
Co-authored-by: David Kim <david.toyou.kim@gmail.com>
Co-authored-by: Eileen Cho <109772371+exlxxn@users.noreply.github.com>
Co-authored-by: Joan Manto <120276510+JoanManto@users.noreply.github.com>
@DavidMNorman
Merge pull request #7 from oslabs-beta/david/w-tinylfu
8f0166e 
Add searchTerms options to w-tinyLFU
@DavidMNorman @davidtoyoukim
@exlxxn @JoanManto
Add comments, clean up code, update readme
4547833 
Co-authored-by: David Norman <120237773+DavidMNorman@users.noreply.github.com>
Co-authored-by: David Kim <david.toyou.kim@gmail.com>
Co-authored-by: Eileen Cho <109772371+exlxxn@users.noreply.github.com>
Co-authored-by: Joan Manto <120276510+JoanManto@users.noreply.github.com>
@exlxxn @DavidMNorman
@davidtoyoukim @JoanManto
added functionality for persistent queries, refactored ObsidianRouter
64687a9 
Co-authored-by: David Norman <120237773+DavidMNorman@users.noreply.github.com>
Co-authored-by: David Kim <david.toyou.kim@gmail.com>
Co-authored-by: Eileen Cho <109772371+exlxxn@users.noreply.github.com>
Co-authored-by: Joan Manto <120276510+JoanManto@users.noreply.github.com>
@exlxxn
Merge pull request #8 from oslabs-beta/persistentQueries
271fa9c 
added functionality for persistent queries, refactored ObsidianRouter
@DavidMNorman @davidtoyoukim
@exlxxn @JoanManto
Fix authors list in readme
5ea5a30 
Co-authored-by: David Norman <120237773+DavidMNorman@users.noreply.github.com>
Co-authored-by: David Kim <david.toyou.kim@gmail.com>
Co-authored-by: Eileen Cho <109772371+exlxxn@users.noreply.github.com>
Co-authored-by: Joan Manto <120276510+JoanManto@users.noreply.github.com>
@DavidMNorman @davidtoyoukim
@exlxxn @JoanManto
Add useCache as Wrapper option to toggle client cache
1047a66 
Co-authored-by: David Norman <120237773+DavidMNorman@users.noreply.github.com>
Co-authored-by: David Kim <david.toyou.kim@gmail.com>
Co-authored-by: Eileen Cho <109772371+exlxxn@users.noreply.github.com>
Co-authored-by: Joan Manto <120276510+JoanManto@users.noreply.github.com>
@DavidMNorman @davidtoyoukim
@exlxxn @JoanManto
Improve Readme to reflect new options and fix logic
8095053 
Co-authored-by: David Norman <120237773+DavidMNorman@users.noreply.github.com>
Co-authored-by: David Kim <david.toyou.kim@gmail.com>
Co-authored-by: Eileen Cho <109772371+exlxxn@users.noreply.github.com>
Co-authored-by: Joan Manto <120276510+JoanManto@users.noreply.github.com>
@DavidMNorman
Merge pull request #9 from oslabs-beta/cacheSetting
91f46e9 
Cache setting
@exlxxn @DavidMNorman
@davidtoyoukim @JoanManto
enable persistent querying within obsidian wrapper. refactor obsidian…
b0a5273 
… router to access queryStr where necessary

Co-authored-by: David Norman <120237773+DavidMNorman@users.noreply.github.com>
Co-authored-by: David Kim <david.toyou.kim@gmail.com>
Co-authored-by: Eileen Cho <109772371+exlxxn@users.noreply.github.com>
Co-authored-by: Joan Manto <120276510+JoanManto@users.noreply.github.com>
@exlxxn
Merge branch 'dev' into persistentWrapper
1e26f7a
exlxxn and others added 28 commits April 6, 2023 18:53
@exlxxn
Merge pull request #10 from oslabs-beta/persistentWrapper
0bbac37 
Persistent wrapper
@exlxxn @DavidMNorman
@davidtoyoukim @JoanManto
refactored cache initialization, refactored cache class (e.g. read, w…
c3bc85f 
…rite, createqueryKey, constructor), refactored Obsidian.ts, refactore normalize.ts

Co-authored-by: David Norman <120237773+DavidMNorman@users.noreply.github.com>
Co-authored-by: David Kim <david.toyou.kim@gmail.com>
Co-authored-by: Eileen Cho <109772371+exlxxn@users.noreply.github.com>
Co-authored-by: Joan Manto <120276510+JoanManto@users.noreply.github.com>
@exlxxn
Merge pull request #11 from oslabs-beta/cleanCache
fb89757 
refactored cache initialization, refactored cache class (e.g. read, w…
@exlxxn @DavidMNorman
@davidtoyoukim @JoanManto
fixed typos
85c959e 
Co-authored-by: David Norman <120237773+DavidMNorman@users.noreply.github.com>
Co-authored-by: David Kim <david.toyou.kim@gmail.com>
Co-authored-by: Eileen Cho <109772371+exlxxn@users.noreply.github.com>
Co-authored-by: Joan Manto <120276510+JoanManto@users.noreply.github.com>
@exlxxn
Merge pull request #12 from oslabs-beta/fixErrors
02b10f3 
fixed typos
@exlxxn @DavidMNorman
@davidtoyoukim @JoanManto
clean up console.logs from Obsidian.ts, queryHash.js, quickCache.js, …
b848e02 
…normalize.ts, ObsidianWrapper.jsx

Co-authored-by: David Norman <120237773+DavidMNorman@users.noreply.github.com>
Co-authored-by: David Kim <david.toyou.kim@gmail.com>
Co-authored-by: Eileen Cho <109772371+exlxxn@users.noreply.github.com>
Co-authored-by: Joan Manto <120276510+JoanManto@users.noreply.github.com>
@exlxxn
Merge pull request #13 from oslabs-beta/cleanUp
87811df 
clean up console.logs from Obsidian.ts, queryHash.js, quickCache.js, …
@DavidMNorman @davidtoyoukim
@exlxxn @JoanManto
Clean code, remove console logs, add comments, remove vestigial files
b2601aa 
Co-authored-by: David Norman <120237773+DavidMNorman@users.noreply.github.com>
Co-authored-by: David Kim <david.toyou.kim@gmail.com>
Co-authored-by: Eileen Cho <109772371+exlxxn@users.noreply.github.com>
Co-authored-by: Joan Manto <120276510+JoanManto@users.noreply.github.com>
@DavidMNorman
Merge pull request #14 from oslabs-beta/cleanup
6925b39 
Clean code, remove console logs, add comments, remove vestigial files
@DavidMNorman @davidtoyoukim
@exlxxn @JoanManto
Update readme, prune router settings
b63649a 
Co-authored-by: David Norman <120237773+DavidMNorman@users.noreply.github.com>
Co-authored-by: David Kim <david.toyou.kim@gmail.com>
Co-authored-by: Eileen Cho <109772371+exlxxn@users.noreply.github.com>
Co-authored-by: Joan Manto <120276510+JoanManto@users.noreply.github.com>
@DavidMNorman @davidtoyoukim
@exlxxn @JoanManto
added all versions of new logo and banner to repo for future use, upd…
61e7af0 
…ated the readme for links to demo and devtool

Co-authored-by: David Norman <120237773+DavidMNorman@users.noreply.github.com>
Co-authored-by: David Kim <david.toyou.kim@gmail.com>
Co-authored-by: Eileen Cho <109772371+exlxxn@users.noreply.github.com>
Co-authored-by: Joan Manto <120276510+JoanManto@users.noreply.github.com>
@JoanManto
Merge pull request #15 from oslabs-beta/logos
51917a4 
added all versions of new logo and banner to repo for future use, upd…
@DavidMNorman
Initial Test setup
06d3536
@DavidMNorman
Merge branch 'dev' into readme
51331a8
@DavidMNorman @davidtoyoukim
@exlxxn @JoanManto
Add W-tinyLFU testing, SLRU bugfix
c460c48 
Co-authored-by: David Norman <120237773+DavidMNorman@users.noreply.github.com>
Co-authored-by: David Kim <david.toyou.kim@gmail.com>
Co-authored-by: Eileen Cho <109772371+exlxxn@users.noreply.github.com>
Co-authored-by: Joan Manto <120276510+JoanManto@users.noreply.github.com>
@DavidMNorman
Merge branch 'dev' into tests
bb161e9
@DavidMNorman
Adjust Readme
b2cadc2
@DavidMNorman
Merge pull request #16 from oslabs-beta/tests
0fb9658 
Tests & bugfix
@DavidMNorman @davidtoyoukim
@exlxxn @JoanManto
fix readme
03158ff 
Co-authored-by: David Norman <120237773+DavidMNorman@users.noreply.github.com>
Co-authored-by: David Kim <david.toyou.kim@gmail.com>
Co-authored-by: Eileen Cho <109772371+exlxxn@users.noreply.github.com>
Co-authored-by: Joan Manto <120276510+JoanManto@users.noreply.github.com>
@DavidMNorman
Merge branch 'dev' into readme
d92b195
@DavidMNorman
Merge pull request #17 from oslabs-beta/readme
c5f7293 
Readme Fixes
@exlxxn @DavidMNorman
@davidtoyoukim @JoanManto
update readme
6dee551 
Co-authored-by: David Norman <120237773+DavidMNorman@users.noreply.github.com>
Co-authored-by: David Kim <david.toyou.kim@gmail.com>
Co-authored-by: Eileen Cho <109772371+exlxxn@users.noreply.github.com>
Co-authored-by: Joan Manto <120276510+JoanManto@users.noreply.github.com>
@exlxxn
Merge pull request #18 from oslabs-beta/EC/updateReadme
a860188 
update readme
@DavidMNorman @davidtoyoukim
@exlxxn @JoanManto
Fix readme for setting up router
d369193 
Co-authored-by: David Norman <120237773+DavidMNorman@users.noreply.github.com>
Co-authored-by: David Kim <david.toyou.kim@gmail.com>
Co-authored-by: Eileen Cho <109772371+exlxxn@users.noreply.github.com>
Co-authored-by: Joan Manto <120276510+JoanManto@users.noreply.github.com>
@DavidMNorman
Merge pull request #19 from oslabs-beta/readmeFix
6fd4bc1 
Fix readme for setting up router
@DavidMNorman
Final readme adjustment
cb40c2c
@DavidMNorman
Merge pull request #20 from oslabs-beta/readmeFix
5e08492 
Final readme adjustment
@DavidMNorman
Merge pull request #21 from oslabs-beta/dev
4d8c9ed 
Merge to main
JoanManto
JoanManto approved these changes Apr 12, 2023
View reviewed changes
Copy link
Contributor

@JoanManto JoanManto left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Lgtm, great job!

@DavidMNorman DavidMNorman merged commit d5d0d4d into open-source-labs:master Apr 12, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@JoanManto JoanManto JoanManto approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@DavidMNorman @JoanManto @exlxxn