Enhanced the cursor to be stateful #1091
Conversation
Fixes stackernews#147 To avoid duplicated results push some state towards the cursor. The cursor uses a Bloom Filter[1], a probabilistic data structure. In this data structure, False positive matches are possible, but false negatives are not – in other words, a query returns either "possibly in set" or "definitely not in set". Elements can be added to the set, but not removed. The more items added, the larger the probability of false positives. [1] https://en.wikipedia.org/wiki/Bloom_filter
|
New and removed dependencies detected. Learn more about Socket for GitHub ↗︎
🚮 Removed packages: npm/puppeteer@20.8.2 |
|
This is going to require using a bloomfilter in postgres queries. It doesn't look like you've gotten to that part in the PR yet so I wanted to be sure to flag that. For a nice to have thing, my "heads up" advice is that it's going to be easy to overengineer a solution here. The requirement this is going to need to meet is "is it worth a probabilistic data structure, another package and new code to solve this minor inconvenience" and "is this the simplest way to solve this problem or merely the best solution that can be tacked on without understanding the rest of the code"? |
That bit is still to be implemented, I guess my original idea was to filter things out, but still alter as little as possible from the existing logic. AFAIK, there is no bloom filter built-in for Postgres. My end goal was to avoid sending duplicate items, but still read them from the database. My same approach can be done in the front end, but just refusing to render duplicated items. Maybe, it could even update the existing item in place, instead of appending it. I'll give it a thought
I agree. There is very little benefit but it was a fun approach nonetheless. |
Description
Fixes #147
To avoid duplicated results push some state towards the cursor.
The cursor uses a Bloom Filter[1], a probabilistic data structure. In this data structure, False positive matches are possible, but false negatives are not – in other words, a query returns either "possibly in set" or "definitely not in set". Elements can be added to the set, but not removed. The more items added, the larger the probability of false positives.
[1] https://en.wikipedia.org/wiki/Bloom_filter