Skip to content

Commit

Permalink
fix: increase default cuckoo filter fingerprint size (#2636)
Browse files Browse the repository at this point in the history
Calculate the default fingerprint size from the `f ≥ log2(1/ε) + log2(2b)`
equation from the cucko filter paper.

Also lowers the reliable threshold to 90% from 95% as sometimes we
go over the threshold for smaller filter sizes.
  • Loading branch information
achingbrain authored Jul 26, 2024
1 parent aa5528f commit 34cf1f7
Showing 1 changed file with 2 additions and 2 deletions.
4 changes: 2 additions & 2 deletions packages/utils/src/filters/cuckoo-filter.ts
Original file line number Diff line number Diff line change
Expand Up @@ -150,7 +150,7 @@ export class CuckooFilter implements Filter {
}

get reliable (): boolean {
return Math.floor(100 * (this.count / this.filterSize)) <= 95
return Math.floor(100 * (this.count / this.filterSize)) <= 90
}
}

Expand Down Expand Up @@ -182,7 +182,7 @@ export function optimize (maxItems: number, errorRate: number = 0.001): CuckooFi

// https://stackoverflow.com/questions/57555236/how-to-size-a-cuckoo-filter/57617208#57617208
const filterSize = Math.round(maxItems / load)
const fingerprintSize = Math.min(Math.ceil(Math.log(filterSize / bucketSize)) + 2, MAX_FINGERPRINT_SIZE)
const fingerprintSize = Math.min(Math.ceil(Math.log2(1 / errorRate) + Math.log2(2 * bucketSize)), MAX_FINGERPRINT_SIZE)

return {
filterSize,
Expand Down

0 comments on commit 34cf1f7

Please sign in to comment.