performance: vaccum filter consistently significantly slower than simple Set.has

[Clouder0]

Issue Report: API Key Vacuum Filter Significantly Hurts Warm-Path Lookup Throughput

Summary

The API key Vacuum Filter (VF) path appears clever, but in warm/ready lookup microbenchmarks it consistently reduces throughput versus a plain Set.has membership check.

Across Bun, Node.js, and Deno runs with repo-like settings:

• Set is faster in 44/45 scenarios.
• Overall geometric-mean speedup of Set vs VF is 5.79x.
• Median speedup is 4.98x.
• Only one scenario is near parity/slight VF win (Node, 100K keys, validRatio=0.00, 0.99x).

For the warm lookup microbench specifically, this is strong evidence that VF introduces substantial overhead rather than performance benefit.

Scope and Intent

This report targets the exact question: warm, ready lookup performance.

Out of scope (by design):

• Redis/DB fallback latency
• startup/reload windows
• key creation/deletion synchronization behavior
• end-to-end request latency across all guards

Benchmark Implementation

Script: test.ts

The script was refined to mimic repo VF build settings instead of simplified params.

Real-case settings used (matching repo)

From src/lib/security/api-key-vacuum-filter.ts behavior:

• fingerprintBits = 32
• maxKickSteps = 500
• targetLoadFactor = 0.96
• desiredLoadFactor = 0.9
• minMaxItems = 128
• build retry up to 6 attempts with growth factor 1.6
• key format matches repo generation: sk-${randomBytes(16).toString("hex")}

Fixed preset used in all runs

• Key counts: 10, 100, 1,000, 10,000, 100,000
• Valid ratios: 1.0, 0.0, 0.01
• Warmup rounds: 1
• Measured rounds: 3
• Queries per scenario: 1,000,000

Reproduction Commands

Bun

bun test.ts

Node.js

bun build test.ts --target=node --outfile test.node.mjs
node test.node.mjs

Deno

Because the Bun bundle imports "crypto" (without node:), run Deno with an import map:

{
  "imports": {
    "crypto": "node:crypto"
  }
}

deno run -A --import-map deno.import-map.json test.node.mjs

Test Environment

• Timestamp: 2026-02-10T11:55:28+08:00
• Repo revision: 8738061b (branch dev)
• OS: Arch Linux (/etc/os-release)
• Kernel: Linux ArchStation 6.18.6-arch1-1 x86_64
• CPU: AMD Ryzen 9 9950X 16-Core Processor (32 vCPU)
• Hypervisor: Microsoft (virtualized environment)
• Memory: 45 GiB total, 22 GiB swap
• Runtime versions:
  • Node.js v24.5.0
  • Bun 1.3.6
  • Deno 2.6.5

Results

All QPS values are median QPS from the benchmark output (M = million ops/s).

Bun results

Keys	valid=1.00 (Set / VF / x)	valid=0.00 (Set / VF / x)	valid=0.01 (Set / VF / x)
10	263.55M / 16.88M / 15.61x	64.14M / 6.32M / 10.15x	113.17M / 6.85M / 16.51x
100	188.17M / 13.90M / 13.54x	90.43M / 6.12M / 14.78x	165.06M / 6.58M / 25.10x
1,000	184.61M / 10.49M / 17.60x	90.58M / 5.46M / 16.58x	89.27M / 5.84M / 15.28x
10,000	99.45M / 7.08M / 14.05x	58.64M / 5.37M / 10.91x	103.60M / 5.79M / 17.88x
100,000	94.71M / 6.38M / 14.84x	27.93M / 3.75M / 7.45x	29.27M / 4.02M / 7.29x

Summary:

• Set faster in 15/15 scenarios
• Speedup range: 7.29x to 25.10x
• Geometric mean speedup: 13.83x

Node.js results

Keys	valid=1.00 (Set / VF / x)	valid=0.00 (Set / VF / x)	valid=0.01 (Set / VF / x)
10	97.20M / 14.27M / 6.81x	32.92M / 12.79M / 2.57x	48.38M / 12.80M / 3.78x
100	73.48M / 13.19M / 5.57x	29.97M / 12.72M / 2.36x	45.19M / 13.22M / 3.42x
1,000	60.27M / 12.11M / 4.98x	24.37M / 12.77M / 1.91x	37.68M / 12.76M / 2.95x
10,000	64.60M / 11.84M / 5.45x	20.84M / 12.24M / 1.70x	35.49M / 12.46M / 2.85x
100,000	36.70M / 10.73M / 3.42x	7.69M / 7.75M / 0.99x	14.45M / 9.00M / 1.61x

Summary:

• Set faster in 14/15 scenarios
• One near-parity case (0.99x) at 100,000 keys, validRatio=0.00
• Geometric mean speedup: 2.96x

Deno results

Keys	valid=1.00 (Set / VF / x)	valid=0.00 (Set / VF / x)	valid=0.01 (Set / VF / x)
10	168.36M / 16.20M / 10.39x	72.03M / 14.87M / 4.84x	72.41M / 14.86M / 4.87x
100	87.14M / 15.25M / 5.71x	66.98M / 14.91M / 4.49x	66.31M / 14.91M / 4.45x
1,000	78.65M / 13.09M / 6.01x	65.13M / 14.19M / 4.59x	64.15M / 14.13M / 4.54x
10,000	77.42M / 12.91M / 5.99x	62.21M / 14.22M / 4.38x	61.85M / 14.20M / 4.35x
100,000	46.77M / 12.53M / 3.73x	33.92M / 10.72M / 3.16x	33.20M / 11.19M / 2.97x

Summary:

• Set faster in 15/15 scenarios
• Geometric mean speedup: 4.75x

Combined summary

• Set faster in 44/45 scenarios
• Combined geometric mean speedup: 5.79x
• Combined median speedup: 4.98x
• VF false positives observed in this run: 0 (all scenarios)

Memory Notes

Retained-memory signals are less stable than QPS in managed runtimes due to GC and allocator behavior, but measured values at 100,000 keys were:

• Bun: Set rss=1.75MiB, VF rss=4.00MiB
• Node.js: Set rss=4.75MiB, heap=5.03MiB, VF rss=7.75MiB, heap=18.78MiB
• Deno: Set rss=4.86MiB, heap=5.00MiB, VF rss=1.64MiB, heap=0.00MiB

Interpretation: memory results are runtime-dependent/noisy here, but throughput regression is large and consistent.

Why This Is a Problem

For the warm lookup path, the current VF layer adds substantial per-query CPU overhead compared with a direct hash lookup.

• Under attack-like mostly-invalid traffic (validRatio=0.01), Set remains clearly faster across all runtimes in this run.
• The performance gap is not marginal in Bun/Deno; it is often multi-x to double-digit-x.

This indicates the VF approach is not just neutral overhead; it is a significant throughput tax in the lookup microbench.

Conclusion

The current repo-like VF configuration significantly hurts warm-path lookup performance versus a plain Set in nearly all measured scenarios.

If the objective is lookup throughput (especially for hot auth checks), current evidence supports replacing VF with exact hash membership (or at minimum making VF disabled-by-default and gated by measured production evidence).

Appendix

Testing Script:

#!/usr/bin/env bun

import { randomBytes } from "node:crypto";
import { VacuumFilter } from "./src/lib/vacuum-filter/vacuum-filter";

const KEY_SIZES = [10, 100, 1_000, 10_000, 100_000] as const;
const VALID_RATIOS = [1.0, 0.0, 0.01] as const;
const OPERATIONS_PER_SCENARIO = 1_000_000;
const WARMUP_ROUNDS = 1;
const ROUNDS = 3;
const INVALID_POOL_MULTIPLIER = 4;
const MIN_INVALID_POOL = 100_000;

// Match src/lib/security/api-key-vacuum-filter.ts build settings
const VF_FINGERPRINT_BITS = 32;
const VF_MAX_KICK_STEPS = 500;
const VF_TARGET_LOAD_FACTOR = 0.96;
const VF_DESIRED_LOAD_FACTOR = 0.9;
const VF_MIN_MAX_ITEMS = 128;
const VF_MAX_BUILD_ATTEMPTS = 6;
const VF_BUILD_GROWTH = 1.6;

type DurationSummary = {
  medianMs: number;
  p95Ms: number;
  medianQps: number;
  p95Qps: number;
  observedHits: number;
};

type ScenarioResult = {
  validRatio: number;
  expectedHits: number;
  set: DurationSummary;
  vacuumFilter: DurationSummary;
  falsePositives: number;
};

type MemorySnapshot = {
  rss: number;
  heapUsed: number;
  arrayBuffers: number;
};

type MemoryDelta = {
  rssMiB: number;
  heapUsedMiB: number;
  arrayBuffersMiB: number;
};

type SizeResult = {
  size: number;
  invalidPoolSize: number;
  setBuildMs: number;
  vacuumFilterBuildMs: number;
  vacuumFilterLoadFactor: number;
  memoryRetained: {
    set: MemoryDelta;
    vacuumFilter: MemoryDelta;
  };
  scenarios: ScenarioResult[];
};

// Matches key format in src/actions/keys.ts and src/actions/users.ts
function generateApiKey(): string {
  return `sk-${randomBytes(16).toString("hex")}`;
}

function generateInvalidKey(): string {
  return `xx-${randomBytes(16).toString("hex")}`;
}

function nowMs(): number {
  return Number(process.hrtime.bigint()) / 1e6;
}

function maybeGc(): void {
  const maybeBun = globalThis as unknown as {
    Bun?: { gc?: (force?: boolean) => void };
  };
  if (typeof maybeBun.Bun?.gc === "function") {
    maybeBun.Bun.gc(true);
  }
}

function toMiB(bytes: number): number {
  return Number((bytes / 1024 / 1024).toFixed(2));
}

function snapshotMemory(): MemorySnapshot {
  maybeGc();
  const m = process.memoryUsage();
  return {
    rss: m.rss,
    heapUsed: m.heapUsed,
    arrayBuffers: m.arrayBuffers,
  };
}

function memoryDiff(before: MemorySnapshot, after: MemorySnapshot): MemoryDelta {
  const rssDelta = Math.max(0, after.rss - before.rss);
  const heapDelta = Math.max(0, after.heapUsed - before.heapUsed);
  const arrayBufferDelta = Math.max(0, after.arrayBuffers - before.arrayBuffers);

  return {
    rssMiB: toMiB(rssDelta),
    heapUsedMiB: toMiB(heapDelta),
    arrayBuffersMiB: toMiB(arrayBufferDelta),
  };
}

function buildUniqueKeys(count: number, generator: () => string): string[] {
  const out = new Set<string>();
  while (out.size < count) {
    out.add(generator());
  }
  return Array.from(out);
}

function computeInitialMaxItems(uniqueCount: number): number {
  return Math.max(
    VF_MIN_MAX_ITEMS,
    Math.ceil((uniqueCount * VF_TARGET_LOAD_FACTOR) / VF_DESIRED_LOAD_FACTOR)
  );
}

function buildVacuumFilterLikeRepository(keys: string[]): VacuumFilter {
  const unique = Array.from(new Set(keys)).filter((key) => key.length > 0);
  let maxItems = computeInitialMaxItems(unique.length);
  let lastError: Error | null = null;

  for (let attempt = 1; attempt <= VF_MAX_BUILD_ATTEMPTS; attempt++) {
    const vacuumFilter = new VacuumFilter({
      maxItems,
      fingerprintBits: VF_FINGERPRINT_BITS,
      maxKickSteps: VF_MAX_KICK_STEPS,
      targetLoadFactor: VF_TARGET_LOAD_FACTOR,
    });

    let okAll = true;
    for (const key of unique) {
      if (!vacuumFilter.add(key)) {
        okAll = false;
        break;
      }
    }

    if (okAll) {
      return vacuumFilter;
    }

    lastError = new Error(`build failed at attempt=${attempt}, maxItems=${maxItems}`);
    maxItems = Math.ceil(maxItems * VF_BUILD_GROWTH);
  }

  throw lastError ?? new Error("Vacuum filter build failed");
}

function percentile(values: number[], p: number): number {
  const sorted = [...values].sort((a, b) => a - b);
  const index = Math.floor((sorted.length - 1) * p);
  return sorted[index];
}

function summarizeDurations(
  durationsMs: number[],
  operations: number,
  observedHits: number
): DurationSummary {
  const medianMs = percentile(durationsMs, 0.5);
  const p95Ms = percentile(durationsMs, 0.95);
  return {
    medianMs: Number(medianMs.toFixed(3)),
    p95Ms: Number(p95Ms.toFixed(3)),
    medianQps: Math.round(operations / (medianMs / 1000)),
    p95Qps: Math.round(operations / (p95Ms / 1000)),
    observedHits,
  };
}

function buildQueryBatch(options: {
  validRatio: number;
  operations: number;
  validPool: string[];
  invalidPool: string[];
}): { queries: string[]; expectedHits: number } {
  const { validRatio, operations, validPool, invalidPool } = options;
  const queries = new Array<string>(operations);

  let expectedHits = 0;
  let state = 0x12345678;
  const nextU32 = () => {
    state = (Math.imul(state, 1664525) + 1013904223) >>> 0;
    return state;
  };

  const threshold = Math.floor(validRatio * 1_000_000);

  for (let i = 0; i < operations; i++) {
    const useValid =
      validRatio === 1 ? true : validRatio === 0 ? false : nextU32() % 1_000_000 < threshold;

    if (useValid) {
      expectedHits++;
      queries[i] = validPool[nextU32() % validPool.length];
    } else {
      queries[i] = invalidPool[nextU32() % invalidPool.length];
    }
  }

  return { queries, expectedHits };
}

function runLookupBenchmark(options: {
  queries: string[];
  expectedHits: number;
  lookup: (key: string) => boolean;
}): DurationSummary {
  const { queries, expectedHits, lookup } = options;

  for (let i = 0; i < WARMUP_ROUNDS; i++) {
    let warmupHits = 0;
    for (let j = 0; j < queries.length; j++) {
      if (lookup(queries[j])) warmupHits++;
    }
    if (warmupHits < 0) {
      throw new Error("Unreachable warmup guard");
    }
  }

  const durationsMs: number[] = [];
  let observedHits = -1;

  for (let round = 0; round < ROUNDS; round++) {
    let hits = 0;
    const start = nowMs();
    for (let i = 0; i < queries.length; i++) {
      if (lookup(queries[i])) hits++;
    }
    const end = nowMs();
    durationsMs.push(end - start);

    if (observedHits === -1) {
      observedHits = hits;
    } else if (hits !== observedHits) {
      throw new Error(`Observed hits changed across rounds: ${observedHits} -> ${hits}`);
    }
  }

  if (observedHits < expectedHits) {
    throw new Error(
      `Observed hits (${observedHits}) is smaller than expected hits (${expectedHits})`
    );
  }

  return summarizeDurations(durationsMs, queries.length, observedHits);
}

function measureSetRetainedMemory(size: number): MemoryDelta {
  let keys = buildUniqueKeys(size, generateApiKey);
  const before = snapshotMemory();

  const set = new Set(keys);
  keys = [];

  set.has("sk-00000000000000000000000000000000");
  const after = snapshotMemory();
  return memoryDiff(before, after);
}

function measureVacuumFilterRetainedMemory(size: number): MemoryDelta {
  let keys = buildUniqueKeys(size, generateApiKey);
  const before = snapshotMemory();

  const vacuumFilter = buildVacuumFilterLikeRepository(keys);

  keys = [];

  vacuumFilter.has("xx-00000000000000000000000000000000");
  const after = snapshotMemory();
  return memoryDiff(before, after);
}

function benchmarkSize(size: number): SizeResult {
  const validKeys = buildUniqueKeys(size, generateApiKey);
  const invalidPoolSize = Math.max(size * INVALID_POOL_MULTIPLIER, MIN_INVALID_POOL);
  const invalidKeys = buildUniqueKeys(invalidPoolSize, generateInvalidKey);

  const setBuildStart = nowMs();
  const set = new Set(validKeys);
  const setBuildMs = Number((nowMs() - setBuildStart).toFixed(3));

  const vacuumFilterBuildStart = nowMs();
  const vacuumFilter = buildVacuumFilterLikeRepository(validKeys);

  const vacuumFilterBuildMs = Number((nowMs() - vacuumFilterBuildStart).toFixed(3));

  const scenarios = VALID_RATIOS.map((validRatio) => {
    const { queries, expectedHits } = buildQueryBatch({
      validRatio,
      operations: OPERATIONS_PER_SCENARIO,
      validPool: validKeys,
      invalidPool: invalidKeys,
    });

    const setResult = runLookupBenchmark({
      queries,
      expectedHits,
      lookup: (key) => set.has(key),
    });

    const vacuumFilterResult = runLookupBenchmark({
      queries,
      expectedHits,
      lookup: (key) => vacuumFilter.has(key),
    });

    return {
      validRatio,
      expectedHits,
      set: setResult,
      vacuumFilter: vacuumFilterResult,
      falsePositives: vacuumFilterResult.observedHits - expectedHits,
    } satisfies ScenarioResult;
  });

  return {
    size,
    invalidPoolSize,
    setBuildMs,
    vacuumFilterBuildMs,
    vacuumFilterLoadFactor: Number(vacuumFilter.loadFactor().toFixed(4)),
    memoryRetained: {
      set: measureSetRetainedMemory(size),
      vacuumFilter: measureVacuumFilterRetainedMemory(size),
    },
    scenarios,
  };
}

function formatQps(qps: number): string {
  if (qps >= 1_000_000) return `${(qps / 1_000_000).toFixed(2)}M`;
  if (qps >= 1_000) return `${(qps / 1_000).toFixed(2)}K`;
  return String(qps);
}

function printReport(results: SizeResult[]): void {
  console.log("Set vs VacuumFilter query benchmark");
  console.log(
    `preset: sizes=[${KEY_SIZES.join(", ")}], valid-ratios=[${VALID_RATIOS.join(", ")}], warmup=${WARMUP_ROUNDS}, rounds=${ROUNDS}, ops=${OPERATIONS_PER_SCENARIO.toLocaleString()}`
  );
  console.log(
    `vf-params(real-case): fingerprintBits=${VF_FINGERPRINT_BITS}, maxKickSteps=${VF_MAX_KICK_STEPS}, targetLoadFactor=${VF_TARGET_LOAD_FACTOR}, desiredLoadFactor=${VF_DESIRED_LOAD_FACTOR}, minMaxItems=${VF_MIN_MAX_ITEMS}, buildRetry=${VF_MAX_BUILD_ATTEMPTS}x growth=${VF_BUILD_GROWTH}`
  );

  for (const result of results) {
    console.log(`\n--- key-count=${result.size.toLocaleString()} ---`);
    console.log(
      `build-ms: set=${result.setBuildMs.toFixed(3)} vf=${result.vacuumFilterBuildMs.toFixed(3)} vf-load=${result.vacuumFilterLoadFactor}`
    );
    console.log(
      `retained-memory-mib: set(rss=${result.memoryRetained.set.rssMiB}, heap=${result.memoryRetained.set.heapUsedMiB}, ab=${result.memoryRetained.set.arrayBuffersMiB}) vf(rss=${result.memoryRetained.vacuumFilter.rssMiB}, heap=${result.memoryRetained.vacuumFilter.heapUsedMiB}, ab=${result.memoryRetained.vacuumFilter.arrayBuffersMiB})`
    );
    console.log(
      "valid-ratio  set-qps(med/p95)      vf-qps(med/p95)       set/vf   vf-false-positive"
    );

    for (const scenario of result.scenarios) {
      const speedup = scenario.set.medianQps / scenario.vacuumFilter.medianQps;
      console.log(
        `${scenario.validRatio.toFixed(2).padEnd(11)}${`${formatQps(scenario.set.medianQps)}/${formatQps(scenario.set.p95Qps)}`.padEnd(24)}${`${formatQps(scenario.vacuumFilter.medianQps)}/${formatQps(scenario.vacuumFilter.p95Qps)}`.padEnd(24)}${`${speedup.toFixed(2)}x`.padEnd(9)}${scenario.falsePositives}`
      );
    }
  }
}

function main(): void {
  const results = KEY_SIZES.map((size) => benchmarkSize(size));
  printReport(results);
}

main();

[coderabbitai]

🔗 Related PRs

#532 - fix: 修复 my-usage 今日统计与只读 API 自助查询 [merged]
#574 - feat: 统一请求特殊设置命中的展示方式 [merged]
#601 - fix: Codex session completion must not inject metadata [merged]
#670 - feat(providers): recluster vendors by host:port when website_url empty [merged]
#734 - feat(security): 加固并优化 API Key 与登录鉴权链路（Vacuum Filter → Redis → DB） [merged]

---

📝 Issue Planner

_{Check the box below or use the @coderabbitai plan command to generate an implementation plan and prompts that you can use with your favorite coding assistant.}

• Create Plan

---

🧪 Issue enrichment is currently in open beta.

You can configure auto-planning by selecting labels in the issue_enrichment configuration.

To disable automatic issue enrichment, add the following to your .coderabbit.yaml:

issue_enrichment:
  auto_enrich:
    enabled: false

💬 Have feedback or questions? Drop into our discord!