Run-time structured generation benchmarks

[lapp0]

Initialization benchmarks are introduced in #542

We should extend these benchmarks to measure the performance of inference.

Goal

Outlines shouldn't be a bottleneck for most inference. A reasonable goal can be set based on

• 1,200 output tok/s on H100 w/ Llama 2 70B on Optimum-Nvidia

Benchmarks will help us achieve and maintain that goal.

What must be benchmarked

• End-to-end CFG and Regex guided generation
• Guided generation with Ray IPC logits processing with vLLM (related VLLM tensor-parallel and RegexLogitsProcessor #524)

Proposed method

It's annoying to need a GPU to run tests. We shouldn't do actual inference in performance benchmarks.

• 1. Create a mock inference engine
• 2. Simple benchmark to ensure unguided mock inference engine takes infinitesimally small time
• 3. Guided benchmarks that show true throughput of outlines

[rlouf]

At this point I think that it would only make sense to benchmark the CFG-guided generation. Regex-guided generation is only a dictionary call at each step, so there really isn't anything we could do that would move the needle.