Frontend: Adding LM Format Enforcer support to V1 engine

[noamgat]

Essential Elements of an Effective PR Description Checklist

• The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
• The test plan, such as providing test command.
• The test results, such as pasting the results comparison before and after, or e2e results
• (Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.

Purpose

To reduce parity between V0 and V1 and introduce LM Format Enforcer as a structured decoding backend in V1 as well

Test Plan

Added LM Format Enforcer configurations to the structured decoding permutations

Test Result

Tests pass

(Optional) Documentation Update

(Note - this is the same PR as the one I closed, reopened to fix DCO issues)

[github-actions]

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can either: Add ready label to the PR or enable auto-merge.

🚀

[gemini-code-assist]

Code Review

This pull request introduces support for the lm-format-enforcer library as a structured decoding backend for the V1 engine, enhancing parity with the V0 engine. The changes include updating the lm-format-enforcer dependency to a specific version, integrating it into the engine's configuration and processing logic, and adding it to the test suite. The tests have been thoughtfully adapted to accommodate the specific characteristics of this new backend, such as its handling of JSON formatting and its feature set regarding grammar support. The implementation appears solid and follows the established patterns for backend integration within the codebase. I have no major concerns with the proposed changes.

[noamgat]

The CI fails, but it doesn't look related to my code:
https://buildkite.com/vllm/fastcheck/builds/34659/steps/canvas?sid=01988df0-1180-4ddc-9c5f-3966568cd413

Is this a flakiness issue?