[Doc] Fix UTF-8 encoding issues in documentation generation on Windows #24361

alhridoy · 2025-09-06T06:11:10Z

Summary

Fixes #24360: Cannot build documentation locally without adding encoding to generate examples

This PR resolves Windows-specific encoding issues when building documentation locally by explicitly specifying UTF-8 encoding for file operations in documentation generation scripts.

Problem

On Windows, Python's default file encoding is often cp1252 (Windows-1252), which cannot handle Unicode characters like \uff5c (fullwidth vertical bar ｜). This caused:

UnicodeEncodeError when processing files with Unicode content
Documentation build failures on Windows systems
Inability for Windows contributors to build docs locally

Error Examples

UnicodeEncodeError: 'charmap' codec can't encode character '\uff5c' in position 61: character maps to <undefined>

Solution

Core Changes

docs/mkdocs/hooks/generate_examples.py: Added encoding='utf-8' to file operations
docs/mkdocs/hooks/generate_argparse.py: Added encoding='utf-8' to file operations
Comprehensive testing: Added Unicode handling tests
Documentation: Added Windows development guide

Files Modified

# Before (problematic)
with open(self.main_file) as f:
    first_line = f.readline().strip()

with open(doc_path, "w+") as f:
    f.write(example.generate())

# After (fixed)
with open(self.main_file, encoding="utf-8") as f:
    first_line = f.readline().strip()

with open(doc_path, "w+", encoding="utf-8") as f:
    f.write(example.generate())

gemini-code-assist

Code Review

This pull request correctly addresses UTF-8 encoding issues for documentation generation on Windows. The changes are well-implemented and include new documentation and tests.

However, this PR critically mixes unrelated concerns by bundling the documentation fix with a significant, unrelated feature: adding sliding window attention support to FlexAttention. This includes changes to vllm/v1/attention/backends/flex_attention.py and new test files.

Mixed-concern pull requests complicate reviews and harm the project's long-term maintainability by polluting the git history. Please split this into two separate PRs: one for the documentation fix and one for the FlexAttention feature. I cannot approve this PR in its current state.

gemini-code-assist · 2025-09-06T06:12:54Z

vllm/v1/attention/backends/flex_attention.py

This function and other related changes for FlexAttention sliding window support are unrelated to this pull request's stated goal of fixing documentation encoding issues. Please move these changes, along with the corresponding tests, to a separate pull request. Mixing unrelated features in a single PR is a critical issue for repository maintainability.

gemini-code-assist · 2025-09-06T06:12:54Z

tests/models/test_gpt_oss_sliding_window.py

This new test file is for FlexAttention sliding window support, which is unrelated to the documentation encoding fix this PR is supposed to address. This file and its corresponding feature implementation should be moved to a separate pull request to maintain a clean and logical commit history.

gemini-code-assist · 2025-09-06T06:12:54Z

tests/v1/attention/backends/test_flex_attention_sliding_window.py

This new test file is for FlexAttention sliding window support, which is unrelated to the documentation encoding fix this PR is supposed to address. This file and its corresponding feature implementation should be moved to a separate pull request to maintain a clean and logical commit history.

hmellor

Thank you for the fix which makes the docs build more flexible!

However, I don't think we need the doc or the tests.

vLLM is not supported on Windows (only via WSL), so development instructions for Windows itself might give the wrong idea.

As for the tests, a comment next to the file opening lines saying # Specify encoding for building on Windows is sufficient.

alhridoy · 2025-09-07T19:37:49Z

@hmellor Thank you for the feedback! You're absolutely right - I've removed the Windows documentation
and comprehensive tests.

The PR now contains only:

UTF-8 encoding fixes in the two documentation generation files
Simple comments explaining the encoding is for Windows builds

hmellor

Thanks for updating!

hmellor · 2025-09-07T22:46:11Z

Please fix DCO then I'll be able to merge

Fixes vllm-project#24358: FlexAttention does not support sliding window yet This commit implements sliding window attention support for the FlexAttention backend, enabling models like GptOss to work with FlexAttention when FlashAttention v2 is unavailable (e.g., on GPUs with compute capability < 8.0). Key changes: - Add sliding_window_causal_mask_mod() function for sliding window masks - Update FlexAttentionImpl to handle sliding window configuration - Enhance FlexAttentionMetadata with sliding_window field - Update FlexAttentionMetadataBuilder to extract sliding window from config - Add comprehensive unit and integration tests The implementation maintains backward compatibility and has no performance impact on models without sliding window. Tested with: - Unit tests for mask functions and metadata handling - Integration tests with FlexAttentionImpl - Edge cases and error conditions - Backward compatibility verification Before this change: - Models with sliding window would fail with NotImplementedError - Users had to manually force different attention backends - GptOss models couldn't use FlexAttention on older GPUs After this change: - FlexAttention supports sliding window attention seamlessly - GptOss models work out of the box with FlexAttention - Maintains full backward compatibility for models without sliding window Signed-off-by: alekramelaheehridoy <aliqramalaheehridoy@gmail.com>

Fixes vllm-project#24360: Cannot build documentation locally without adding encoding This commit resolves Windows-specific encoding issues when building documentation locally by explicitly specifying UTF-8 encoding for file operations in documentation generation scripts. Changes: - Add encoding='utf-8' to file operations in generate_examples.py - Add encoding='utf-8' to file operations in generate_argparse.py - Add comprehensive tests for Unicode handling - Add Windows development guide with encoding troubleshooting Root Cause: Windows uses cp1252 (Windows-1252) as default encoding, which cannot handle Unicode characters like \uff5c (fullwidth vertical bar ｜). This caused UnicodeEncodeError when processing files with Unicode content. Solution: Explicitly specify UTF-8 encoding for all file read/write operations in documentation generation hooks, ensuring consistent behavior across all platforms. Testing: - Added unit tests for Unicode character handling - Tested with problematic characters from the original error - Verified backward compatibility with ASCII-only content - Added edge case testing for empty files and missing titles Benefits: - Enables local documentation building on Windows - Improves cross-platform development experience - Prevents similar encoding issues in the future - Maintains full backward compatibility Before this change: - Documentation build failed on Windows with UnicodeEncodeError - Contributors on Windows couldn't build docs locally - Unicode characters in examples caused build failures After this change: - Documentation builds successfully on Windows - Full Unicode support in documentation content - Consistent behavior across all platforms Signed-off-by: alekramelaheehridoy <alekramelaheehridoy@gmail.com> Signed-off-by: alekramelaheehridoy <aliqramalaheehridoy@gmail.com>

This commit removes FlexAttention sliding window support changes that were incorrectly included in the documentation encoding fix PR. These changes belong in a separate PR to maintain clean commit history and proper separation of concerns. Removed: - FlexAttention sliding window implementation - Related test files for sliding window functionality - Unrelated changes to flex_attention.py This PR should now focus solely on fixing UTF-8 encoding issues in documentation generation scripts as originally intended. Signed-off-by: alekramelaheehridoy <alekramelaheehridoy@gmail.com> Signed-off-by: alekramelaheehridoy <aliqramalaheehridoy@gmail.com>

Specify encoding explicitly in file operations to prevent UnicodeEncodeError when building docs on Windows systems that default to cp1252 encoding. Fixes vllm-project#24360 Signed-off-by: alekramelaheehridoy <alekramelaheehridoy@gmail.com> Signed-off-by: alekramelaheehridoy <aliqramalaheehridoy@gmail.com>

alhridoy · 2025-09-07T23:53:51Z

@hmellor DCO is now fixed! All commits have been properly signed off with the correct email address. Thanks again!

vllm-project#24361) Signed-off-by: alekramelaheehridoy <aliqramalaheehridoy@gmail.com> Signed-off-by: alekramelaheehridoy <alekramelaheehridoy@gmail.com> Co-authored-by: alekramelaheehridoy <alekramelaheehridoy@gmail.com>

vllm-project#24361) Signed-off-by: alekramelaheehridoy <aliqramalaheehridoy@gmail.com> Signed-off-by: alekramelaheehridoy <alekramelaheehridoy@gmail.com> Co-authored-by: alekramelaheehridoy <alekramelaheehridoy@gmail.com> Signed-off-by: xuebwang-amd <xuebwang@amd.com>

alhridoy requested review from DarkLight1337, WoosukKwon, alexm-redhat, comaniac, hmellor, njhill, robertgshaw2-redhat and ywang96 as code owners September 6, 2025 06:11

mergify bot added documentation Improvements or additions to documentation gpt-oss Related to GPT-OSS models v1 labels Sep 6, 2025

gemini-code-assist bot reviewed Sep 6, 2025

View reviewed changes

hmellor requested changes Sep 7, 2025

View reviewed changes

alhridoy force-pushed the fix-docs-encoding-windows branch 2 times, most recently from 0846e4d to 797ce24 Compare September 7, 2025 19:36

hmellor approved these changes Sep 7, 2025

View reviewed changes

alhridoy force-pushed the fix-docs-encoding-windows branch 2 times, most recently from 0e1529f to 2ff9efb Compare September 7, 2025 23:33

alekramelaheehridoy and others added 4 commits September 7, 2025 17:50

alhridoy force-pushed the fix-docs-encoding-windows branch from 2ff9efb to c558860 Compare September 7, 2025 23:51

vllm-bot merged commit 8c892b1 into vllm-project:main Sep 8, 2025
12 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

[Doc] Fix UTF-8 encoding issues in documentation generation on Windows #24361

[Doc] Fix UTF-8 encoding issues in documentation generation on Windows #24361

Uh oh!

alhridoy commented Sep 6, 2025 •

edited by github-actions bot

Loading

Uh oh!

gemini-code-assist bot left a comment

Uh oh!

gemini-code-assist bot Sep 6, 2025

Uh oh!

gemini-code-assist bot Sep 6, 2025

Uh oh!

gemini-code-assist bot Sep 6, 2025

Uh oh!

hmellor left a comment

Uh oh!

alhridoy commented Sep 7, 2025

Uh oh!

hmellor left a comment

Uh oh!

hmellor commented Sep 7, 2025

Uh oh!

alhridoy commented Sep 7, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

[Doc] Fix UTF-8 encoding issues in documentation generation on Windows #24361

[Doc] Fix UTF-8 encoding issues in documentation generation on Windows #24361

Uh oh!

Conversation

alhridoy commented Sep 6, 2025 • edited by github-actions bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Problem

Error Examples

Solution

Core Changes

Files Modified

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

gemini-code-assist bot Sep 6, 2025

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist bot Sep 6, 2025

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist bot Sep 6, 2025

Choose a reason for hiding this comment

Uh oh!

hmellor left a comment

Choose a reason for hiding this comment

Uh oh!

alhridoy commented Sep 7, 2025

Uh oh!

hmellor left a comment

Choose a reason for hiding this comment

Uh oh!

hmellor commented Sep 7, 2025

Uh oh!

alhridoy commented Sep 7, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

alhridoy commented Sep 6, 2025 •

edited by github-actions bot

Loading