gh-128571: Document UTF-16/32 native byte order #139974

Prhmma · 2025-10-11T23:28:33Z

Summary

add a note in Doc/library/codecs.rst pointing out that Python’s UTF-16/UTF-32 codecs default to the platform’s native endianness when no BOM is present
highlight that this differs from the Unicode specification’s big-endian default and explain why Python does it for compatibility

Issue

Indicate discrepencies with Unicode specifications for UTF-16/32 schemes #128571

📚 Documentation preview 📚: https://cpython-previews--139974.org.readthedocs.build/

python-cla-bot · 2025-10-11T23:28:36Z

All commit authors signed the Contributor License Agreement.

StanFromIreland · 2025-10-12T07:16:35Z

Doc/library/codecs.rst

+.. note::
+
+   **Python UTF-16 and UTF-32 Codec Behavior**
+
+   Python's ``UTF-16`` and ``UTF-32`` codecs (when used without an explicit
+   byte order suffix like ``-BE`` or ``-LE``) follow the platform's native
+   byte order when no BOM is present. This differs from the Unicode Standard
+   specification, which states that UTF-16 and UTF-32 encoding schemes should
+   default to big-endian byte order when no BOM is present and no higher-level
+   protocol specifies the byte order.
+
+   This behavior was chosen for practical compatibility reasons, as it avoids
+   byte swapping on the most common platforms, but developers should be aware
+   of this difference when exchanging data with systems that strictly follow
+   the Unicode specification.


This is currently explained much more tersely on lines 993/994. I would suggest improving and slightly expanding the wording there, rather than duplicating content with this long note.

Thanks @StanFromIreland
I pushed another commit. Please let me know if it is descriptive enough

…ssion in the issue

youkidearitai · 2025-10-15T02:49:09Z

Thank you, Looks good to me.

malemburg

LGTM.

Doc/library/codecs.rst

StanFromIreland

Some little formalisation suggestions.

Doc/library/codecs.rst

Co-authored-by: Stan Ulbrych <89152624+StanFromIreland@users.noreply.github.com>

StanFromIreland · 2025-10-18T11:16:17Z

Please do not use the Update Branch button unless necessary (e.g. fixing conflicts, jogging the CI, or very old PRs) as it uses valuable resources. For more information see the devguide.

StanFromIreland

LGTM (I left a minor suggestion to deduplicate two sentences), Marc, please merge this when you have a chance.

Doc/library/codecs.rst

Co-authored-by: Stan Ulbrych <89152624+StanFromIreland@users.noreply.github.com>

malemburg · 2025-10-18T18:48:10Z

Should we try a backport of this ?

StanFromIreland · 2025-10-18T18:51:40Z

Sounds good to me, I think this paragraph is the same on all branches (and it's not like the specification has changed in the meantime :-), if there are conflicts I can fix.

miss-islington-app · 2025-10-18T18:51:57Z

Thanks @Prhmma for the PR, and @malemburg for merging it 🌮🎉.. I'm working now to backport this PR to: 3.13.
🐍🍒⛏🤖 I'm not a witch! I'm not a witch!

miss-islington-app · 2025-10-18T18:51:57Z

Thanks @Prhmma for the PR, and @malemburg for merging it 🌮🎉.. I'm working now to backport this PR to: 3.14.
🐍🍒⛏🤖

Closes pythonGH-128571 (cherry picked from commit 920de7c) Co-authored-by: Parham MohammadAlizadeh <prhmma@gmail.com> Co-authored-by: Stan Ulbrych <89152624+StanFromIreland@users.noreply.github.com>

bedevere-app · 2025-10-18T18:52:08Z

GH-140308 is a backport of this pull request to the 3.13 branch.

bedevere-app · 2025-10-18T18:52:12Z

GH-140309 is a backport of this pull request to the 3.14 branch.

…140309) Closes GH-128571 (cherry picked from commit 920de7c) Co-authored-by: Parham MohammadAlizadeh <prhmma@gmail.com> Co-authored-by: Stan Ulbrych <89152624+StanFromIreland@users.noreply.github.com>

…140308) Closes GH-128571 (cherry picked from commit 920de7c) Co-authored-by: Parham MohammadAlizadeh <prhmma@gmail.com> Co-authored-by: Stan Ulbrych <89152624+StanFromIreland@users.noreply.github.com>

malemburg · 2025-10-18T19:03:53Z

Thanks, both.

pythongh-128571: Document UTF-16/32 native byte order

96f31ef

bedevere-app bot added docs Documentation in the Doc dir skip news labels Oct 11, 2025

github-project-automation bot added this to Docs PRs Oct 11, 2025

bedevere-app bot mentioned this pull request Oct 11, 2025

Indicate discrepencies with Unicode specifications for UTF-16/32 schemes #128571

Closed

bedevere-app bot added the awaiting review label Oct 11, 2025

github-project-automation bot moved this to Todo in Docs PRs Oct 11, 2025

StanFromIreland requested a review from malemburg October 12, 2025 06:56

StanFromIreland reviewed Oct 12, 2025

View reviewed changes

Removed the note and improved existing description based on the discu…

9a5ee89

…ssion in the issue

Prhmma force-pushed the gh-128571-codecs-doc branch from 974f000 to 9a5ee89 Compare October 12, 2025 10:23

Merge branch 'main' into pythongh-128571-codecs-doc

ce69567

malemburg approved these changes Oct 16, 2025

View reviewed changes

Doc/library/codecs.rst Outdated Show resolved Hide resolved

bedevere-app bot added awaiting merge and removed awaiting review labels Oct 16, 2025

StanFromIreland reviewed Oct 16, 2025

View reviewed changes

Prhmma and others added 8 commits October 17, 2025 11:09

Update Doc/library/codecs.rst

171869f

Co-authored-by: Stan Ulbrych <89152624+StanFromIreland@users.noreply.github.com>

Update Doc/library/codecs.rst

bc661bc

Co-authored-by: Stan Ulbrych <89152624+StanFromIreland@users.noreply.github.com>

Update Doc/library/codecs.rst

9c93da5

Co-authored-by: Stan Ulbrych <89152624+StanFromIreland@users.noreply.github.com>

Update Doc/library/codecs.rst

9204430

Co-authored-by: Stan Ulbrych <89152624+StanFromIreland@users.noreply.github.com>

Update Doc/library/codecs.rst

b0fa2ba

Co-authored-by: Stan Ulbrych <89152624+StanFromIreland@users.noreply.github.com>

changing e.g. to for example

ff82ed5

Merge branch 'main' into pythongh-128571-codecs-doc

f5923e3

Merge branch 'main' into pythongh-128571-codecs-doc

8cda849

StanFromIreland self-requested a review October 18, 2025 11:16

StanFromIreland approved these changes Oct 18, 2025

View reviewed changes

Doc/library/codecs.rst Outdated Show resolved Hide resolved

Update Doc/library/codecs.rst - deduplication

8fda436

Co-authored-by: Stan Ulbrych <89152624+StanFromIreland@users.noreply.github.com>

malemburg merged commit 920de7c into python:main Oct 18, 2025
27 checks passed

github-project-automation bot moved this from Todo to Done in Docs PRs Oct 18, 2025

bedevere-app bot removed the awaiting merge label Oct 18, 2025

StanFromIreland added needs backport to 3.13 bugs and security fixes needs backport to 3.14 bugs and security fixes labels Oct 18, 2025

bedevere-app bot removed the needs backport to 3.13 bugs and security fixes label Oct 18, 2025

bedevere-app bot removed the needs backport to 3.14 bugs and security fixes label Oct 18, 2025

Prhmma mentioned this pull request Oct 19, 2025

Convert all "e.g." to "for example" in the documentation #140330

Closed

Uh oh!

gh-128571: Document UTF-16/32 native byte order #139974

gh-128571: Document UTF-16/32 native byte order #139974

Uh oh!

Conversation

Prhmma commented Oct 11, 2025 • edited by github-actions bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Issue

Uh oh!

python-cla-bot bot commented Oct 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

StanFromIreland Oct 12, 2025

Choose a reason for hiding this comment

Uh oh!

Prhmma Oct 12, 2025

Choose a reason for hiding this comment

Uh oh!

youkidearitai commented Oct 15, 2025

Uh oh!

malemburg left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

StanFromIreland left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

StanFromIreland commented Oct 18, 2025

Uh oh!

StanFromIreland left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

malemburg commented Oct 18, 2025

Uh oh!

StanFromIreland commented Oct 18, 2025

Uh oh!

miss-islington-app bot commented Oct 18, 2025

Uh oh!

miss-islington-app bot commented Oct 18, 2025

Uh oh!

bedevere-app bot commented Oct 18, 2025

Uh oh!

bedevere-app bot commented Oct 18, 2025

Uh oh!

malemburg commented Oct 18, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Prhmma commented Oct 11, 2025 •

edited by github-actions bot

Loading

python-cla-bot bot commented Oct 11, 2025 •

edited

Loading