Document PyUnicode_* API

[avassalotti]

BPO	1944
Nosy	@malemburg, @birkenfeld, @vstinner, @avassalotti, @berkerpeksag, @vadmium, @serhiy-storchaka, @shihai1991, @furkanonder
PRs	bpo-1944: wrap functions with macro #20011
Files	unicode.patch: docs for PyUnicodes C-API functions: FromFormat, FromFormatV, FromString, FromStringAndSize, Partition, RPartition and RSplit

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2008-01-27.06:26:42.019>
labels = ['type-feature', 'docs']
title = 'Document PyUnicode_* API'
updated_at = <Date 2020-06-21.12:39:22.129>
user = 'https://github.com/avassalotti'

bugs.python.org fields:

activity = <Date 2020-06-21.12:39:22.129>
actor = 'shihai1991'
assignee = 'docs@python'
closed = False
closed_date = None
closer = None
components = ['Documentation']
creation = <Date 2008-01-27.06:26:42.019>
creator = 'alexandre.vassalotti'
dependencies = []
files = ['13717']
hgrepos = []
issue_num = 1944
keywords = ['patch']
message_count = 9.0
messages = ['61734', '86116', '86121', '89100', '185552', '185725', '264571', '264575', '368582']
nosy_count = 11.0
nosy_names = ['lemburg', 'georg.brandl', 'vstinner', 'alexandre.vassalotti', 'donlorenzo', 'docs@python', 'berker.peksag', 'martin.panter', 'serhiy.storchaka', 'shihai1991', 'furkanonder']
pr_nums = ['20011']
priority = 'normal'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue1944'
versions = ['Python 3.5', 'Python 3.6']

[edit @encukou]: Converting the lists below to checkboxes:

• PyUnicode_Resize
• PyUnicode_InternImmortal (removed in 3.12)
• PyUnicode_GetDefaultEncoding
• PyUnicode_SetDefaultEncoding (removed in 3.2)
• PyUnicode_BuildEncodingMap
• PyUnicode_FromFormatV
• PyUnicode_UTF7
• PyUnicode_AsEncodedObject
• PyUnicode_FromOrdinal
• PyUnicode_DecodeFSDefault
• PyUnicode_DecodeFSDefaultAndSize
• PyUnicode_DecodeUTF7
• PyUnicode_DecodeUTF7Stateful
• PyUnicode_EncodeDecimal
• PyUnicode_EncodeUTF7
• PyUnicode_FromFormat
• PyUnicode_FromString
• PyUnicode_FromStringAndSize
• PyUnicode_GetMax
• PyUnicode_Partition
• PyUnicode_RPartition
• PyUnicode_RSplit
• PyUnicode_IsIdentifier
• PyUnicode_Append
• PyUnicode_AppendAndDel
• Py_UNICODE_REPLACEMENT_CHARACTER
• PyUnicodeIter_Type
• PyUnicode_AsDecodedObject
• PyUnicode_AsDecodedUnicode
• PyUnicode_AsEncodedUnicode
• _PyUnicode_ClearStaticStrings (removed in 3.9)
• _PyUnicode_EQ (removed in 3.14)
• _PyUnicode_FromId (private)

Linked PRs

• gh-46236: Add missing document of PyUnicode_DecodeCodePageStateful #127934
• [3.13] gh-46236: Document PyUnicode_DecodeCodePageStateful (GH-127934) #129961
• [3.12] gh-46236: Document PyUnicode_DecodeCodePageStateful (GH-127934) #129962
• gh-46236: PyUnicode docs improvements #129966
• gh-46236: Document PyUnicode_RSplit, PyUnicode_Partition and PyUnicode_RPartition #130191
• gh-46236: Add missing PyUnicode_GetDefaultEncoding() doc #130335
• [3.13] gh-46236: Document PyUnicode_RSplit, PyUnicode_Partition and PyUnicode_RPartition (GH-130191) #130360
• [3.12] gh-46236: Document PyUnicode_RSplit, PyUnicode_Partition and PyUnicode_RPartition (GH-130191) #130361
• [3.13] gh-46236: Add docs for PyUnicode_GetDefaultEncoding() doc (GH-130335) #130511
• [3.12] gh-46236: Add docs for PyUnicode_GetDefaultEncoding() doc (GH-130335) #130512
• gh-46236: Add missing PyUnicode_Append() doc #130531
• gh-46236: Add missing PyUnicode_FromOrdinal() doc #132040
• [3.13] gh-46236: Add missing PyUnicode_FromOrdinal() doc (#132040) #132521
• gh-46236: Add missing PyUnicode_Resize() doc #132628
• gh-46236: Document PyUnicode_Resize #132703
• gh-46236: Add Py_UNICODE_REPLACEMENT_CHARACTER doc #132706
• gh-46236: Document PyUnicodeIter_Type #132925

[avassalotti]

I was wandering whether the pointer returned by PyUnicode_AsString needs
to be freed after usage (It turned it doesn't since the result is
cached). However, I found out that there isn't any documentation on
docs.python.org about the PyUnicode_AsString and
PyUnicode_AsStringAndSize functions. Although, both are documented in
the public unicodeobject.h header.

I notice that the documentation for several other unicode functions is
missing. Quickly, I see:

PyUnicode_Resize
PyUnicode_InternImmortal
PyUnicode_GetDefaultEncoding
PyUnicode_SetDefaultEncoding
PyUnicode_BuildEncodingMap
PyUnicode_FromFormatV
PyUnicode_*UTF7*
PyUnicode_AsEncodedObject
PyUnicode_FromOrdinal
PyUnicode_DecodeFSDefault
PyUnicode_DecodeFSDefaultAndSize

It would probably be a good idea to polish up the documentation for
PyUnicode as much as possible for Python 3000, since extension
developers will certainly need to refer to it a lot during the
transition from 2.x.

[donlorenzo]

In addition to the above mentioned functions I found these to be
undocumented:

PyUnicode_DecodeUTF7
PyUnicode_DecodeUTF7Stateful
PyUnicode_EncodeDecimal
PyUnicode_EncodeUTF7
PyUnicode_FromFormat
PyUnicode_FromString
PyUnicode_FromStringAndSize
PyUnicode_GetMax
PyUnicode_Partition
PyUnicode_RPartition
PyUnicode_RSplit

From the original list the following functions seem to have been removed:

PyUnicode_InternImmortal
PyUnicode_DecodeFSDefault
PyUnicode_DecodeFSDefaultAndSize

I try to put together a patch for some of these during the weekend.

[donlorenzo]

Ok, here is my shot at a patch for at least some of the undocumented
functions. Namely the following functions are being documented in the patch:

PyUnicode_FromFormat
PyUnicode_FromFormatV
PyUnicode_FromString
PyUnicode_FromStringAndSize
PyUnicode_Partition
PyUnicode_RPartition
PyUnicode_RSplit

Please thoroughly review this patch since I didn't really digg into the
source to find out what the functions do but rather just copied old
PyString documentation or derived it from the docs for the Python API.

[avassalotti]

The patch looks alright. I don't like the documentation for
PyUnicode_FromFormatV, however. Here's my attempt to document it:

.. cfunction:: PyObject* PyUnicode_FromFormatV(const char *format,
va_list vargs)

Equivalent to the function :cfunc:`PyUnicode_FromFormat`, except that
it takes a va_list instead of variable number of arguments.

[BreamoreBoy]

Is it worth applying the patch given the complete rewrite of unicode for 3.3 via PEP-393?

[malemburg]

On 30.03.2013 13:09, Mark Lawrence wrote:

Is it worth applying the patch given the complete rewrite of unicode for 3.3 via PEP-393?

PEP-393 only changed the way Unicode is internally stored.
The Unicode API is mostly unaffected by this change.

[berkerpeksag]

Remaining undocumented functions:

From this issue:

PyUnicode_RSplit
PyUnicode_Partition
PyUnicode_RPartition

From bpo-10435:

PyUnicode_IsIdentifier
PyUnicode_Append
PyUnicode_AppendAndDel
PyUnicode_GetDefaultEncoding
PyUnicode_FromOrdinal
PyUnicode_Resize
PyUnicode_GetMax
PyUnicode_InternImmortal
PyUnicode_CHECK_INTERNED

From bpo-18688:

Py_UNICODE_REPLACEMENT_CHARACTER
PyUnicodeIter_Type
PyUnicode_AsDecodedObject
PyUnicode_AsDecodedUnicode
PyUnicode_AsEncodedObject
PyUnicode_AsEncodedUnicode
PyUnicode_BuildEncodingMap

[serhiy-storchaka]

PyUnicode_DecodeCodePageStateful

Following functions likely should be wrapped with "#ifndef Py_LIMITED_API":

_PyUnicode_ClearStaticStrings
_PyUnicode_EQ
_PyUnicode_FromId

[vstinner]

"""
Following functions likely should be wrapped with "#ifndef Py_LIMITED_API":

_PyUnicode_ClearStaticStrings
_PyUnicode_EQ
_PyUnicode_FromId
"""

It's already the case since at least Python 3.7. Extract of Python 3.7 Include/unicodeobject.h:

#ifndef Py_LIMITED_API
/* Return an interned Unicode object for an Identifier; may fail if there is no memory.*/
PyAPI_FUNC(PyObject*) _PyUnicode_FromId(_Py_Identifier*);
/* Clear all static strings. */
PyAPI_FUNC(void) _PyUnicode_ClearStaticStrings(void);

/* Fast equality check when the inputs are known to be exact unicode types
   and where the hash values are equal (i.e. a very probable match) */
PyAPI_FUNC(int) _PyUnicode_EQ(PyObject *, PyObject *);
#endif /* !Py_LIMITED_API */

[rruuaanng]

@encukou Let's remove the PyUnicode_GetMax entry, which was removed in Misc/NEWS.d/3.10.0a1.rst.

.. bpo: 41123
.. date: 2020-06-28-11-39-22
.. nonce: sjJWjQ
.. section: C API

Removed ``PyUnicode_GetMax()``.

[vstinner]

Let's remove the PyUnicode_GetMax entry

I checked the checkbox :-)

[StanFromIreland]

PyUnicode_EncodeDecimal was removed in 3.11

[StanFromIreland]

PyUnicode_AsDecodedObject, PyUnicode_AsDecodedUnicode, PyUnicode_AsEncodedUnicode, PyUnicode_AsEncodedObject are deprecated since 3.6 and should probably be removed by now, do we still want to document them?

cpython/Include/unicodeobject.h

Line 351 in 05d0559

Py_DEPRECATED(3.6) PyAPI_FUNC(PyObject*) PyUnicode_AsDecodedObject(

cpython/Include/unicodeobject.h

Line 364 in 05d0559

Py_DEPRECATED(3.6) PyAPI_FUNC(PyObject*) PyUnicode_AsDecodedUnicode(

cpython/Include/unicodeobject.h

Line 400 in 05d0559

Py_DEPRECATED(3.6) PyAPI_FUNC(PyObject*) PyUnicode_AsEncodedUnicode(

cpython/Include/unicodeobject.h

Line 378 in 05d0559

Py_DEPRECATED(3.6) PyAPI_FUNC(PyObject*) PyUnicode_AsEncodedObject(

[vstinner]

No, I don't think that it's worth it to document them if they are going to be removed.

[StanFromIreland]

PyUnicode_FromFormatV is documented.

[StanFromIreland]

No, I don't think that it's worth it to document them if they are going to be removed.

I agree, though they have been deprecated for many years but there is no planned removal date that I can find. Should we just remove them now?

[vstinner]

I agree, though they have been deprecated for many years but there is no planned removal date that I can find. Should we just remove them now?

IMO now it's too late for the 3.14 release cycle. It would be better to schedule such removal at the start of a new dev cycle, such as Python 3.15. Someone also has to check if these functions are used in the wild (ex: run a code search).

[StanFromIreland]

cc @serhiy-storchaka who originally deprecated them in 0093907

A new issue is probably best for this, scheduling for 3.15 seems good for now.