Add functions to get the width in columns of a character

[vstinner]

BPO	12568
Nosy	@malemburg, @loewis, @terryjreedy, @vstinner, @benjaminp, @ezio-melotti, @merwok, @bitdancer, @serhiy-storchaka, @Vermeille, @ishigoya, @bianjp
Files	locale_width.patch width.py

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2018-11-08.21:45:08.936>
created_at = <Date 2011-07-14.22:43:56.356>
labels = ['type-feature', '3.8', 'expert-unicode']
title = 'Add functions to get the width in columns of a character'
updated_at = <Date 2018-11-08.21:45:08.934>
user = 'https://github.com/vstinner'

bugs.python.org fields:

activity = <Date 2018-11-08.21:45:08.934>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2018-11-08.21:45:08.936>
closer = 'vstinner'
components = ['Unicode']
creation = <Date 2011-07-14.22:43:56.356>
creator = 'vstinner'
dependencies = []
files = ['23401', '24773']
hgrepos = []
issue_num = 12568
keywords = ['patch']
message_count = 39.0
messages = ['140376', '140488', '141936', '145497', '145498', '145523', '145535', '145748', '145778', '155223', '155236', '155307', '155313', '155323', '155324', '155337', '155342', '155343', '155344', '155345', '155346', '155361', '155370', '155373', '155379', '155382', '156337', '156348', '181149', '238425', '255421', '297129', '297489', '297492', '297564', '297569', '298322', '323731', '329488']
nosy_count = 19.0
nosy_names = ['lemburg', 'loewis', 'terry.reedy', 'vstinner', 'benjamin.peterson', 'ezio.melotti', 'eric.araujo', 'Arfrever', 'r.david.murray', 'inigoserna', 'zeha', 'poq', 'Nicholas.Cole', 'tchrist', 'serhiy.storchaka', 'Socob', 'Guillaume Sanchez', 'ishigoya', 'bianjp']
pr_nums = []
priority = 'normal'
resolution = 'wont fix'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue12568'
versions = ['Python 3.8']

[vstinner]

Some characters take more than one column in a terminal, especially CJK (chinese, japanese, korean) characters. If you use such character in a terminal without taking care of the width in columns of each character, the text alignment can be broken. Issue bpo-2382 is an example of this problem.

bpo-2382 and bpo-6755 have patches implementing such function:

• unicode_width.patch of bpo-2382 adds unicode.width() method
• ucs2w.c of bpo-6755 creates a new ucs2w module with two functions: unichr2w() (width of a character) and ucs2w() (width of a string)

Use test_ucs2w.py of bpo-6755 to test these new functions/methods.

[loewis]

In the bpo-2382 code, how is the Windows case supposed to work? Also, what about systems that don't have wcswidth? IOW, the patch appears to be incorrect.

I like the bpo-6755 approach better, except that it shouldn't be using hard-coded tables, but instead integrate with Python's version of the UCD. In addition, it should use an accepted, published strategy for determining the width, preferably coming from the Unicode consortium.

[tchrist]

I can attest that being able to get the columns of a grapheme cluster is very important for printing, because you need this to do correct linebreaking. There might be something you can steal from

http://search.cpan.org/perldoc?Unicode::GCString
http://search.cpan.org/perldoc?Unicode::LineBreak

which implements UAX#14 on linebreaking and UAX#11 on East Asian widths.

I use this in my own code to help format Unicode strings my columns or lines. The right way would be to build this sort of knowledge into string.format(), but that is much harder, so an intermediary library module seems good enough for now.

[vstinner]

There might be something you can steal from ...

I don't think that Python should reinvent the wheel. We should just reuse wcswidth().

Here is a simple patch exposing wcswidth() function as locale.width().

Example:

>>> import locale
>>> text = '\u3042\u3044\u3046\u3048\u304a'
>>> len(text)
5
>>> locale.width(text)
10
>>> locale.width(' ')
1
>>> locale.width('\U0010abcd')
1
>>> locale.width('\uDC80')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
locale.Error: the string is not printable
>>> locale.width('\U0010FFFF')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
locale.Error: the string is not printable

I don't think that we need locale.width() on Windows because its console has already bigger issues with Unicode: see issue bpo-1602. If you want to display correctly non-ASCII characters on Windows, just avoid the Windows console and use a graphical widget.

[vstinner]

Oh, unicode_width.patch of issue bpo-2382 implements the width on Windows using:

WideCharToMultiByte(CP_ACP, 0, buf, len, NULL, 0, NULL, NULL);

It computes the length of byte string encoded to the ANSI code page. I don't know if it can be seen as the "width" of a character string in the console...

[loewis]

I think the WideCharToMultibyte approach is just incorrect.

I'm -1 on using wcswidth, though. We already have unicodedata.east_asian_width, which implements http://unicode.org/reports/tr11/
The outcomes of this function are these:

• F: full-width, width 2, compatibility character for a narrow char
• H: half-width, width 1, compatibility character for a narrow char
• W: wide, width 2
• Na: narrow, width 1
• A: ambiguous; width 2 in Asian context, width 1 in non-Asian context
• N: neutral; not used in Asian text, so has no width. Practically, width can be considered as 1

[tchrist]

Martin v. Löwis <martin@v.loewis.de> added the comment:

I think the WideCharToMultibyte approach is just incorrect.

I'm -1 on using wcswidth, though.

Like you, I too seriously question using wcswidth() for this at all:

The wcswidth() function either shall return 0 (if pwcs points to a
null wide-character code), or return the number of column positions
to be occupied by the wide-character string pointed to by pwcs, or
return -1 (if any of the first n wide-character codes in the wide-
character string pointed to by pwcs is not a printable wide-
character code).

I would be willing to bet (a small amount of) money it does not correctly
inplmented Unicode print widths, even though one would certainly *think* it
does according to this:

 The wcswidth() function determines the number of column positions
 required for the first n characters of pwcs, or until a null wide
 character (L'\0') is encountered.

There are a bunch of "interesting" cases I would want it tested against.

We already have unicodedata.east_asian_width, which implements http://unicode.org/reports/tr11/

The outcomes of this function are these:

• F: full-width, width 2, compatibility character for a narrow char
• H: half-width, width 1, compatibility character for a narrow char
• W: wide, width 2
• Na: narrow, width 1
• A: ambiguous; width 2 in Asian context, width 1 in non-Asian context
• N: neutral; not used in Asian text, so has no width. Practically, width can be considered as 1

Um, East_Asian_Width=Ambiguous (EA=A) isn't actually good enough for this.
And EA=N cannot be consider 1, either.

For example, some of the Marks are EA=A and some are EA=N, yet how may
print columns they take varies. It is usually 0, but can be 1 at the start
of the file/string or immediately after a linebreak sequence. Then there
are things like the variation selectors which are never anything.

Now consider the many \pC code points, like

U+0009  CHARACTER TABULATION
U+00AD  SOFT HYPHEN 
U+200C  ZERO WIDTH NON-JOINER
U+FEFF  ZERO WIDTH NO-BREAK SPACE
U+2062  INVISIBLE TIMES

A TAB is its own problem but SHY we know is only width=1 immediately
before a linebreak or EOF, and ZWNJ and ZWNBSP are both certainly
width=0. So are the INVISIBLE * code points.

Context:

Imagine you're trying to format a string so that it takes up exactly 20
columns: you need to know how many spaces to pad it with based on the
print width. That is what the bpo-12568 is needing
to do, and you have to do much more than East Asian Width properties.

I really do think that what bpo-12568 is asking for is to have the equivalent
of the Perl Unicode::GCString's columns() method, and that you aren't going
to be able to handle text alignment of Unicode with anything that is much
less of that. After all, bpo-12568's title is "Add functions to get the width
in columns of a character". I would very much like to compare what
columns() thinks compared with what wcswidth() thinks. I bet wcswidth() is
very simple-minded at best.

I may of course be wrong.

--tom

[vstinner]

I'm -1 on using wcswidth, though.

When you write text into a console on Linux (e.g. displayed by gnome-terminal or konsole), I suppose that wcswidth() can be used to compute the width of a line. It would help to fix bpo-2382.

Or do you think that wcswidth() gives the wrong result for this use case?

[loewis]

> I'm -1 on using wcswidth, though.

When you write text into a console on Linux (e.g. displayed by
gnome-terminal or konsole), I suppose that wcswidth() can be used to
compute the width of a line. It would help to fix bpo-2382.

Or do you think that wcswidth() gives the wrong result for this use
case?

No, I think that using it is not necessary. If you want to compute the
width of a line, use unicodedata.east_asian_width. And yes, wcswidth
may sometimes produce "incorrect" results (although it's probably
correct most of the time).

[NicholasCole]

Could we have an update on the status of this? I ask because if 3.3 is going to (finally) fix unicode for curses, it would be really nice if it were possible to calculate the width of what's being displayed! It looks as if there was never quite agreement on the proper API....

[loewis]

Nicholas: I consider this issue fixed. There already *is* any API to compute the width of a character. Closing this as "works for me".

[NicholasCole]

Martin: sorry to be completely dense, but I can't get this to work properly with the python3.3a1 build. Could you post some example code?

[loewis]

Please see the attached width.py for an example