Enable non-ASCII extension module names

[Suzumizaki]

BPO	20485
Nosy	@brettcannon, @amauryfa, @ncoghlan, @scoder, @vstinner, @jkloth, @ericsnowcurrently, @serhiy-storchaka
Files	20140202_patch_for_python_default_branch_88885.patch: Enable "import .pyd" patch for default branch 20140202_patch_for_python3.3_88884.patch: Enable "import .pyd" patch for 3.3 branch sample_importing_non_ascii_pyd.zip: Samples

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2017-06-13.23:10:49.397>
created_at = <Date 2014-02-02.14:20:28.154>
labels = ['interpreter-core', 'type-feature']
title = 'Enable non-ASCII extension module names'
updated_at = <Date 2017-06-13.23:10:49.396>
user = 'https://bugs.python.org/Suzumizaki'

bugs.python.org fields:

activity = <Date 2017-06-13.23:10:49.396>
actor = 'ncoghlan'
assignee = 'none'
closed = True
closed_date = <Date 2017-06-13.23:10:49.397>
closer = 'ncoghlan'
components = ['Interpreter Core']
creation = <Date 2014-02-02.14:20:28.154>
creator = 'Suzumizaki'
dependencies = []
files = ['33866', '33867', '33868']
hgrepos = []
issue_num = 20485
keywords = ['patch']
message_count = 20.0
messages = ['209988', '209998', '210002', '210059', '210064', '210076', '210082', '210096', '210120', '210124', '210125', '210177', '210209', '210211', '210220', '210221', '210292', '210293', '242049', '295968']
nosy_count = 10.0
nosy_names = ['brett.cannon', 'amaury.forgeotdarc', 'ncoghlan', 'scoder', 'vstinner', 'jkloth', 'Arfrever', 'Suzumizaki', 'eric.snow', 'serhiy.storchaka']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue20485'
versions = ['Python 3.5']

[Suzumizaki]

Currently, the name of .pyd modules is limited within 7 bit US-ASCII.

I want to do "import X" to import X.pyd, where X contains Unicode characters.
I tried to make the patch and my work seems to be done successfully.
I will post the patch with this issue, and next what should I do?

About the solution:
To make the export entry 'PyInit_xxx' kept inside 7 bit, I use the simple encoding 'szm62' for unicode, called in the patch.

1. 'szm62' is used once per module(.pyd), only for 'PyInit_xxx'.
2. 'szm62' is used only when non-ASCII characters are in the name of the module.
3. 'szm62' generates short string as UTF-8, except 0-9A-Za-z are encoded to 2 bytes.
4. 'szm62' is very simple, much easier than UTF-8.
5. I tested it only with MS VC++, but I believe highly compatible with other environments.
6. 'szm62' also can decode 7 bits to unicode, but only the encoding is used for this issue.

Notes:
The simplicity is important for the project like Cython -- it generates .pyd files.
The codepoints over 16bits are also simply supported. They will be encoded to 4 alnum(0-9A-Za-z) characters.
0-9A-Za-z are (always) encoded to 2 alnums. They will be simply prefixed with '0'(U+000030).
When the generating 'Non-ASCII.pyd' with MSVC toolkit, the report 'LINK : warning LNK4232' will be raised on linking. But no problem. The warning says "When someone try to link with LoadLibraryEx'A' function, it may or may not fail, depends on the user locale." Our Python.exe uses always LoadLibraryEx'W', it never fail with locale issue.

Or if you have any question, please tell me that.

Regards,
Suzumizaki-Kimitaka

[serhiy-storchaka]

I think that if you need a module with non-ASCII name, you can wrap it in Python module.

=== späm.py ===

from _spam import *
from _spam import __all__, __doc__
=== späm.py
```===

[amauryfa]

I'd use a much simpler encoding.
Maybe something like
name.encode('unicode-escape').replace(b'\\', b'_')
As you said, simplicity is important for tools which generate code!

[Suzumizaki]

Thank you for reply.

The hack msg209998 is interesting, but how to name submodule with non latin like languages, especially keeping native reable? X(

The reason I don't use like "name.encode('unicode-escape').replace(b'\\', b'_')" is the length limits of the identifiers.

In fact, Visual C++ can accept 2047 chars(bytes) and gcc have no logical limits. But the PEP-7 says we should use C89. And even C99 assumes first 63 bytes are significant. I don't know what C89 says, And my C99 reference is below, this means real-C99 is possibly different:
http://www.open-std.org/JTC1/SC22/wg14/www/docs/n1124.pdf

If we should keep C99 order above, 63 chars are too short to use 'unicode-escape' like. 'PyInit_' takes 7, remains 56. When each characters encoded as 5 chars like '_3010', only we can use 11 unicode-codepoints. When 6 chars, only 9 chars.

a) If we can break C99 or real-C89/C99 don't have 63 chars rule, we can simply use as Amaury Forgeot d'Arc says.
b) If we should keep 63 chars rules, the encodings longer than 'szm62' is not acceptable.
c) Of course, when there is no reason to make entry point 'PyInit_modulename', Make enable stable constant name 'PyInit' or 'PyInitUnicodeNamedModule' without individual module name is another acceptable idea.

[vstinner]

The PyInit_NAME symbol is not the only place where NAME is used. The NAME is also present in the PyModuleDef structure. It looks lie Python expects UTF-8 here. You encode it to UTF-8 and use "\xHH\xHH\xHH..." syntax to keep ASCII encoding for the C file? The NAME may also be mentionned in docstrings, C comments, type names, etc.

I don't like the idea of a new encoding just for one very specific function in C. There are already too many encodings in the world :-( The C language supports non-ASCII identifiers, but I don't know how they are encoded in the symbol table. I would prefer to rely on the C compiler if you would like to play in the playground of non-ASCII identifiers.

In Python/dynload_win.c, _PyImport_GetDynLoadWindows() uses GetProcAddressA().

Is it a theorical feature request, or you really have a Python module with a non-ASCII name?

I'm not sure that it's really useful to support non-ASCII module names for C modules, even if I spend many months to support non-ASCII module names for Python modules :-)

[Suzumizaki]

Thank you for reply, STINNER.

You encode it to UTF-8 and use "\xHH\xHH\xHH..." syntax to keep ASCII
encoding for the C file? The NAME may also be mentionned in docstrings,
C comments, type names, etc.

The main purpose of this issue is "I want use Cython like Python without any trouble." You don't have to worry about mentioned above. I made, and will fix when needed, the patch for Cython to convert them automatically.

The sample C codes I posted uses UTF-8 directly only OUTSIDE of the "quotation", but they can be fixed if we really have to fix.

I don't like the idea of a new encoding just for one very specific
function in C. There are already too many encodings in the world :-(

Of course I will accept any encoding and/or any solution to resolve this issue. I made new encoding only to keep the condition as possible as I can, and not to limit the naming too short when using non-ASCII characters.

The patch don't include encoding-module for any purpose. For this issue, decoding is not required inside the Python.

The C language supports non-ASCII identifiers, but I don't know how they
are encoded in the symbol table.

That's why we should resolve this problem, shouldn't we? Also the standards don't define about the symbol table.

I would prefer to rely on the C compiler if you would like to play in the
playground of non-ASCII identifiers.

The problem is we CAN'T as you say. Or, at least, if you really think that, any ASCII limiting against dynamic loading should be removed.

In Python/dynload_win.c, _PyImport_GetDynLoadWindows() uses GetProcAddressA().

_PyImport_GetDynLoadWindows() seems to be called only to resolve PyInit_xxx entry from _PyImport_LoadDynamicModule() in Python/importdl.c. I have already resolved with the posted patch before.

Is it a theorical feature request, or you really have a Python module with
a non-ASCII name?

As I told, NO to 1st, YES to 2nd. I have many '<non-ASCII>.py' which I want to convert using Cython to '<non-ASCII>.pyd' files.

I'm not sure that it's really useful to support non-ASCII module names
for C modules, even if I spend many months to support non-ASCII module
names for Python modules :-)

Because you are both English and Python expert. Thanks a lot to daily Python work!

Thank you for reading this long description.

[serhiy-storchaka]

The hack msg209998 is interesting, but how to name submodule with non latin like languages, especially keeping native reable? X(

It is left to your discretion. You can use idna, punycode, utf-7, szm62 or romaji.

[ncoghlan]

Updating the C extension loading API to take advantage of PEP-451 is on the
to do list for 3.5, so I'll see if we can do something about this as well.
However, as Victor noted, it will depend on whether or not we can figure
out a compiler independent cross platform way to look up a non-ASCII symbol
in the extension module's symbol table.

[Suzumizaki]

Thanks for taking into account this issue for PEP-451.

Honestly to say, I can't imagine why or/and how this issue(or my patches) causes any problems especially compatibility issues. If someone can point them, I will try to resolve.

Note that I extend only the definition of "PyInit_xxxx". I don't touch the code for loading modules.

[ncoghlan]

As Victor noted, inventing our own encoding scheme just for this use case
isn't desirable, although it's certainly a good fallback option that will
ensure the feature remains feasible even if trying to handle the Unicode
issues at the C compiler level proves too challenging.

The other aspect is that changes to the extension module initialisation API
always need to go into a PEP regardless, since we need to ensure such
changes are usable for both handwritten extensions and extension module
generators like Cython, cffi and SWIG.

[vstinner]

Oh, the topic was already discussed some years ago. Start:
https://mail.python.org/pipermail/python-dev/2011-May/111279.html

[Suzumizaki]

Thank you Nick, I understand the behavior of this issue should be written on PEP.

By the way, Can I continue the discussion here? or is there elsewhere suitable place for the PEP?

[ncoghlan]

import-sig@python.org would be the appropriate list for this one.

However, we can't do anything about it until Python 3.5 next year at the earliest, and I'm already planning to write a follow-up to http://www.python.org/dev/peps/pep-0451/ that adapts the extension module import mechanism to support those APIs (addressing a number of longstanding feature requests from the Cython developers).

That said, this is an independent proposal, so if you were willing to write it up as a separate PEP, that would be probably be a good idea. Our two choices to consider would be:

1. Using a custom 7-bit ASCII compatible encoding to support this on arbitrary C compilers (at the cost of making the identifiers unintuitive). (i.e. the approach in your patch)

2. Using the "Universal Character Name" support originally specified in C99, but retained in C11 (these are the \Uxxxxxxxx and \uxxxx escapes familiar from Python text literals). Note that *CPython* still won't need to be compiled with a compiler that supports UCN for this to work - we'll just need the dynamic linking API to support us looking for a symbol containing such a name.

Option 2 is what I think we *should* do, but there will be some research involved in figuring out how good the current support for UCN C identifiers is in at least gcc, clang and Visual Studio 2013, as well as what the dynamic linker APIs support in terms of passing identifiers containing Unicode escapes to be looked up in the exported symbols.

[vstinner]

we *should* do, but there will be some research involved in figuring out how good the current support for UCN C identifiers is in at least gcc, clang and Visual Studio 2013

Python 3.4 uses Visual Studio 2010. I'm not sure that you can build an
extension with VS 2013 if Python was build with VS 2010.

[ncoghlan]

Oh, you're right - I temporarily forgot that the C runtime compatibility was compiler version specific on Windows. So such an approach *would* require updating the CPython compiler on Windows to at least VS2013 for 3.5. Still, we're likely to want to do that anyway - VS2010 will be as old in 2015 as VS2008 is now, and the latter is already causing hassles for building 2.7 extension modules.

[Suzumizaki]

Both Visual Studio 2012 and 2013 CANNOT install on Windows Vista. That's OK for you even Vista alive until April 2017?

[Suzumizaki]

Thank you Victor about msg210125, I read the discussion on ML, May 2011.

Inside the articles, the previous discussion on tracker is found:
"On Windows, don't encode filenames in the import machinery"
http://bugs.python.org/issue11619

Here is my memo, might be helpful to review the discussions.

-- About Window CE --

• Windows CE series have GetProcAddress() at First.
• with Windows CE 3.0, GetProcAddressA() is added.
• but Python community chose 'A' version to support Windows CE.
• Windows CE continues as Windows Embedded Compact today.
• but Python3 for Windows CE seems not to be distributed.

-- About Windows Desktop and Servers --

• Windows Desktops and Servers have GetProcAddress() only, neither A nor W postfix appended.
• GetProcAddress() on Windows Desktop and Servers takes LPCSTR as the 2nd parameter.
• but the parameter, in this case, is null-terminated binary block. neither MBCS nor UTF-8.
• Visual C++ 2010 encodes non-ASCII export symbols as UTF-8.
• Because the 2 reasons described above the 2 lines, We can give UTF-8 encoded string to GetProcAddress().

I checked the last fact with my Window Japanese Editions:

• XP Home Edition (32bit)
• Vista Home Premium (64bit)
• Windows 8.1 Pro (64bit)

GetProcAddress (Windows CE)
The type of the 2nd parameter is LPC"W"STR, and the document says LPCSTR version added on CE 3.0.
http://msdn.microsoft.com/en-us/library/ms885634.aspx

GetProcAddress (Windows Desktop/Server)
The type of the 2nd parameter is LPCSTR, nor LPC"T"STR neither LPC"W"STR.
Note that the example seems to be wrong about using TEXT macro.
http://msdn.microsoft.com/en-us/library/windows/desktop/ms683212(v=vs.85).aspx

PythonCE (seems stopped at Python 2.5 compatible)
http://pythonce.sourceforge.net/

Symbols seem to be encoded utf-8 inside Windows Executable
https://mail.python.org/pipermail/python-dev/2011-May/111325.html

-- About C/C++ Standards --

• C99 says the significant length of identifiers are 63.
• C99 allows to use Unicode to name identifiers.
• but not define how to translate \uNNNN or \uNNNNNNNN forms used in "quotations".
• C++11 defines u8"" literals. we can make utf-8 char* string inside u8"quotes" with \u formats.
• but the encoding of source file is platform dependent.
• also, how to export symbols is platform dependent.

-- About C/C++ tool kits --

• Window Executable can contain 2048 chars per each exported symbol.
• Visual C++ 2010 seems to encode exporting symbols with UTF-8.
• gcc don't have logical limit of the length of identifiers.
• Currently, Visual C++ 2010 and LLVM/Clang supports using UTF-8 in whole source code.
• gcc only support \uNNNN or \uNNNNNNNN form.
• About GetProcAddress() functions, see previous memo about Windows.

[Suzumizaki]

Thank you Nick about msg210209.

I would like to try making PEP, but the work looks somewhat difficult. It may take the time.

BTW, C/C++ Standards only allow the encoding of source code as platform dependent. They don't define "the standard encoding of source codes"...

This means we have to choose to resolve this issue, one is giving up readability, the other is allowing platform-dependent feature, using UTF-8 to write the C code.

[scoder]

PEP-489 (Redesigning extension module loading) includes the proposal to fix this by using punycode.

[ncoghlan]

PEP-489 was accepted and implemented, so Python 3.5+ supports non-ASCII extension module names as described in https://www.python.org/dev/peps/pep-0489/#export-hook-name