Enable non-ASCII extension module names

[Suzumizaki]

BPO	20485
Nosy	@brettcannon, @amauryfa, @ncoghlan, @scoder, @vstinner, @jkloth, @ericsnowcurrently, @serhiy-storchaka
Files	20140202_patch_for_python_default_branch_88885.patch: Enable "import .pyd" patch for default branch 20140202_patch_for_python3.3_88884.patch: Enable "import .pyd" patch for 3.3 branch sample_importing_non_ascii_pyd.zip: Samples

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2017-06-13.23:10:49.397>
created_at = <Date 2014-02-02.14:20:28.154>
labels = ['interpreter-core', 'type-feature']
title = 'Enable non-ASCII extension module names'
updated_at = <Date 2017-06-13.23:10:49.396>
user = 'https://bugs.python.org/Suzumizaki'

bugs.python.org fields:

activity = <Date 2017-06-13.23:10:49.396>
actor = 'ncoghlan'
assignee = 'none'
closed = True
closed_date = <Date 2017-06-13.23:10:49.397>
closer = 'ncoghlan'
components = ['Interpreter Core']
creation = <Date 2014-02-02.14:20:28.154>
creator = 'Suzumizaki'
dependencies = []
files = ['33866', '33867', '33868']
hgrepos = []
issue_num = 20485
keywords = ['patch']
message_count = 20.0
messages = ['209988', '209998', '210002', '210059', '210064', '210076', '210082', '210096', '210120', '210124', '210125', '210177', '210209', '210211', '210220', '210221', '210292', '210293', '242049', '295968']
nosy_count = 10.0
nosy_names = ['brett.cannon', 'amaury.forgeotdarc', 'ncoghlan', 'scoder', 'vstinner', 'jkloth', 'Arfrever', 'Suzumizaki', 'eric.snow', 'serhiy.storchaka']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue20485'
versions = ['Python 3.5']

[Suzumizaki]

Currently, the name of .pyd modules is limited within 7 bit US-ASCII.

I want to do "import X" to import X.pyd, where X contains Unicode characters.
I tried to make the patch and my work seems to be done successfully.
I will post the patch with this issue, and next what should I do?

About the solution:
To make the export entry 'PyInit_xxx' kept inside 7 bit, I use the simple encoding 'szm62' for unicode, called in the patch.

1. 'szm62' is used once per module(.pyd), only for 'PyInit_xxx'.
2. 'szm62' is used only when non-ASCII characters are in the name of the module.
3. 'szm62' generates short string as UTF-8, except 0-9A-Za-z are encoded to 2 bytes.
4. 'szm62' is very simple, much easier than UTF-8.
5. I tested it only with MS VC++, but I believe highly compatible with other environments.
6. 'szm62' also can decode 7 bits to unicode, but only the encoding is used for this issue.

Notes:
The simplicity is important for the project like Cython -- it generates .pyd files.
The codepoints over 16bits are also simply supported. They will be encoded to 4 alnum(0-9A-Za-z) characters.
0-9A-Za-z are (always) encoded to 2 alnums. They will be simply prefixed with '0'(U+000030).
When the generating 'Non-ASCII.pyd' with MSVC toolkit, the report 'LINK : warning LNK4232' will be raised on linking. But no problem. The warning says "When someone try to link with LoadLibraryEx'A' function, it may or may not fail, depends on the user locale." Our Python.exe uses always LoadLibraryEx'W', it never fail with locale issue.

Or if you have any question, please tell me that.

Regards,
Suzumizaki-Kimitaka

[serhiy-storchaka]

I think that if you need a module with non-ASCII name, you can wrap it in Python module.

=== späm.py ===

from _spam import *
from _spam import __all__, __doc__
=== späm.py
```===

[amauryfa]

I'd use a much simpler encoding.
Maybe something like
name.encode('unicode-escape').replace(b'\\', b'_')
As you said, simplicity is important for tools which generate code!

[Suzumizaki]

Thank you for reply.

The hack msg209998 is interesting, but how to name submodule with non latin like languages, especially keeping native reable? X(

The reason I don't use like "name.encode('unicode-escape').replace(b'\\', b'_')" is the length limits of the identifiers.

In fact, Visual C++ can accept 2047 chars(bytes) and gcc have no logical limits. But the PEP-7 says we should use C89. And even C99 assumes first 63 bytes are significant. I don't know what C89 says, And my C99 reference is below, this means real-C99 is possibly different:
http://www.open-std.org/JTC1/SC22/wg14/www/docs/n1124.pdf

If we should keep C99 order above, 63 chars are too short to use 'unicode-escape' like. 'PyInit_' takes 7, remains 56. When each characters encoded as 5 chars like '_3010', only we can use 11 unicode-codepoints. When 6 chars, only 9 chars.

a) If we can break C99 or real-C89/C99 don't have 63 chars rule, we can simply use as Amaury Forgeot d'Arc says.
b) If we should keep 63 chars rules, the encodings longer than 'szm62' is not acceptable.
c) Of course, when there is no reason to make entry point 'PyInit_modulename', Make enable stable constant name 'PyInit' or 'PyInitUnicodeNamedModule' without individual module name is another acceptable idea.

[vstinner]

The PyInit_NAME symbol is not the only place where NAME is used. The NAME is also present in the PyModuleDef structure. It looks lie Python expects UTF-8 here. You encode it to UTF-8 and use "\xHH\xHH\xHH..." syntax to keep ASCII encoding for the C file? The NAME may also be mentionned in docstrings, C comments, type names, etc.

I don't like the idea of a new encoding just for one very specific function in C. There are already too many encodings in the world :-( The C language supports non-ASCII identifiers, but I don't know how they are encoded in the symbol table. I would prefer to rely on the C compiler if you would like to play in the playground of non-ASCII identifiers.

In Python/dynload_win.c, _PyImport_GetDynLoadWindows() uses GetProcAddressA().

Is it a theorical feature request, or you really have a Python module with a non-ASCII name?

I'm not sure that it's really useful to support non-ASCII module names for C modules, even if I spend many months to support non-ASCII module names for Python modules :-)

[Suzumizaki]

Thank you for reply, STINNER.

You encode it to UTF-8 and use "\xHH\xHH\xHH..." syntax to keep ASCII
encoding for the C file? The NAME may also be mentionned in docstrings,
C comments, type names, etc.

The main purpose of this issue is "I want use Cython like Python without any trouble." You don't have to worry about mentioned above. I made, and will fix when needed, the patch for Cython to convert them automatically.

The sample C codes I posted uses UTF-8 directly only OUTSIDE of the "quotation", but they can be fixed if we really have to fix.

I don't like the idea of a new encoding just for one very specific
function in C. There are already too many encodings in the world :-(

Of course I will accept any encoding and/or any solution to resolve this issue. I made new encoding only to keep the condition as possible as I can, and not to limit the naming too short when using non-ASCII characters.

The patch don't include encoding-module for any purpose. For this issue, decoding is not required inside the Python.

The C language supports non-ASCII identifiers, but I don't know how they
are encoded in the symbol table.

That's why we should resolve this problem, shouldn't we? Also the standards don't define about the symbol table.

I would prefer to rely on the C compiler if you would like to play in the
playground of non-ASCII identifiers.

The problem is we CAN'T as you say. Or, at least, if you really think that, any ASCII limiting against dynamic loading should be removed.

In Python/dynload_win.c, _PyImport_GetDynLoadWindows() uses GetProcAddressA().

_PyImport_GetDynLoadWindows() seems to be called only to resolve PyInit_xxx entry from _PyImport_LoadDynamicModule() in Python/importdl.c. I have already resolved with the posted patch before.

Is it a theorical feature request, or you really have a Python module with
a non-ASCII name?

As I told, NO to 1st, YES to 2nd. I have many '<non-ASCII>.py' which I want to convert using Cython to '<non-ASCII>.pyd' files.

I'm not sure that it's really useful to support non-ASCII module names
for C modules, even if I spend many months to support non-ASCII module
names for Python modules :-)

Because you are both English and Python expert. Thanks a lot to daily Python work!

Thank you for reading this long description.

[serhiy-storchaka]

The hack msg209998 is interesting, but how to name submodule with non latin like languages, especially keeping native reable? X(

It is left to your discretion. You can use idna, punycode, utf-7, szm62 or romaji.

[ncoghlan]

Updating the C extension loading API to take advantage of PEP-451 is on the
to do list for 3.5, so I'll see if we can do something about this as well.
However, as Victor noted, it will depend on whether or not we can figure
out a compiler independent cross platform way to look up a non-ASCII symbol
in the extension module's symbol table.

[Suzumizaki]

Thanks for taking into account this issue for PEP-451.

Honestly to say, I can't imagine why or/and how this issue(or my patches) causes any problems especially compatibility issues. If someone can point them, I will try to resolve.

Note that I extend only the definition of "PyInit_xxxx". I don't touch the code for loading modules.

[ncoghlan]

As Victor noted, inventing our own encoding scheme just for this use case
isn't desirable, although it's certainly a good fallback option that will
ensure the feature remains feasible even if trying to handle the Unicode
issues at the C compiler level proves too challenging.

The other aspect is that changes to the extension module initialisation API
always need to go into a PEP regardless, since we need to ensure such
changes are usable for both handwritten extensions and extension module
generators like Cython, cffi and SWIG.

[vstinner]

Oh, the topic was already discussed some years ago. Start:
https://mail.python.org/pipermail/python-dev/2011-May/111279.html