Runtime exception: ImportError: cannot import name exceptions when executing python app binary consuming asynpg

[sreenandan]

We create binary build using pyinstaller.
Earlier we were using sqlite3, but now we have switched to postgres with asynpg.
When we build binary using pyinstaller, we hit issue described below.

Lets take hello world sample:
root@test:/scripts/test# cat hello.py
from asyncpg.exceptions import UniqueViolationError as IntegrityError
print("hello world")

Building Binary:
root@mdcapbin-engine-01:/scripts/test# python3.7 -m PyInstaller --paths /scripts/test --paths /usr/local/lib/python3.7/site-packages --name hello.bin --onefile --distpath ./hello --key=hello --windowed hello.py --hidden-import=databases.backends --hidden-import=databases.backends.postgres --exclude-module MySQLdb --exclude-module psycopg2 --hidden-import=asyncpg.exceptions --hidden-import=asyncpg.pgproto.pgproto --hidden-import=asyncpg.protocol.protocol --hidden-import=asyncpg
35 INFO: PyInstaller: 3.6
35 INFO: Python: 3.7.0
36 INFO: Platform: Linux-3.10.0-1062.9.1.el7.x86_64-x86_64-with-debian-9.5
36 INFO: wrote /scripts/test/hello.bin.spec
38 INFO: UPX is not available.
39 INFO: Extending PYTHONPATH with paths
['/scripts/test',
'/scripts/test',
'/usr/local/lib/python3.7/site-packages',
'/scripts/test']
39 INFO: Will encrypt Python bytecode with key: 00000000000hello
39 INFO: Adding dependencies on pyi_crypto.py module
40 INFO: checking Analysis
47 INFO: Building because /usr/local/lib/python3.7/site-packages/asyncpg/pgproto/pgproto.cpython-37m-x86_64-linux-gnu.so changed
47 INFO: Initializing module dependency graph...
48 INFO: Caching module graph hooks...
52 INFO: Analyzing base_library.zip ...
2923 INFO: Caching module dependency graph...
3002 INFO: running Analysis Analysis-00.toc
3014 INFO: Analyzing /scripts/test/hello.py
4270 INFO: Analyzing hidden import 'databases.backends'
5602 INFO: Processing pre-find module path hook distutils
5603 INFO: distutils: retargeting to non-venv dir '/usr/local/lib/python3.7'
5971 INFO: Analyzing hidden import 'databases.backends.postgres'
6223 INFO: Analyzing hidden import 'asyncpg.pgproto.pgproto'
6223 INFO: Analyzing hidden import 'Crypto.Cipher._AES'
6279 INFO: Processing module hooks...
6279 INFO: Loading module hook "hook-Crypto.py"...
6295 INFO: Loading module hook "hook-distutils.py"...
6297 INFO: Loading module hook "hook-encodings.py"...
6335 INFO: Loading module hook "hook-pkg_resources.py"...
6754 INFO: Processing pre-safe import module hook win32com
6756 INFO: Excluding import 'main'
6757 INFO: Removing import of main from module pkg_resources
6757 INFO: Loading module hook "hook-pydoc.py"...
6757 INFO: Loading module hook "hook-sqlalchemy.py"...
6882 INFO: Found 3 sqlalchemy hidden imports
6882 WARNING: Hidden import "pysqlite2" not found!
7473 WARNING: Hidden import "sqlalchemy.sql.functions.func" not found!
7483 INFO: Import to be excluded not found: 'sqlalchemy.testing'
7483 INFO: Loading module hook "hook-sqlite3.py"...
7517 INFO: Loading module hook "hook-sysconfig.py"...
7524 INFO: Loading module hook "hook-xml.py"...
7767 INFO: Looking for ctypes DLLs
7820 INFO: Analyzing run-time hooks ...
7825 INFO: Including run-time hook 'pyi_rth_multiprocessing.py'
7829 INFO: Including run-time hook 'pyi_rth_pkgres.py'
7838 INFO: Looking for dynamic libraries
8276 INFO: Looking for eggs
8276 INFO: Using Python library /usr/local/lib/libpython3.7m.so.1.0
8283 INFO: Warnings written to /scripts/test/build/hello.bin/warn-hello.bin.txt
8326 INFO: Graph cross-reference written to /scripts/test/build/hello.bin/xref-hello.bin.html
8340 INFO: checking PYZ
8355 INFO: Building because toc changed
8355 INFO: Building PYZ (ZlibArchive) /scripts/test/build/hello.bin/PYZ-00.pyz
9268 INFO: Building PYZ (ZlibArchive) /scripts/test/build/hello.bin/PYZ-00.pyz completed successfully.
9277 INFO: checking PKG
9278 INFO: Building because toc changed
9278 INFO: Building PKG (CArchive) PKG-00.pkg
18509 INFO: Building PKG (CArchive) PKG-00.pkg completed successfully.
18513 INFO: Bootloader /usr/local/lib/python3.7/site-packages/PyInstaller/bootloader/Linux-64bit/run
18513 INFO: checking EXE
18514 INFO: Building because toc changed
18514 INFO: Building EXE from EXE-00.toc
18516 INFO: Appending archive to ELF section in EXE ./hello/hello.bin
18549 INFO: Building EXE from EXE-00.toc completed successfully.

Run binary and hit Runtime Exception:
root@mdcapbin-engine-01:/scripts/test# ./hello/hello.bin
Traceback (most recent call last):
File "hello.py", line 1, in
from asyncpg.exceptions import UniqueViolationError as IntegrityError
File "/usr/local/lib/python3.7/site-packages/PyInstaller/loader/pyimod03_importers.py", line 623, in exec_module
exec(bytecode, module.dict)
File "site-packages/asyncpg/init.py", line 8, in
File "/usr/local/lib/python3.7/site-packages/PyInstaller/loader/pyimod03_importers.py", line 623, in exec_module
exec(bytecode, module.dict)
File "site-packages/asyncpg/connection.py", line 19, in
File "/usr/local/lib/python3.7/site-packages/PyInstaller/loader/pyimod03_importers.py", line 623, in exec_module
exec(bytecode, module.dict)
File "site-packages/asyncpg/connect_utils.py", line 27, in
File "/usr/local/lib/python3.7/site-packages/PyInstaller/loader/pyimod03_importers.py", line 623, in exec_module
exec(bytecode, module.dict)
File "site-packages/asyncpg/protocol/init.py", line 8, in
File "asyncpg/protocol/protocol.pyx", line 1, in init asyncpg.protocol.protocol
File "asyncpg/pgproto/./buffer.pyx", line 12, in init asyncpg.pgproto.pgproto
ImportError: cannot import name exceptions
[1174] Failed to execute script hello

The fix is as described in this change.

[1st1]

Can you also drop the exceptions.py file then?

[sreenandan]

Sure, I could drop the file as its not consumed anywhere else.

[1st1]

Thanks!

[flaviomilan]

Hi! I got the same problem with exceptions.

This is the stacktrace of error:

File "asyncpg/protocol/__init__.py", line 8, in <module>
File "asyncpg/protocol/protocol.pyx", line 1, in init asyncpg.protocol.protocol
File "asyncpg/pgproto/./buffer.pyx", line 12, in init asyncpg.pgproto.pgproto
ImportError: cannot import name exceptions

I added the asyncpg.pgproto.pgproto when I compiled my application but the error persists.

Could you help me?

Thanks!