Skip to content

./Programs/_freeze_module fails with MSAN: Uninitialized value was created by an allocation of 'stat.i' #91043

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
vstinner opened this issue Mar 1, 2022 · 8 comments
Closed

./Programs/_freeze_module fails with MSAN: Uninitialized value was created by an allocation of 'stat.i' #91043

vstinner opened this issue Mar 1, 2022 · 8 comments
Labels
3.11 only security fixes build The build process and cross-build

Comments

@vstinner
Copy link
Member

vstinner commented Mar 1, 2022

BPO 46887
Nosy @vstinner
PRs
  • bpo-46887: Work around clang MSAN bug on stat()/fstat() #31633
    		•

    Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
closed_at = None
created_at = <Date 2022-03-01.09:38:30.397>
labels = ['build', '3.11']
title = "./Programs/_freeze_module fails with MSAN: Uninitialized value was created by an allocation of 'stat.i'"
updated_at = <Date 2022-03-01.15:01:07.589>
user = 'https://github.com/vstinner'

    bugs.python.org fields:

    activity = <Date 2022-03-01.15:01:07.589>
actor = 'vstinner'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Build']
creation = <Date 2022-03-01.09:38:30.397>
creator = 'vstinner'
dependencies = []
files = []
hgrepos = []
issue_num = 46887
keywords = ['patch']
message_count = 6.0
messages = ['414249', '414264', '414267', '414268', '414269', '414271']
nosy_count = 1.0
nosy_names = ['vstinner']
pr_nums = ['31633']
priority = 'normal'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = None
url = 'https://bugs.python.org/issue46887'
versions = ['Python 3.11']
    @vstinner
    Copy link
    Member Author

    vstinner commented Mar 1, 2022

    "./Programs/_freeze_module zipimport" fails with MSAN:
    ---

    $ make SHELL="bash -x"
(...)
+ ./Programs/_freeze_module zipimport ./Lib/zipimport.py Python/frozen_modules/zipimport.h
==110524==WARNING: MemorySanitizer: use-of-uninitialized-value
==110523==WARNING: MemorySanitizer: use-of-uninitialized-value
==110526==WARNING: MemorySanitizer: use-of-uninitialized-value
==110525==WARNING: MemorySanitizer: use-of-uninitialized-value
    #0 0x4b8cfc in read_text /home/vstinner/python/main/Programs/_freeze_module.c:109:9
    #1 0x4b8cfc in main /home/vstinner/python/main/Programs/_freeze_module.c:224:24
    #2 0x7fe8acb8555f in __libc_start_call_main (/lib64/libc.so.6+0x2d55f)
    #3 0x7fe8acb8560b in __libc_start_main@GLIBC_2.2.5 (/lib64/libc.so.6+0x2d60b)
    #4 0x435f44 in _start (/home/vstinner/python/main/Programs/_freeze_module+0x435f44)

    Uninitialized value was created by an allocation of 'stat.i' in the stack frame of function 'main'
    #0 0x4b72e0 in main /home/vstinner/python/main/Programs/_freeze_module.c:205

    SUMMARY: MemorySanitizer: use-of-uninitialized-value /home/vstinner/python/main/Programs/_freeze_module.c:109:9 in read_text
    Exiting
    make: *** [Makefile:1184: Python/frozen_modules/importlib._bootstrap_external.h] Error 1
    make: *** Waiting for unfinished jobs....
    #0 0x4b8cfc in read_text /home/vstinner/python/main/Programs/_freeze_module.c:109:9
    #1 0x4b8cfc in main /home/vstinner/python/main/Programs/_freeze_module.c:224:24
    #2 0x7f64ecd1355f in __libc_start_call_main (/lib64/libc.so.6+0x2d55f)
    #3 0x7f64ecd1360b in __libc_start_main@GLIBC_2.2.5 (/lib64/libc.so.6+0x2d60b)
    #4 0x435f44 in _start (/home/vstinner/python/main/Programs/_freeze_module+0x435f44)

    Uninitialized value was created by an allocation of 'stat.i' in the stack frame of function 'main'
    #0 0x4b72e0 in main /home/vstinner/python/main/Programs/_freeze_module.c:205

    SUMMARY: MemorySanitizer: use-of-uninitialized-value /home/vstinner/python/main/Programs/_freeze_module.c:109:9 in read_text
    Exiting
    make: *** [Makefile:1176: Python/frozen_modules/getpath.h] Error 1
    #0 0x4b8cfc in read_text /home/vstinner/python/main/Programs/_freeze_module.c:109:9
    #1 0x4b8cfc in main /home/vstinner/python/main/Programs/_freeze_module.c:224:24
    #2 0x7fb2a178d55f in __libc_start_call_main (/lib64/libc.so.6+0x2d55f)
    #3 0x7fb2a178d60b in __libc_start_main@GLIBC_2.2.5 (/lib64/libc.so.6+0x2d60b)
    #4 0x435f44 in _start (/home/vstinner/python/main/Programs/_freeze_module+0x435f44)

    #0 0x4b8cfc in read_text /home/vstinner/python/main/Programs/_freeze_module.c:109:9
#1 0x4b8cfc in main /home/vstinner/python/main/Programs/_freeze_module.c:224:24
#2 0x7f4d471ca55f in __libc_start_call_main (/lib64/libc.so.6+0x2d55f)
#3 0x7f4d471ca60b in __libc_start_main@GLIBC_2.2.5 (/lib64/libc.so.6+0x2d60b)
#4 0x435f44 in _start (/home/vstinner/python/main/Programs/_freeze_module+0x435f44)

    Uninitialized value was created by an allocation of 'stat.i' in the stack frame of function 'main'
    Uninitialized value was created by an allocation of 'stat.i' in the stack frame of function 'main'
    #0 0x4b72e0 in main /home/vstinner/python/main/Programs/_freeze_module.c:205

    #0 0x4b72e0 in main /home/vstinner/python/main/Programs/_freeze_module.c:205

    SUMMARY: MemorySanitizer: use-of-uninitialized-value /home/vstinner/python/main/Programs/_freeze_module.c:109:9 in read_text
    SUMMARY: MemorySanitizer: use-of-uninitialized-value /home/vstinner/python/main/Programs/_freeze_module.c:109:9 in read_text
    Exiting
    Exiting
    make: *** [Makefile:1181: Python/frozen_modules/importlib._bootstrap.h] Error 1
    make: *** [Makefile:1187: Python/frozen_modules/zipimport.h] Error 1
    ---

    Build Python with clang MSAN:
    ---
    ./configure --with-pydebug --with-memory-sanitizer CC=clang LD=clang
    make
    ---

    @vstinner vstinner added 3.11 only security fixes build The build process and cross-build labels Mar 1, 2022
    @vstinner
    Copy link
    Member Author

    vstinner commented Mar 1, 2022

    It looks like a bug in clang MSAN:
    llvm/llvm-project#54131

    @vstinner
    Copy link
    Member Author

    vstinner commented Mar 1, 2022

    Hum, later "import ssl" in setup.py also fails. Simplified code:
    ---

    import _ssl
print(_ssl.txt2obj('1.3.6.1.5.5.7.3.1', name=False)) # server OID

    Error:
    ---

    $ ./python x.py 
Uninitialized bytes in MemcmpInterceptorCommon at offset 0 inside [0x701000000032, 8)
==135651==WARNING: MemorySanitizer: use-of-uninitialized-value
    #0 0x49467c in memcmp (/home/vstinner/python/main/python+0x49467c)
    #1 0x7f5546825adc in OBJ_bsearch_ex_ (/lib64/libcrypto.so.1.1+0x14eadc)
    #2 0x7f5546826ff1 in OBJ_obj2nid (/lib64/libcrypto.so.1.1+0x14fff1)
    #3 0x7f554675b434  (/lib64/libcrypto.so.1.1+0x84434)
    #4 0x7f554675b6af in d2i_ASN1_OBJECT (/lib64/libcrypto.so.1.1+0x846af)
    #5 0x7f5546827702 in OBJ_txt2obj (/lib64/libcrypto.so.1.1+0x150702)
    #6 0x7f5546a78c31 in _ssl_txt2obj_impl /home/vstinner/python/main/Modules/_ssl.c:5276:11
    #7 0x7f5546a78c31 in _ssl_txt2obj /home/vstinner/python/main/Modules/clinic/_ssl.c.h:1216:20
    (...)

    Uninitialized value was created by a heap allocation
    #0 0x466642 in __interceptor_malloc (/home/vstinner/python/main/python+0x466642)
    #1 0x7f55468276b3 in OBJ_txt2obj (/lib64/libcrypto.so.1.1+0x1506b3)

    SUMMARY: MemorySanitizer: use-of-uninitialized-value (/home/vstinner/python/main/python+0x49467c) in memcmp
    Exiting
    ---

    The Python code calls OBJ_txt2obj("1.3.6.1.5.5.7.3.1", 0) in C: the OpenSSL function.

    @vstinner
    Copy link
    Member Author

    vstinner commented Mar 1, 2022

    It looks like a bug in clang MSAN:
    llvm/llvm-project#54131

    I wrote #75814 to work around the false alarm on stat() and fstat().

    @vstinner
    Copy link
    Member Author

    vstinner commented Mar 1, 2022

    The Python code calls OBJ_txt2obj("1.3.6.1.5.5.7.3.1", 0) in C: the OpenSSL function.

    This error is unrelated to Python, but comes from OpenSSL.

    @vstinner
    Copy link
    Member Author

    vstinner commented Mar 1, 2022

    The Python code calls OBJ_txt2obj("1.3.6.1.5.5.7.3.1", 0) in C: the OpenSSL function.

    I reported this issue to OpenSSL: openssl/openssl#17784

    @ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
    @vstinner
    Copy link
    Member Author

    vstinner commented Nov 3, 2022

    It's unclear to me if it's an issue in Python itself, Clang/GCC, or how the glibc was built. For example, openssl/openssl#17784 (comment) was closed OpenSSL must be built with -fsanitize=memory.

    Anyway, I close the issue.

    @vstinner vstinner closed this as completed Nov 3, 2022
    @illia-v illia-v mentioned this issue Mar 7, 2023
    Merged
    @illia-v
    Copy link
    Contributor

    illia-v commented Mar 7, 2023

    It's unclear to me if it's an issue in Python itself, Clang/GCC, or how the glibc was built. For example, openssl/openssl#17784 (comment) was closed OpenSSL must be built with -fsanitize=memory.

    I've recently tried building and compiling CPython with CC=clang ./configure --with-memory-sanitizer && make -j12 against the system OpenSSL.

    After making the #102510 change, a slightly different use-of-uninitialized-value error tracing to /lib/x86_64-linux-gnu/libcrypto.so.3 appeared.

    So, I tried compiling it against OpenSSL built with -fsanitize=memory, and that fixed an error.

    But it looks like there is a sequence of other use-of-uninitialized-value appearing one by one.
    The next one looks to be fixable by this

    diff --git a/Python/dynload_shlib.c b/Python/dynload_shlib.c
index 6761bba457..ac379d5d98 100644
--- a/Python/dynload_shlib.c
+++ b/Python/dynload_shlib.c
@@ -56,7 +56,7 @@ _PyImport_FindSharedFuncptr(const char *prefix,
 {
     dl_funcptr p;
     void *handle;
-    char funcname[258];
+    char funcname[258] = "";
     char pathbuf[260];
     int dlopenflags=0;

    I stopped digging on this one, but noticed the symbol lookup error: /home/user/openssl/lib64/libcrypto.so.3: undefined symbol: __msan_retval_tls warnings:

    Uninitialized bytes in __interceptor_strlen at offset 0 inside [0x701000003040, 1)
==91306==WARNING: MemorySanitizer: use-of-uninitialized-value
/usr/bin/llvm-symbolizer-15: symbol lookup error: /home/user/openssl/lib64/libcrypto.so.3: undefined symbol: __msan_retval_tls
==91306==WARNING: external symbolizer didn't start up correctly!
==91306==WARNING: Can't write to symbolizer at fd 6
/usr/bin/llvm-symbolizer-15: symbol lookup error: /home/user/openssl/lib64/libcrypto.so.3: undefined symbol: __msan_retval_tls
==91306==WARNING: external symbolizer didn't start up correctly!
==91306==WARNING: Can't write to symbolizer at fd 6
/usr/bin/llvm-symbolizer-15: symbol lookup error: /home/user/openssl/lib64/libcrypto.so.3: undefined symbol: __msan_retval_tls
==91306==WARNING: external symbolizer didn't start up correctly!
==91306==WARNING: Can't write to symbolizer at fd 6
/usr/bin/llvm-symbolizer-15: symbol lookup error: /home/user/openssl/lib64/libcrypto.so.3: undefined symbol: __msan_retval_tls
==91306==WARNING: external symbolizer didn't start up correctly!
==91306==WARNING: Can't write to symbolizer at fd 6
/usr/bin/llvm-symbolizer-15: symbol lookup error: /home/user/openssl/lib64/libcrypto.so.3: undefined symbol: __msan_retval_tls
==91306==WARNING: external symbolizer didn't start up correctly!
==91306==WARNING: Failed to use and restart external symbolizer!
    #0 0x7f98fe69104b  (/lib/x86_64-linux-gnu/libreadline.so.8+0x3204b) (BuildId: 990587ed58ff96de9f94756458d4aaa2f51aaa3d)
    #1 0x7f98fe880ab2  (/home/user/cpython/build/lib.linux-x86_64-3.12/readline.cpython-312-x86_64-linux-gnu.so+0x3ab2) (BuildId: a50d39ef2c15d0868c490d8e02f64b74edea405b)
    #2 0x557f32c393af  (/home/user/cpython/python+0x9a03af) (BuildId: 61b467aa3df7c96f1560120596af9896ed67767d)
    #3 0x557f32c36cb8  (/home/user/cpython/python+0x99dcb8) (BuildId: 61b467aa3df7c96f1560120596af9896ed67767d)
    #4 0x557f327b7d68  (/home/user/cpython/python+0x51ed68) (BuildId: 61b467aa3df7c96f1560120596af9896ed67767d)
    #5 0x557f326002e9  (/home/user/cpython/python+0x3672e9) (BuildId: 61b467aa3df7c96f1560120596af9896ed67767d)
    #6 0x557f326012b5  (/home/user/cpython/python+0x3682b5) (BuildId: 61b467aa3df7c96f1560120596af9896ed67767d)
    #7 0x557f326016e3  (/home/user/cpython/python+0x3686e3) (BuildId: 61b467aa3df7c96f1560120596af9896ed67767d)
    #8 0x557f32afd88f  (/home/user/cpython/python+0x86488f) (BuildId: 61b467aa3df7c96f1560120596af9896ed67767d)
    #9 0x557f32ad0cc2  (/home/user/cpython/python+0x837cc2) (BuildId: 61b467aa3df7c96f1560120596af9896ed67767d)
    #10 0x557f32cc4ce3  (/home/user/cpython/python+0xa2bce3) (BuildId: 61b467aa3df7c96f1560120596af9896ed67767d)
    #11 0x557f32cbbefe  (/home/user/cpython/python+0xa22efe) (BuildId: 61b467aa3df7c96f1560120596af9896ed67767d)
    #12 0x557f32cba316  (/home/user/cpython/python+0xa21316) (BuildId: 61b467aa3df7c96f1560120596af9896ed67767d)
    #13 0x557f32d6975b  (/home/user/cpython/python+0xad075b) (BuildId: 61b467aa3df7c96f1560120596af9896ed67767d)
    #14 0x557f32d6b41a  (/home/user/cpython/python+0xad241a) (BuildId: 61b467aa3df7c96f1560120596af9896ed67767d)
    #15 0x557f32d6b5cd  (/home/user/cpython/python+0xad25cd) (BuildId: 61b467aa3df7c96f1560120596af9896ed67767d)
    #16 0x557f324375d8  (/home/user/cpython/python+0x19e5d8) (BuildId: 61b467aa3df7c96f1560120596af9896ed67767d)
    #17 0x7f990342350f  (/lib/x86_64-linux-gnu/libc.so.6+0x2350f) (BuildId: d1704d25fbbb72fa95d517b883131828c0883fe9)
    #18 0x7f99034235c8  (/lib/x86_64-linux-gnu/libc.so.6+0x235c8) (BuildId: d1704d25fbbb72fa95d517b883131828c0883fe9)
    #19 0x557f323b0e54  (/home/user/cpython/python+0x117e54) (BuildId: 61b467aa3df7c96f1560120596af9896ed67767d)

  Uninitialized value was created by a heap allocation
    #0 0x557f323e4f00  (/home/user/cpython/python+0x14bf00) (BuildId: 61b467aa3df7c96f1560120596af9896ed67767d)
    #1 0x7f98fe699ddc  (/lib/x86_64-linux-gnu/libreadline.so.8+0x3addc) (BuildId: 990587ed58ff96de9f94756458d4aaa2f51aaa3d)

SUMMARY: MemorySanitizer: use-of-uninitialized-value (/lib/x86_64-linux-gnu/libreadline.so.8+0x3204b) (BuildId: 990587ed58ff96de9f94756458d4aaa2f51aaa3d)

    @vstinner vstinner mentioned this issue Aug 22, 2023
    Closed
    @vstinner vstinner mentioned this issue Sep 20, 2023
    Closed
    @vstinner vstinner mentioned this issue Mar 16, 2024
    Closed
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Assignees
    No one assigned
    Labels
    3.11 only security fixes build The build process and cross-build
    Projects
    None yet
    Milestone
    No milestone
    Development

    No branches or pull requests
    2 participants
    @vstinner @illia-v