Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

python3.5.0 incompatibility #14

Closed
anthraxx opened this issue Dec 5, 2015 · 1 comment · Fixed by #15
Closed

python3.5.0 incompatibility #14

anthraxx opened this issue Dec 5, 2015 · 1 comment · Fixed by #15

Comments

@anthraxx
Copy link
Contributor

anthraxx commented Dec 5, 2015

Would it be possible to make this library functional for python3?
The python2 tests are running fine, however if I invoke it with python 3.5.0, I get a ValueError:

python3:

# python test.py ../Testing/example_data/2005NISSE.txt ../Testing/example_data/1english-only.txt
tlsh.hash hex1 06D29517F780237185070293B60E36FAB735C0F833D66460688DA22D6756E751B7BAEB
tlsh.hash hex2 E951784702042376169012B1BA5A76EAF36092FC3311A595B4856235278F9F973763EF
tlsh.diff(hex1, hex2) 427
tlsh.diff(hex2, hex1) 427
tlsh.Tlsh hex1 06D29517F780237185070293B60E36FAB735C0F833D66460688DA22D6756E751B7BAEB
tlsh.Tlsh hex2 E951784702042376169012B1BA5A76EAF36092FC3311A595B4856235278F9F973763EF
h1.diff(h2) 427
h2.diff(h1) 427
h1.diff(hex2) 427
h2.diff(hex1) 427
Traceback (most recent call last):
  File "test.py", line 39, in <module>
    h3.fromTlshStr(hex2)
ValueError: argument is not a TLSH hex string
@anthraxx anthraxx changed the title python3 incompatibility python3.5.0 incompatibility Dec 5, 2015
@anthraxx
Copy link
Contributor Author

anthraxx commented Dec 5, 2015

This requires bytes object

if (PyBytes_AsStringAndSize(arg, &str, &len) == -1) {

This create a str object

"Get the computed TLSH as a string object containing only hexadecimal digits."

return Py_BuildValue("s", hash);

and finally this test passes the str object to a function that requires bytes object:

h3.fromTlshStr(hex2)

calladoum-elastic added a commit to calladoum-elastic/tlsh that referenced this issue Mar 27, 2024
There is an off-by-one in `to_hex` (null byte OOB write) resulting in the module crash when running for a while.

Found by ASAN

```text
==10996==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x12ab094a0928 at pc 0x7ff8602c914d bp 0x00e354bebb60 sp 0x00e354bebb68
WRITE of size 1 at 0x12ab094a0928 thread T0
    #0 0x7ff8602c914c in to_hex C:\git\tlsh\src\tlsh_util.cpp:2675
    #1 0x7ff8602b73ce in TlshImpl::hash C:\git\tlsh\src\tlsh_impl.cpp:1042
    #2 0x7ff8602b6ba9 in TlshImpl::hash C:\git\tlsh\src\tlsh_impl.cpp:1065
    #3 0x7ff8602c7c4b in Tlsh::getHash C:\git\tlsh\src\tlsh.cpp:152
    #4 0x7ff8602828c6 in `nanobind::cpp_function_def<std::basic_string<char,std::char_traits<char>,std::allocator<char> > const ,Tlsh,unsigned char,nanobind::scope,nanobind::name,nanobind::is_method>'::`2'::<lambda_1>::operator() C:\Users\chris\AppData\Local\Temp\pip-build-env-csp14ltk\overlay\Lib\site-packages\nanobind\include\nanobind\nb_func.h:248
    #5 0x7ff860283b22 in `nanobind::detail::func_create<0,1,`nanobind::cpp_function_def<std::basic_string<char,std::char_traits<char>,std::allocator<char> > const ,Tlsh,unsigned char,nanobind::scope,nanobind::name,nanobind::is_method>'::`2'::<lambda_1>,std::basic_string<char,std::char_traits<char>,std::allocator<char> > const ,Tlsh const *,unsigned char,0,1,nanobind::scope,nanobind::name,nanobind::is_method>'::`2'::<lambda_1>::operator() C:\Users\chris\AppData\Local\Temp\pip-build-env-csp14ltk\overlay\Lib\site-packages\nanobind\include\nanobind\nb_func.h:153
    #6 0x7ff860272654 in `nanobind::detail::func_create<0,1,`nanobind::cpp_function_def<std::basic_string<char,std::char_traits<char>,std::allocator<char> > const ,Tlsh,unsigned char,nanobind::scope,nanobind::name,nanobind::is_method>'::`2'::<lambda_1>,std::basic_string<char,std::char_traits<char>,std::allocator<char> > const ,Tlsh const *,unsigned char,0,1,nanobind::scope,nanobind::name,nanobind::is_method>'::`2'::<lambda_1>::<lambda_invoker_cdecl> C:\Users\chris\AppData\Local\Temp\pip-build-env-csp14ltk\overlay\Lib\site-packages\nanobind\include\nanobind\nb_func.h:163
    trendmicro#7 0x7ff86029c360 in nanobind::detail::nb_func_vectorcall_simple C:\Users\chris\AppData\Local\Temp\pip-build-env-csp14ltk\overlay\Lib\site-packages\nanobind\src\nb_func.cpp:758
    trendmicro#8 0x7ff871a495c6 in PyObject_Vectorcall+0x5e6 (C:\python311\python311.dll+0x1800395c6)
    trendmicro#9 0x7ff871a4a92a in PyEval_EvalFrameDefault+0x79a (C:\python311\python311.dll+0x18003a92a)
    trendmicro#10 0x7ff871a42263 in PyFunction_Vectorcall+0x1a3 (C:\python311\python311.dll+0x180032263)
    trendmicro#11 0x7ff871a66c96 in PyObject_CallObject+0x406 (C:\python311\python311.dll+0x180056c96)
    trendmicro#12 0x7ff871a4f524 in PyEval_EvalFrameDefault+0x5394 (C:\python311\python311.dll+0x18003f524)
    trendmicro#13 0x7ff871a42263 in PyFunction_Vectorcall+0x1a3 (C:\python311\python311.dll+0x180032263)
    trendmicro#14 0x7ff871a66c96 in PyObject_CallObject+0x406 (C:\python311\python311.dll+0x180056c96)
    trendmicro#15 0x7ff871a4f524 in PyEval_EvalFrameDefault+0x5394 (C:\python311\python311.dll+0x18003f524)
    trendmicro#16 0x7ff871a42263 in PyFunction_Vectorcall+0x1a3 (C:\python311\python311.dll+0x180032263)
    trendmicro#17 0x7ff871a7ac01 in PyObject_FastCallDictTstate+0xc1 (C:\python311\python311.dll+0x18006ac01)
    trendmicro#18 0x7ff871a79382 in PyObject_Call_Prepend+0x7e (C:\python311\python311.dll+0x180069382)
    trendmicro#19 0x7ff871a792af in PyDict_GetItemStringWithError+0x19b (C:\python311\python311.dll+0x1800692af)
    trendmicro#20 0x7ff871a582b2 in PyObject_MakeTpCall+0x3a2 (C:\python311\python311.dll+0x1800482b2)
    trendmicro#21 0x7ff871a4926e in PyObject_Vectorcall+0x28e (C:\python311\python311.dll+0x18003926e)
    trendmicro#22 0x7ff871a4a92a in PyEval_EvalFrameDefault+0x79a (C:\python311\python311.dll+0x18003a92a)
    trendmicro#23 0x7ff871a42263 in PyFunction_Vectorcall+0x1a3 (C:\python311\python311.dll+0x180032263)
    trendmicro#24 0x7ff871a66c96 in PyObject_CallObject+0x406 (C:\python311\python311.dll+0x180056c96)
    trendmicro#25 0x7ff871a4f524 in PyEval_EvalFrameDefault+0x5394 (C:\python311\python311.dll+0x18003f524)
    trendmicro#26 0x7ff871a42263 in PyFunction_Vectorcall+0x1a3 (C:\python311\python311.dll+0x180032263)
    trendmicro#27 0x7ff871a7ac01 in PyObject_FastCallDictTstate+0xc1 (C:\python311\python311.dll+0x18006ac01)
    trendmicro#28 0x7ff871a79382 in PyObject_Call_Prepend+0x7e (C:\python311\python311.dll+0x180069382)
    trendmicro#29 0x7ff871a792af in PyDict_GetItemStringWithError+0x19b (C:\python311\python311.dll+0x1800692af)
    trendmicro#30 0x7ff871a66cc0 in PyObject_CallObject+0x430 (C:\python311\python311.dll+0x180056cc0)
    trendmicro#31 0x7ff871a4f524 in PyEval_EvalFrameDefault+0x5394 (C:\python311\python311.dll+0x18003f524)
    trendmicro#32 0x7ff871a42263 in PyFunction_Vectorcall+0x1a3 (C:\python311\python311.dll+0x180032263)
    trendmicro#33 0x7ff871a66c96 in PyObject_CallObject+0x406 (C:\python311\python311.dll+0x180056c96)
    trendmicro#34 0x7ff871a4f524 in PyEval_EvalFrameDefault+0x5394 (C:\python311\python311.dll+0x18003f524)
    trendmicro#35 0x7ff871a42263 in PyFunction_Vectorcall+0x1a3 (C:\python311\python311.dll+0x180032263)
    trendmicro#36 0x7ff871a66c96 in PyObject_CallObject+0x406 (C:\python311\python311.dll+0x180056c96)
    trendmicro#37 0x7ff871a4f524 in PyEval_EvalFrameDefault+0x5394 (C:\python311\python311.dll+0x18003f524)
    trendmicro#38 0x7ff871a42263 in PyFunction_Vectorcall+0x1a3 (C:\python311\python311.dll+0x180032263)
    trendmicro#39 0x7ff871a7ac01 in PyObject_FastCallDictTstate+0xc1 (C:\python311\python311.dll+0x18006ac01)
    trendmicro#40 0x7ff871a79382 in PyObject_Call_Prepend+0x7e (C:\python311\python311.dll+0x180069382)
    trendmicro#41 0x7ff871a792af in PyDict_GetItemStringWithError+0x19b (C:\python311\python311.dll+0x1800692af)
    trendmicro#42 0x7ff871a582b2 in PyObject_MakeTpCall+0x3a2 (C:\python311\python311.dll+0x1800482b2)
    trendmicro#43 0x7ff871a4926e in PyObject_Vectorcall+0x28e (C:\python311\python311.dll+0x18003926e)
    trendmicro#44 0x7ff871a4a92a in PyEval_EvalFrameDefault+0x79a (C:\python311\python311.dll+0x18003a92a)
    trendmicro#45 0x7ff871a42263 in PyFunction_Vectorcall+0x1a3 (C:\python311\python311.dll+0x180032263)
    trendmicro#46 0x7ff871a66c96 in PyObject_CallObject+0x406 (C:\python311\python311.dll+0x180056c96)
    trendmicro#47 0x7ff871a4f524 in PyEval_EvalFrameDefault+0x5394 (C:\python311\python311.dll+0x18003f524)
    trendmicro#48 0x7ff871a42263 in PyFunction_Vectorcall+0x1a3 (C:\python311\python311.dll+0x180032263)
    trendmicro#49 0x7ff871a7ac01 in PyObject_FastCallDictTstate+0xc1 (C:\python311\python311.dll+0x18006ac01)
    trendmicro#50 0x7ff871a79382 in PyObject_Call_Prepend+0x7e (C:\python311\python311.dll+0x180069382)
    trendmicro#51 0x7ff871a792af in PyDict_GetItemStringWithError+0x19b (C:\python311\python311.dll+0x1800692af)
    trendmicro#52 0x7ff871a582b2 in PyObject_MakeTpCall+0x3a2 (C:\python311\python311.dll+0x1800482b2)
    trendmicro#53 0x7ff871a4926e in PyObject_Vectorcall+0x28e (C:\python311\python311.dll+0x18003926e)
    trendmicro#54 0x7ff871a4a92a in PyEval_EvalFrameDefault+0x79a (C:\python311\python311.dll+0x18003a92a)
    trendmicro#55 0x7ff871a42263 in PyFunction_Vectorcall+0x1a3 (C:\python311\python311.dll+0x180032263)
    trendmicro#56 0x7ff871a66c96 in PyObject_CallObject+0x406 (C:\python311\python311.dll+0x180056c96)
    trendmicro#57 0x7ff871a4f524 in PyEval_EvalFrameDefault+0x5394 (C:\python311\python311.dll+0x18003f524)
    trendmicro#58 0x7ff871a42263 in PyFunction_Vectorcall+0x1a3 (C:\python311\python311.dll+0x180032263)
    trendmicro#59 0x7ff871a7ac01 in PyObject_FastCallDictTstate+0xc1 (C:\python311\python311.dll+0x18006ac01)
    trendmicro#60 0x7ff871a79382 in PyObject_Call_Prepend+0x7e (C:\python311\python311.dll+0x180069382)
    trendmicro#61 0x7ff871a792af in PyDict_GetItemStringWithError+0x19b (C:\python311\python311.dll+0x1800692af)
    trendmicro#62 0x7ff871a582b2 in PyObject_MakeTpCall+0x3a2 (C:\python311\python311.dll+0x1800482b2)
    trendmicro#63 0x7ff871a4926e in PyObject_Vectorcall+0x28e (C:\python311\python311.dll+0x18003926e)
    trendmicro#64 0x7ff871a4a92a in PyEval_EvalFrameDefault+0x79a (C:\python311\python311.dll+0x18003a92a)
    trendmicro#65 0x7ff871a7a4f6 in PyMapping_Check+0x1ea (C:\python311\python311.dll+0x18006a4f6)
    trendmicro#66 0x7ff871a78a3a in PyEval_EvalCode+0x96 (C:\python311\python311.dll+0x180068a3a)
    trendmicro#67 0x7ff871afc48f in Py_SourceAsString+0x3db (C:\python311\python311.dll+0x1800ec48f)
    trendmicro#68 0x7ff871afc35f in Py_SourceAsString+0x2ab (C:\python311\python311.dll+0x1800ec35f)
    trendmicro#69 0x7ff871a5fccf in PyUnicode_RichCompare+0x1eef (C:\python311\python311.dll+0x18004fccf)
    trendmicro#70 0x7ff871a495c6 in PyObject_Vectorcall+0x5e6 (C:\python311\python311.dll+0x1800395c6)
    trendmicro#71 0x7ff871a4a92a in PyEval_EvalFrameDefault+0x79a (C:\python311\python311.dll+0x18003a92a)
    trendmicro#72 0x7ff871a42263 in PyFunction_Vectorcall+0x1a3 (C:\python311\python311.dll+0x180032263)
    trendmicro#73 0x7ff871a67016 in PyObject_Call+0x5a (C:\python311\python311.dll+0x180057016)
    trendmicro#74 0x7ff871a9e217 in PyObject_GetDictPtr+0x133 (C:\python311\python311.dll+0x18008e217)
    trendmicro#75 0x7ff871b0355a in Py_EncodeUTF8Ex+0x6f6 (C:\python311\python311.dll+0x1800f355a)
    trendmicro#76 0x7ff871b037c0 in Py_RunMain+0x14 (C:\python311\python311.dll+0x1800f37c0)
    trendmicro#77 0x7ff871b86cac in Py_Main+0x24 (C:\python311\python311.dll+0x180176cac)
    trendmicro#78 0x7ff7d9ad122f  (C:\python311\python.exe+0x14000122f)
    trendmicro#79 0x7ff8b8d67343 in BaseThreadInitThunk+0x13 (C:\WINDOWS\System32\KERNEL32.DLL+0x180017343)
    trendmicro#80 0x7ff8bace26b0 in RtlUserThreadStart+0x20 (C:\WINDOWS\SYSTEM32\ntdll.dll+0x1800526b0)

0x12ab094a0928 is located 0 bytes to the right of 72-byte region [0x12ab094a08e0,0x12ab094a0928)
allocated by thread T0 here:
    #0 0x7ff8602c9825 in operator new D:\a\_work\1\s\src\vctools\asan\llvm\compiler-rt\lib\asan\asan_win_new_scalar_thunk.cpp:40
    #1 0x7ff8602871a2 in std::_Default_allocate_traits::_Allocate C:\Program Files\Microsoft Visual Studio\2022\Professional\VC\Tools\MSVC\14.39.33519\include\xmemory:90
    #2 0x7ff8602741bd in std::_Allocate<16,std::_Default_allocate_traits,0> C:\Program Files\Microsoft Visual Studio\2022\Professional\VC\Tools\MSVC\14.39.33519\include\xmemory:248
    #3 0x7ff86028a12f in std::allocator<unsigned char>::allocate C:\Program Files\Microsoft Visual Studio\2022\Professional\VC\Tools\MSVC\14.39.33519\include\xmemory:982
    #4 0x7ff8602742c5 in std::_Allocate_at_least_helper<std::allocator<unsigned char> > C:\Program Files\Microsoft Visual Studio\2022\Professional\VC\Tools\MSVC\14.39.33519\include\xmemory:2211      
    #5 0x7ff8602c4f2f in std::vector<unsigned char,std::allocator<unsigned char> >::_Resize_reallocate<std::_Value_init_tag> C:\Program Files\Microsoft Visual Studio\2022\Professional\VC\Tools\MSVC\14.39.33519\include\vector:1522
    #6 0x7ff8602c4b96 in std::vector<unsigned char,std::allocator<unsigned char> >::_Resize<std::_Value_init_tag> C:\Program Files\Microsoft Visual Studio\2022\Professional\VC\Tools\MSVC\14.39.33519\include\vector:1568
    trendmicro#7 0x7ff8602c775e in std::vector<unsigned char,std::allocator<unsigned char> >::resize C:\Program Files\Microsoft Visual Studio\2022\Professional\VC\Tools\MSVC\14.39.33519\include\vector:1590    
    trendmicro#8 0x7ff8602b6b6c in TlshImpl::hash C:\git\tlsh\src\tlsh_impl.cpp:1062
    trendmicro#9 0x7ff8602c7c4b in Tlsh::getHash C:\git\tlsh\src\tlsh.cpp:152
    trendmicro#10 0x7ff8602828c6 in `nanobind::cpp_function_def<std::basic_string<char,std::char_traits<char>,std::allocator<char> > const ,Tlsh,unsigned char,nanobind::scope,nanobind::name,nanobind::is_method>'::`2'::<lambda_1>::operator() C:\Users\chris\AppData\Local\Temp\pip-build-env-csp14ltk\overlay\Lib\site-packages\nanobind\include\nanobind\nb_func.h:248
    trendmicro#11 0x7ff860283b22 in `nanobind::detail::func_create<0,1,`nanobind::cpp_function_def<std::basic_string<char,std::char_traits<char>,std::allocator<char> > const ,Tlsh,unsigned char,nanobind::scope,nanobind::name,nanobind::is_method>'::`2'::<lambda_1>,std::basic_string<char,std::char_traits<char>,std::allocator<char> > const ,Tlsh const *,unsigned char,0,1,nanobind::scope,nanobind::name,nanobind::is_method>'::`2'::<lambda_1>::operator() C:\Users\chris\AppData\Local\Temp\pip-build-env-csp14ltk\overlay\Lib\site-packages\nanobind\include\nanobind\nb_func.h:153
    trendmicro#12 0x7ff860272654 in `nanobind::detail::func_create<0,1,`nanobind::cpp_function_def<std::basic_string<char,std::char_traits<char>,std::allocator<char> > const ,Tlsh,unsigned char,nanobind::scope,nanobind::name,nanobind::is_method>'::`2'::<lambda_1>,std::basic_string<char,std::char_traits<char>,std::allocator<char> > const ,Tlsh const *,unsigned char,0,1,nanobind::scope,nanobind::name,nanobind::is_method>'::`2'::<lambda_1>::<lambda_invoker_cdecl> C:\Users\chris\AppData\Local\Temp\pip-build-env-csp14ltk\overlay\Lib\site-packages\nanobind\include\nanobind\nb_func.h:163
    trendmicro#13 0x7ff86029c360 in nanobind::detail::nb_func_vectorcall_simple C:\Users\chris\AppData\Local\Temp\pip-build-env-csp14ltk\overlay\Lib\site-packages\nanobind\src\nb_func.cpp:758
    trendmicro#14 0x7ff871a495c6 in PyObject_Vectorcall+0x5e6 (C:\python311\python311.dll+0x1800395c6)
    trendmicro#15 0x7ff871a4a92a in PyEval_EvalFrameDefault+0x79a (C:\python311\python311.dll+0x18003a92a)
    trendmicro#16 0x7ff871a42263 in PyFunction_Vectorcall+0x1a3 (C:\python311\python311.dll+0x180032263)
    trendmicro#17 0x7ff871a66c96 in PyObject_CallObject+0x406 (C:\python311\python311.dll+0x180056c96)
    trendmicro#18 0x7ff871a4f524 in PyEval_EvalFrameDefault+0x5394 (C:\python311\python311.dll+0x18003f524)
    trendmicro#19 0x7ff871a42263 in PyFunction_Vectorcall+0x1a3 (C:\python311\python311.dll+0x180032263)
    trendmicro#20 0x7ff871a66c96 in PyObject_CallObject+0x406 (C:\python311\python311.dll+0x180056c96)
    trendmicro#21 0x7ff871a4f524 in PyEval_EvalFrameDefault+0x5394 (C:\python311\python311.dll+0x18003f524)
    trendmicro#22 0x7ff871a42263 in PyFunction_Vectorcall+0x1a3 (C:\python311\python311.dll+0x180032263)
    trendmicro#23 0x7ff871a7ac01 in PyObject_FastCallDictTstate+0xc1 (C:\python311\python311.dll+0x18006ac01)
    trendmicro#24 0x7ff871a79382 in PyObject_Call_Prepend+0x7e (C:\python311\python311.dll+0x180069382)
    trendmicro#25 0x7ff871a792af in PyDict_GetItemStringWithError+0x19b (C:\python311\python311.dll+0x1800692af)
    trendmicro#26 0x7ff871a582b2 in PyObject_MakeTpCall+0x3a2 (C:\python311\python311.dll+0x1800482b2)
    trendmicro#27 0x7ff871a4926e in PyObject_Vectorcall+0x28e (C:\python311\python311.dll+0x18003926e)
    trendmicro#28 0x7ff871a4a92a in PyEval_EvalFrameDefault+0x79a (C:\python311\python311.dll+0x18003a92a)

SUMMARY: AddressSanitizer: heap-buffer-overflow C:\git\tlsh\src\tlsh_util.cpp:2675 in to_hex
Shadow bytes around the buggy address:
  0x04f26a7140d0: fd fa fa fa fa fa 00 00 00 00 00 00 00 00 06 fa
  0x04f26a7140e0: fa fa fa fa fd fd fd fd fd fd fd fd fd fa fa fa
  0x04f26a7140f0: fa fa fd fd fd fd fd fd fd fd fd fd fa fa fa fa
  0x04f26a714100: fd fd fd fd fd fd fd fd fd fa fa fa fa fa fd fd
  0x04f26a714110: fd fd fd fd fd fd fd fd fa fa fa fa 00 00 00 00
=>0x04f26a714120: 00 00 00 00 00[fa]fa fa fa fa fd fd fd fd fd fd
  0x04f26a714130: fd fd fd fa fa fa fa fa fd fd fd fd fd fd fd fd
  0x04f26a714140: fd fa fa fa fa fa fd fd fd fd fd fd fd fd fd fd
  0x04f26a714150: fa fa fa fa fd fd fd fd fd fd fd fd fd fa fa fa
  0x04f26a714160: fa fa fd fd fd fd fd fd fd fd fd fd fa fa fa fa
  0x04f26a714170: fd fd fd fd fd fd fd fd fd fa fa fa fa fa fd fd
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
==10996==ABORTING
```
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant