what if I want to calculate tlsh value for short strings #56
Comments
|
You can calculate the hash for strings as short as 50 bytes if you use the force option The force option has confused many people |
|
resolved in version 3.17.0 |
calladoum-elastic
added a commit
to calladoum-elastic/tlsh
that referenced
this issue
Mar 27, 2024
There is an off-by-one in `to_hex` (null byte OOB write) resulting in the module crash when running for a while.
Found by ASAN
```text
==10996==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x12ab094a0928 at pc 0x7ff8602c914d bp 0x00e354bebb60 sp 0x00e354bebb68
WRITE of size 1 at 0x12ab094a0928 thread T0
#0 0x7ff8602c914c in to_hex C:\git\tlsh\src\tlsh_util.cpp:2675
#1 0x7ff8602b73ce in TlshImpl::hash C:\git\tlsh\src\tlsh_impl.cpp:1042
#2 0x7ff8602b6ba9 in TlshImpl::hash C:\git\tlsh\src\tlsh_impl.cpp:1065
#3 0x7ff8602c7c4b in Tlsh::getHash C:\git\tlsh\src\tlsh.cpp:152
#4 0x7ff8602828c6 in `nanobind::cpp_function_def<std::basic_string<char,std::char_traits<char>,std::allocator<char> > const ,Tlsh,unsigned char,nanobind::scope,nanobind::name,nanobind::is_method>'::`2'::<lambda_1>::operator() C:\Users\chris\AppData\Local\Temp\pip-build-env-csp14ltk\overlay\Lib\site-packages\nanobind\include\nanobind\nb_func.h:248
#5 0x7ff860283b22 in `nanobind::detail::func_create<0,1,`nanobind::cpp_function_def<std::basic_string<char,std::char_traits<char>,std::allocator<char> > const ,Tlsh,unsigned char,nanobind::scope,nanobind::name,nanobind::is_method>'::`2'::<lambda_1>,std::basic_string<char,std::char_traits<char>,std::allocator<char> > const ,Tlsh const *,unsigned char,0,1,nanobind::scope,nanobind::name,nanobind::is_method>'::`2'::<lambda_1>::operator() C:\Users\chris\AppData\Local\Temp\pip-build-env-csp14ltk\overlay\Lib\site-packages\nanobind\include\nanobind\nb_func.h:153
#6 0x7ff860272654 in `nanobind::detail::func_create<0,1,`nanobind::cpp_function_def<std::basic_string<char,std::char_traits<char>,std::allocator<char> > const ,Tlsh,unsigned char,nanobind::scope,nanobind::name,nanobind::is_method>'::`2'::<lambda_1>,std::basic_string<char,std::char_traits<char>,std::allocator<char> > const ,Tlsh const *,unsigned char,0,1,nanobind::scope,nanobind::name,nanobind::is_method>'::`2'::<lambda_1>::<lambda_invoker_cdecl> C:\Users\chris\AppData\Local\Temp\pip-build-env-csp14ltk\overlay\Lib\site-packages\nanobind\include\nanobind\nb_func.h:163
trendmicro#7 0x7ff86029c360 in nanobind::detail::nb_func_vectorcall_simple C:\Users\chris\AppData\Local\Temp\pip-build-env-csp14ltk\overlay\Lib\site-packages\nanobind\src\nb_func.cpp:758
trendmicro#8 0x7ff871a495c6 in PyObject_Vectorcall+0x5e6 (C:\python311\python311.dll+0x1800395c6)
trendmicro#9 0x7ff871a4a92a in PyEval_EvalFrameDefault+0x79a (C:\python311\python311.dll+0x18003a92a)
trendmicro#10 0x7ff871a42263 in PyFunction_Vectorcall+0x1a3 (C:\python311\python311.dll+0x180032263)
trendmicro#11 0x7ff871a66c96 in PyObject_CallObject+0x406 (C:\python311\python311.dll+0x180056c96)
trendmicro#12 0x7ff871a4f524 in PyEval_EvalFrameDefault+0x5394 (C:\python311\python311.dll+0x18003f524)
trendmicro#13 0x7ff871a42263 in PyFunction_Vectorcall+0x1a3 (C:\python311\python311.dll+0x180032263)
trendmicro#14 0x7ff871a66c96 in PyObject_CallObject+0x406 (C:\python311\python311.dll+0x180056c96)
trendmicro#15 0x7ff871a4f524 in PyEval_EvalFrameDefault+0x5394 (C:\python311\python311.dll+0x18003f524)
trendmicro#16 0x7ff871a42263 in PyFunction_Vectorcall+0x1a3 (C:\python311\python311.dll+0x180032263)
trendmicro#17 0x7ff871a7ac01 in PyObject_FastCallDictTstate+0xc1 (C:\python311\python311.dll+0x18006ac01)
trendmicro#18 0x7ff871a79382 in PyObject_Call_Prepend+0x7e (C:\python311\python311.dll+0x180069382)
trendmicro#19 0x7ff871a792af in PyDict_GetItemStringWithError+0x19b (C:\python311\python311.dll+0x1800692af)
trendmicro#20 0x7ff871a582b2 in PyObject_MakeTpCall+0x3a2 (C:\python311\python311.dll+0x1800482b2)
trendmicro#21 0x7ff871a4926e in PyObject_Vectorcall+0x28e (C:\python311\python311.dll+0x18003926e)
trendmicro#22 0x7ff871a4a92a in PyEval_EvalFrameDefault+0x79a (C:\python311\python311.dll+0x18003a92a)
trendmicro#23 0x7ff871a42263 in PyFunction_Vectorcall+0x1a3 (C:\python311\python311.dll+0x180032263)
trendmicro#24 0x7ff871a66c96 in PyObject_CallObject+0x406 (C:\python311\python311.dll+0x180056c96)
trendmicro#25 0x7ff871a4f524 in PyEval_EvalFrameDefault+0x5394 (C:\python311\python311.dll+0x18003f524)
trendmicro#26 0x7ff871a42263 in PyFunction_Vectorcall+0x1a3 (C:\python311\python311.dll+0x180032263)
trendmicro#27 0x7ff871a7ac01 in PyObject_FastCallDictTstate+0xc1 (C:\python311\python311.dll+0x18006ac01)
trendmicro#28 0x7ff871a79382 in PyObject_Call_Prepend+0x7e (C:\python311\python311.dll+0x180069382)
trendmicro#29 0x7ff871a792af in PyDict_GetItemStringWithError+0x19b (C:\python311\python311.dll+0x1800692af)
trendmicro#30 0x7ff871a66cc0 in PyObject_CallObject+0x430 (C:\python311\python311.dll+0x180056cc0)
trendmicro#31 0x7ff871a4f524 in PyEval_EvalFrameDefault+0x5394 (C:\python311\python311.dll+0x18003f524)
trendmicro#32 0x7ff871a42263 in PyFunction_Vectorcall+0x1a3 (C:\python311\python311.dll+0x180032263)
trendmicro#33 0x7ff871a66c96 in PyObject_CallObject+0x406 (C:\python311\python311.dll+0x180056c96)
trendmicro#34 0x7ff871a4f524 in PyEval_EvalFrameDefault+0x5394 (C:\python311\python311.dll+0x18003f524)
trendmicro#35 0x7ff871a42263 in PyFunction_Vectorcall+0x1a3 (C:\python311\python311.dll+0x180032263)
trendmicro#36 0x7ff871a66c96 in PyObject_CallObject+0x406 (C:\python311\python311.dll+0x180056c96)
trendmicro#37 0x7ff871a4f524 in PyEval_EvalFrameDefault+0x5394 (C:\python311\python311.dll+0x18003f524)
trendmicro#38 0x7ff871a42263 in PyFunction_Vectorcall+0x1a3 (C:\python311\python311.dll+0x180032263)
trendmicro#39 0x7ff871a7ac01 in PyObject_FastCallDictTstate+0xc1 (C:\python311\python311.dll+0x18006ac01)
trendmicro#40 0x7ff871a79382 in PyObject_Call_Prepend+0x7e (C:\python311\python311.dll+0x180069382)
trendmicro#41 0x7ff871a792af in PyDict_GetItemStringWithError+0x19b (C:\python311\python311.dll+0x1800692af)
trendmicro#42 0x7ff871a582b2 in PyObject_MakeTpCall+0x3a2 (C:\python311\python311.dll+0x1800482b2)
trendmicro#43 0x7ff871a4926e in PyObject_Vectorcall+0x28e (C:\python311\python311.dll+0x18003926e)
trendmicro#44 0x7ff871a4a92a in PyEval_EvalFrameDefault+0x79a (C:\python311\python311.dll+0x18003a92a)
trendmicro#45 0x7ff871a42263 in PyFunction_Vectorcall+0x1a3 (C:\python311\python311.dll+0x180032263)
trendmicro#46 0x7ff871a66c96 in PyObject_CallObject+0x406 (C:\python311\python311.dll+0x180056c96)
trendmicro#47 0x7ff871a4f524 in PyEval_EvalFrameDefault+0x5394 (C:\python311\python311.dll+0x18003f524)
trendmicro#48 0x7ff871a42263 in PyFunction_Vectorcall+0x1a3 (C:\python311\python311.dll+0x180032263)
trendmicro#49 0x7ff871a7ac01 in PyObject_FastCallDictTstate+0xc1 (C:\python311\python311.dll+0x18006ac01)
trendmicro#50 0x7ff871a79382 in PyObject_Call_Prepend+0x7e (C:\python311\python311.dll+0x180069382)
trendmicro#51 0x7ff871a792af in PyDict_GetItemStringWithError+0x19b (C:\python311\python311.dll+0x1800692af)
trendmicro#52 0x7ff871a582b2 in PyObject_MakeTpCall+0x3a2 (C:\python311\python311.dll+0x1800482b2)
trendmicro#53 0x7ff871a4926e in PyObject_Vectorcall+0x28e (C:\python311\python311.dll+0x18003926e)
trendmicro#54 0x7ff871a4a92a in PyEval_EvalFrameDefault+0x79a (C:\python311\python311.dll+0x18003a92a)
trendmicro#55 0x7ff871a42263 in PyFunction_Vectorcall+0x1a3 (C:\python311\python311.dll+0x180032263)
trendmicro#56 0x7ff871a66c96 in PyObject_CallObject+0x406 (C:\python311\python311.dll+0x180056c96)
trendmicro#57 0x7ff871a4f524 in PyEval_EvalFrameDefault+0x5394 (C:\python311\python311.dll+0x18003f524)
trendmicro#58 0x7ff871a42263 in PyFunction_Vectorcall+0x1a3 (C:\python311\python311.dll+0x180032263)
trendmicro#59 0x7ff871a7ac01 in PyObject_FastCallDictTstate+0xc1 (C:\python311\python311.dll+0x18006ac01)
trendmicro#60 0x7ff871a79382 in PyObject_Call_Prepend+0x7e (C:\python311\python311.dll+0x180069382)
trendmicro#61 0x7ff871a792af in PyDict_GetItemStringWithError+0x19b (C:\python311\python311.dll+0x1800692af)
trendmicro#62 0x7ff871a582b2 in PyObject_MakeTpCall+0x3a2 (C:\python311\python311.dll+0x1800482b2)
trendmicro#63 0x7ff871a4926e in PyObject_Vectorcall+0x28e (C:\python311\python311.dll+0x18003926e)
trendmicro#64 0x7ff871a4a92a in PyEval_EvalFrameDefault+0x79a (C:\python311\python311.dll+0x18003a92a)
trendmicro#65 0x7ff871a7a4f6 in PyMapping_Check+0x1ea (C:\python311\python311.dll+0x18006a4f6)
trendmicro#66 0x7ff871a78a3a in PyEval_EvalCode+0x96 (C:\python311\python311.dll+0x180068a3a)
trendmicro#67 0x7ff871afc48f in Py_SourceAsString+0x3db (C:\python311\python311.dll+0x1800ec48f)
trendmicro#68 0x7ff871afc35f in Py_SourceAsString+0x2ab (C:\python311\python311.dll+0x1800ec35f)
trendmicro#69 0x7ff871a5fccf in PyUnicode_RichCompare+0x1eef (C:\python311\python311.dll+0x18004fccf)
trendmicro#70 0x7ff871a495c6 in PyObject_Vectorcall+0x5e6 (C:\python311\python311.dll+0x1800395c6)
trendmicro#71 0x7ff871a4a92a in PyEval_EvalFrameDefault+0x79a (C:\python311\python311.dll+0x18003a92a)
trendmicro#72 0x7ff871a42263 in PyFunction_Vectorcall+0x1a3 (C:\python311\python311.dll+0x180032263)
trendmicro#73 0x7ff871a67016 in PyObject_Call+0x5a (C:\python311\python311.dll+0x180057016)
trendmicro#74 0x7ff871a9e217 in PyObject_GetDictPtr+0x133 (C:\python311\python311.dll+0x18008e217)
trendmicro#75 0x7ff871b0355a in Py_EncodeUTF8Ex+0x6f6 (C:\python311\python311.dll+0x1800f355a)
trendmicro#76 0x7ff871b037c0 in Py_RunMain+0x14 (C:\python311\python311.dll+0x1800f37c0)
trendmicro#77 0x7ff871b86cac in Py_Main+0x24 (C:\python311\python311.dll+0x180176cac)
trendmicro#78 0x7ff7d9ad122f (C:\python311\python.exe+0x14000122f)
trendmicro#79 0x7ff8b8d67343 in BaseThreadInitThunk+0x13 (C:\WINDOWS\System32\KERNEL32.DLL+0x180017343)
trendmicro#80 0x7ff8bace26b0 in RtlUserThreadStart+0x20 (C:\WINDOWS\SYSTEM32\ntdll.dll+0x1800526b0)
0x12ab094a0928 is located 0 bytes to the right of 72-byte region [0x12ab094a08e0,0x12ab094a0928)
allocated by thread T0 here:
#0 0x7ff8602c9825 in operator new D:\a\_work\1\s\src\vctools\asan\llvm\compiler-rt\lib\asan\asan_win_new_scalar_thunk.cpp:40
#1 0x7ff8602871a2 in std::_Default_allocate_traits::_Allocate C:\Program Files\Microsoft Visual Studio\2022\Professional\VC\Tools\MSVC\14.39.33519\include\xmemory:90
#2 0x7ff8602741bd in std::_Allocate<16,std::_Default_allocate_traits,0> C:\Program Files\Microsoft Visual Studio\2022\Professional\VC\Tools\MSVC\14.39.33519\include\xmemory:248
#3 0x7ff86028a12f in std::allocator<unsigned char>::allocate C:\Program Files\Microsoft Visual Studio\2022\Professional\VC\Tools\MSVC\14.39.33519\include\xmemory:982
#4 0x7ff8602742c5 in std::_Allocate_at_least_helper<std::allocator<unsigned char> > C:\Program Files\Microsoft Visual Studio\2022\Professional\VC\Tools\MSVC\14.39.33519\include\xmemory:2211
#5 0x7ff8602c4f2f in std::vector<unsigned char,std::allocator<unsigned char> >::_Resize_reallocate<std::_Value_init_tag> C:\Program Files\Microsoft Visual Studio\2022\Professional\VC\Tools\MSVC\14.39.33519\include\vector:1522
#6 0x7ff8602c4b96 in std::vector<unsigned char,std::allocator<unsigned char> >::_Resize<std::_Value_init_tag> C:\Program Files\Microsoft Visual Studio\2022\Professional\VC\Tools\MSVC\14.39.33519\include\vector:1568
trendmicro#7 0x7ff8602c775e in std::vector<unsigned char,std::allocator<unsigned char> >::resize C:\Program Files\Microsoft Visual Studio\2022\Professional\VC\Tools\MSVC\14.39.33519\include\vector:1590
trendmicro#8 0x7ff8602b6b6c in TlshImpl::hash C:\git\tlsh\src\tlsh_impl.cpp:1062
trendmicro#9 0x7ff8602c7c4b in Tlsh::getHash C:\git\tlsh\src\tlsh.cpp:152
trendmicro#10 0x7ff8602828c6 in `nanobind::cpp_function_def<std::basic_string<char,std::char_traits<char>,std::allocator<char> > const ,Tlsh,unsigned char,nanobind::scope,nanobind::name,nanobind::is_method>'::`2'::<lambda_1>::operator() C:\Users\chris\AppData\Local\Temp\pip-build-env-csp14ltk\overlay\Lib\site-packages\nanobind\include\nanobind\nb_func.h:248
trendmicro#11 0x7ff860283b22 in `nanobind::detail::func_create<0,1,`nanobind::cpp_function_def<std::basic_string<char,std::char_traits<char>,std::allocator<char> > const ,Tlsh,unsigned char,nanobind::scope,nanobind::name,nanobind::is_method>'::`2'::<lambda_1>,std::basic_string<char,std::char_traits<char>,std::allocator<char> > const ,Tlsh const *,unsigned char,0,1,nanobind::scope,nanobind::name,nanobind::is_method>'::`2'::<lambda_1>::operator() C:\Users\chris\AppData\Local\Temp\pip-build-env-csp14ltk\overlay\Lib\site-packages\nanobind\include\nanobind\nb_func.h:153
trendmicro#12 0x7ff860272654 in `nanobind::detail::func_create<0,1,`nanobind::cpp_function_def<std::basic_string<char,std::char_traits<char>,std::allocator<char> > const ,Tlsh,unsigned char,nanobind::scope,nanobind::name,nanobind::is_method>'::`2'::<lambda_1>,std::basic_string<char,std::char_traits<char>,std::allocator<char> > const ,Tlsh const *,unsigned char,0,1,nanobind::scope,nanobind::name,nanobind::is_method>'::`2'::<lambda_1>::<lambda_invoker_cdecl> C:\Users\chris\AppData\Local\Temp\pip-build-env-csp14ltk\overlay\Lib\site-packages\nanobind\include\nanobind\nb_func.h:163
trendmicro#13 0x7ff86029c360 in nanobind::detail::nb_func_vectorcall_simple C:\Users\chris\AppData\Local\Temp\pip-build-env-csp14ltk\overlay\Lib\site-packages\nanobind\src\nb_func.cpp:758
trendmicro#14 0x7ff871a495c6 in PyObject_Vectorcall+0x5e6 (C:\python311\python311.dll+0x1800395c6)
trendmicro#15 0x7ff871a4a92a in PyEval_EvalFrameDefault+0x79a (C:\python311\python311.dll+0x18003a92a)
trendmicro#16 0x7ff871a42263 in PyFunction_Vectorcall+0x1a3 (C:\python311\python311.dll+0x180032263)
trendmicro#17 0x7ff871a66c96 in PyObject_CallObject+0x406 (C:\python311\python311.dll+0x180056c96)
trendmicro#18 0x7ff871a4f524 in PyEval_EvalFrameDefault+0x5394 (C:\python311\python311.dll+0x18003f524)
trendmicro#19 0x7ff871a42263 in PyFunction_Vectorcall+0x1a3 (C:\python311\python311.dll+0x180032263)
trendmicro#20 0x7ff871a66c96 in PyObject_CallObject+0x406 (C:\python311\python311.dll+0x180056c96)
trendmicro#21 0x7ff871a4f524 in PyEval_EvalFrameDefault+0x5394 (C:\python311\python311.dll+0x18003f524)
trendmicro#22 0x7ff871a42263 in PyFunction_Vectorcall+0x1a3 (C:\python311\python311.dll+0x180032263)
trendmicro#23 0x7ff871a7ac01 in PyObject_FastCallDictTstate+0xc1 (C:\python311\python311.dll+0x18006ac01)
trendmicro#24 0x7ff871a79382 in PyObject_Call_Prepend+0x7e (C:\python311\python311.dll+0x180069382)
trendmicro#25 0x7ff871a792af in PyDict_GetItemStringWithError+0x19b (C:\python311\python311.dll+0x1800692af)
trendmicro#26 0x7ff871a582b2 in PyObject_MakeTpCall+0x3a2 (C:\python311\python311.dll+0x1800482b2)
trendmicro#27 0x7ff871a4926e in PyObject_Vectorcall+0x28e (C:\python311\python311.dll+0x18003926e)
trendmicro#28 0x7ff871a4a92a in PyEval_EvalFrameDefault+0x79a (C:\python311\python311.dll+0x18003a92a)
SUMMARY: AddressSanitizer: heap-buffer-overflow C:\git\tlsh\src\tlsh_util.cpp:2675 in to_hex
Shadow bytes around the buggy address:
0x04f26a7140d0: fd fa fa fa fa fa 00 00 00 00 00 00 00 00 06 fa
0x04f26a7140e0: fa fa fa fa fd fd fd fd fd fd fd fd fd fa fa fa
0x04f26a7140f0: fa fa fd fd fd fd fd fd fd fd fd fd fa fa fa fa
0x04f26a714100: fd fd fd fd fd fd fd fd fd fa fa fa fa fa fd fd
0x04f26a714110: fd fd fd fd fd fd fd fd fa fa fa fa 00 00 00 00
=>0x04f26a714120: 00 00 00 00 00[fa]fa fa fa fa fd fd fd fd fd fd
0x04f26a714130: fd fd fd fa fa fa fa fa fd fd fd fd fd fd fd fd
0x04f26a714140: fd fa fa fa fa fa fd fd fd fd fd fd fd fd fd fd
0x04f26a714150: fa fa fa fa fd fd fd fd fd fd fd fd fd fa fa fa
0x04f26a714160: fa fa fd fd fd fd fd fd fd fd fd fd fa fa fa fa
0x04f26a714170: fd fd fd fd fd fd fd fd fd fa fa fa fa fa fd fd
Shadow byte legend (one shadow byte represents 8 application bytes):
Addressable: 00
Partially addressable: 01 02 03 04 05 06 07
Heap left redzone: fa
Freed heap region: fd
Stack left redzone: f1
Stack mid redzone: f2
Stack right redzone: f3
Stack after return: f5
Stack use after scope: f8
Global redzone: f9
Global init order: f6
Poisoned by user: f7
Container overflow: fc
Array cookie: ac
Intra object redzone: bb
ASan internal: fe
Left alloca redzone: ca
Right alloca redzone: cb
==10996==ABORTING
```
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
I am intersted in calculating the hash value of short strings that are less that 256 bytes or 50 characters, in order to compare between short words. How can I do that ?
The text was updated successfully, but these errors were encountered: