-
Notifications
You must be signed in to change notification settings - Fork 29
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
shrink_to_fit
method for the bucket backend is unsound
#46
Comments
Hey, sorry for not responding for a long time. What is your opinion? Do you see a valid use case for keeping |
As the description says, this backend is very useful when you need to intern a lot of static strings, which is very common to do on interpreters and compilers, so I definitely see a valid use case for it :) |
Are we talking about >(10^3) static strings or just like <(10^2)? Because I guess it won't matter then. Have you got benchmarks? |
Hmm, probably less than 100, but that would depend on the compiler, I think.
No benchmarks, unfortunately. However, that backend has the advantage of consuming very little memory when the majority of interned strings are static, which is always very appreciated |
Sorry! I forgot to open an issue when I talked about this problem. Better late than never I guess haha
The
StringInterner::shrink_to_fit
method causes undefined behaviour when using the bucket backend.A simple test demonstrates this problem:
This fails with the following message:
Additionally, running
cargo +nightly miri test
errors with:I'll try to work on a fix shortly.
Explanation (taken from #42 (comment))
Explanation
The definition of the
BucketBackend
is as follows:string-interner/src/backend/bucket/mod.rs
Lines 55 to 60 in d552e76
This backend works by creating several
FixedString
buffers of a fixed size, storing them in aVec<String>
and creatingNonNull<str>
references (abstracted byInternedString
) to each of them in order to obtain the internedstr
s.However,
FixedString
must always ensure that it won't be moved to another memory location, otherwise the previously createdNonNull<str>
references will point to unallocated memory (use after free).Unfortunately, when we call the
shrink_to_fit
method from the backend, it internally calls theshrink_to_fit
method fromFixedString
:string-interner/src/backend/bucket/fixed_str.rs
Lines 60 to 63 in d552e76
And calling
shrink_to_fit
on aString
CAN move its contents to another memory location to reduce its allocated space. From the definition ofVec::shrink_to_fit
:This calls
RawVec::shrink_to_fit
:Which calls
RawVec::shrink
:Which calls
Allocator::shrink
, and in the implementation ofshrink
forGlobal
, the default global allocator, in the branch corresponding to the case wherenew_layout.size() < old_layout.size()
:Oops, the old memory is deallocated! This causes an use after free and results in undefined behaviour.
The text was updated successfully, but these errors were encountered: