-
-
Notifications
You must be signed in to change notification settings - Fork 5.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
add sizehint! for first and make append!/prepend! safer #51903
Merged
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
vtjnash
force-pushed
the
jn/safer_preappend
branch
from
October 27, 2023 17:26
5a3ba2b
to
3b34b2c
Compare
@nanosoldier |
Your benchmark job has completed - possible performance regressions were detected. A full report can be found here. |
rfourquet
reviewed
Nov 1, 2023
vtjnash
force-pushed
the
jn/safer_preappend
branch
from
November 8, 2023 14:36
3b34b2c
to
7d2f6ed
Compare
First we add an optional API parameter for `sizehint!` that controls whether it is for `push!` (default) or `pushfirst!`. Secondly, we make the offset zero when using `sizehint!` to shrink an array from the end, or the trailing size zero when using it to shring from the beginning. Then we replace the prior implementations of `prepend!` and `append!` with ones that are more safe, even if the iterator changes length during the operation or if convert fails. The result of `prepend!` may be in an undefined order (because of the `reverse!` call) in the presence of concurrent modifications or errors, but at least all of the elements will be present and all entries will be valid afterwards. Replaces and closes #49905 Replaces and closes #47391 Fixes #15868 Co-authored-by: Sukera <Seelengrab@users.noreply.github.com> Co-authored-by: MasonProtter <mason.protter@icloud.com>
vtjnash
force-pushed
the
jn/safer_preappend
branch
from
November 8, 2023 15:05
7d2f6ed
to
c739628
Compare
5 tasks
oscardssmith
added
arrays
[a, r, r, a, y, s]
and removed
merge me
PR is reviewed. Merge when all tests are passing
labels
Nov 10, 2023
This was referenced Aug 11, 2024
oscardssmith
pushed a commit
that referenced
this pull request
Aug 12, 2024
…55470) Fix #55459 In Julia 1.10, `push!` and `append!` would be functional for `AbstractVector` implementations if `resize!` and `setindex!` were defined. As of #51903 by @vtjnash as in Julia 1.11.0-rc2, `append!` now depends on an implementation of `sizehint!` and `push!`. Since `push!` also depends on `append!`, a stack overflow situation can easily be created. To avoid this, this pull request defines the following * Add generic versions of `push!(a::AbstractVector, x)` which do not depend on `append!` * Add default implementation of `sizehint!` that is a no-op The implementation of `push!(a::AbstractVector, x)` is a generic version based on the implementation of `push!(a::Vector, x)` without depending on internals. # Example for SimpleArray Consider the `SimpleArray` example from test/abstractarray.jl: ```julia mutable struct SimpleArray{T} <: AbstractVector{T} els::Vector{T} end Base.size(sa::SimpleArray) = size(sa.els) Base.getindex(sa::SimpleArray, idx...) = getindex(sa.els, idx...) Base.setindex!(sa::SimpleArray, v, idx...) = setindex!(sa.els, v, idx...) Base.resize!(sa::SimpleArray, n) = resize!(sa.els, n) Base.copy(sa::SimpleArray) = SimpleArray(copy(sa.els)) ``` Note that `setindex!` and `resize!` are implemented for `SimpleArray`. ## Julia 1.10.4: push! is functional On Julia 1.10.4, `push!` has a functional implementation for `SimpleArray` ```julia-repl julia> push!(SimpleArray{Int}(zeros(Int,5)), 6) 6-element SimpleArray{Int64}: 0 0 0 0 0 6 ``` ## Julia 1.11.0-rc2 and nightly: push! requires sizehint! and is prone to stack overflow Before this pull request, on Julia 1.11.0-rc2 and nightly, `push!` fails for want of `sizehint!`. ```julia-repl julia> push!(SimpleArray{Int}(zeros(Int,5)), 6) ERROR: MethodError: no method matching sizehint!(::SimpleArray{Int64}, ::Int64) The function `sizehint!` exists, but no method is defined for this combination of argument types. ... ``` After implementing `sizehint!`, `push!` still fails with a stack overflow. ```julia-repl julia> Base.sizehint!(a::SimpleArray, x) = a julia> push!(SimpleArray{Int}(zeros(Int, 5)), 6) Warning: detected a stack overflow; program state may be corrupted, so further execution might be unreliable. ERROR: StackOverflowError: Stacktrace: [1] _append! @ ./array.jl:1344 [inlined] [2] append! @ ./array.jl:1335 [inlined] [3] push!(a::SimpleArray{Int64}, iter::Int64) @ Base ./array.jl:1336 --- the above 3 lines are repeated 79982 more times --- [239950] _append! @ ./array.jl:1344 [inlined] [239951] append! @ ./array.jl:1335 [inlined] ``` This is because the new implementation of `append!` depends on `push!`. ## After this pull request, push! is functional. After this pull request, there is a functional `push!` for `SimpleArray` again as in Julia 1.10.4: ```julia-repl julia> push!(SimpleArray{Int}(zeros(Int, 5), 6) 6-element SimpleArray{Int64}: 0 0 0 0 0 6 ```
KristofferC
pushed a commit
that referenced
this pull request
Aug 13, 2024
…55470) Fix #55459 In Julia 1.10, `push!` and `append!` would be functional for `AbstractVector` implementations if `resize!` and `setindex!` were defined. As of #51903 by @vtjnash as in Julia 1.11.0-rc2, `append!` now depends on an implementation of `sizehint!` and `push!`. Since `push!` also depends on `append!`, a stack overflow situation can easily be created. To avoid this, this pull request defines the following * Add generic versions of `push!(a::AbstractVector, x)` which do not depend on `append!` * Add default implementation of `sizehint!` that is a no-op The implementation of `push!(a::AbstractVector, x)` is a generic version based on the implementation of `push!(a::Vector, x)` without depending on internals. # Example for SimpleArray Consider the `SimpleArray` example from test/abstractarray.jl: ```julia mutable struct SimpleArray{T} <: AbstractVector{T} els::Vector{T} end Base.size(sa::SimpleArray) = size(sa.els) Base.getindex(sa::SimpleArray, idx...) = getindex(sa.els, idx...) Base.setindex!(sa::SimpleArray, v, idx...) = setindex!(sa.els, v, idx...) Base.resize!(sa::SimpleArray, n) = resize!(sa.els, n) Base.copy(sa::SimpleArray) = SimpleArray(copy(sa.els)) ``` Note that `setindex!` and `resize!` are implemented for `SimpleArray`. ## Julia 1.10.4: push! is functional On Julia 1.10.4, `push!` has a functional implementation for `SimpleArray` ```julia-repl julia> push!(SimpleArray{Int}(zeros(Int,5)), 6) 6-element SimpleArray{Int64}: 0 0 0 0 0 6 ``` ## Julia 1.11.0-rc2 and nightly: push! requires sizehint! and is prone to stack overflow Before this pull request, on Julia 1.11.0-rc2 and nightly, `push!` fails for want of `sizehint!`. ```julia-repl julia> push!(SimpleArray{Int}(zeros(Int,5)), 6) ERROR: MethodError: no method matching sizehint!(::SimpleArray{Int64}, ::Int64) The function `sizehint!` exists, but no method is defined for this combination of argument types. ... ``` After implementing `sizehint!`, `push!` still fails with a stack overflow. ```julia-repl julia> Base.sizehint!(a::SimpleArray, x) = a julia> push!(SimpleArray{Int}(zeros(Int, 5)), 6) Warning: detected a stack overflow; program state may be corrupted, so further execution might be unreliable. ERROR: StackOverflowError: Stacktrace: [1] _append! @ ./array.jl:1344 [inlined] [2] append! @ ./array.jl:1335 [inlined] [3] push!(a::SimpleArray{Int64}, iter::Int64) @ Base ./array.jl:1336 --- the above 3 lines are repeated 79982 more times --- [239950] _append! @ ./array.jl:1344 [inlined] [239951] append! @ ./array.jl:1335 [inlined] ``` This is because the new implementation of `append!` depends on `push!`. ## After this pull request, push! is functional. After this pull request, there is a functional `push!` for `SimpleArray` again as in Julia 1.10.4: ```julia-repl julia> push!(SimpleArray{Int}(zeros(Int, 5), 6) 6-element SimpleArray{Int64}: 0 0 0 0 0 6 ``` (cherry picked from commit cf4c30a)
lazarusA
pushed a commit
to lazarusA/julia
that referenced
this pull request
Aug 17, 2024
…uliaLang#55470) Fix JuliaLang#55459 In Julia 1.10, `push!` and `append!` would be functional for `AbstractVector` implementations if `resize!` and `setindex!` were defined. As of JuliaLang#51903 by @vtjnash as in Julia 1.11.0-rc2, `append!` now depends on an implementation of `sizehint!` and `push!`. Since `push!` also depends on `append!`, a stack overflow situation can easily be created. To avoid this, this pull request defines the following * Add generic versions of `push!(a::AbstractVector, x)` which do not depend on `append!` * Add default implementation of `sizehint!` that is a no-op The implementation of `push!(a::AbstractVector, x)` is a generic version based on the implementation of `push!(a::Vector, x)` without depending on internals. # Example for SimpleArray Consider the `SimpleArray` example from test/abstractarray.jl: ```julia mutable struct SimpleArray{T} <: AbstractVector{T} els::Vector{T} end Base.size(sa::SimpleArray) = size(sa.els) Base.getindex(sa::SimpleArray, idx...) = getindex(sa.els, idx...) Base.setindex!(sa::SimpleArray, v, idx...) = setindex!(sa.els, v, idx...) Base.resize!(sa::SimpleArray, n) = resize!(sa.els, n) Base.copy(sa::SimpleArray) = SimpleArray(copy(sa.els)) ``` Note that `setindex!` and `resize!` are implemented for `SimpleArray`. ## Julia 1.10.4: push! is functional On Julia 1.10.4, `push!` has a functional implementation for `SimpleArray` ```julia-repl julia> push!(SimpleArray{Int}(zeros(Int,5)), 6) 6-element SimpleArray{Int64}: 0 0 0 0 0 6 ``` ## Julia 1.11.0-rc2 and nightly: push! requires sizehint! and is prone to stack overflow Before this pull request, on Julia 1.11.0-rc2 and nightly, `push!` fails for want of `sizehint!`. ```julia-repl julia> push!(SimpleArray{Int}(zeros(Int,5)), 6) ERROR: MethodError: no method matching sizehint!(::SimpleArray{Int64}, ::Int64) The function `sizehint!` exists, but no method is defined for this combination of argument types. ... ``` After implementing `sizehint!`, `push!` still fails with a stack overflow. ```julia-repl julia> Base.sizehint!(a::SimpleArray, x) = a julia> push!(SimpleArray{Int}(zeros(Int, 5)), 6) Warning: detected a stack overflow; program state may be corrupted, so further execution might be unreliable. ERROR: StackOverflowError: Stacktrace: [1] _append! @ ./array.jl:1344 [inlined] [2] append! @ ./array.jl:1335 [inlined] [3] push!(a::SimpleArray{Int64}, iter::Int64) @ Base ./array.jl:1336 --- the above 3 lines are repeated 79982 more times --- [239950] _append! @ ./array.jl:1344 [inlined] [239951] append! @ ./array.jl:1335 [inlined] ``` This is because the new implementation of `append!` depends on `push!`. ## After this pull request, push! is functional. After this pull request, there is a functional `push!` for `SimpleArray` again as in Julia 1.10.4: ```julia-repl julia> push!(SimpleArray{Int}(zeros(Int, 5), 6) 6-element SimpleArray{Int64}: 0 0 0 0 0 6 ```
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
First we add an optional API parameter for
sizehint!
that controls whether it is forpush!
(default) orpushfirst!
. Secondly, we make the offset zero when usingsizehint!
to shrink an array from the end, or the trailing size zero when using it to shring from the beginning.Then we replace the prior implementations of
prepend!
andappend!
with ones that are safe even if the iterator changes length during the operation or if convert fails. The result ofprepend!
may be in an undefined order (because of thereverse!
call) in the presence of concurrent modifications or errors, but at least all of the elements will be present and valid afterwards.Replaces and closes #49905
Replaces and closes #47391
Fixes #15868
Benchmarks show that repeated
push!
performance (with sizehint) is nearly equivalent to the old append performance: