strings/cstring: transcode: prevent Windows sysimage invalidation
#58038
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Should prevent some sysimage invalidation that happens on 64-bit Windows when running: ```julia struct I <: Integer end function Base.:(<<)(::I, ::Int) end ```
nsajko
added
system:windows
nsajko
changed the title
transcode: prevent some sysimage invalidationtranscode: prevent Windows sysimage invalidation
|
The change prevents these invalidations, tested on Linux with Wine: {
"invalidation_count": 16,
"trees": [
{
"method": "<<(::Main.RawInvalidations.I, ::Int64) @ Main.RawInvalidations \\home\\nsajko\\invalidations\\script\\reproducer.jl:19",
"reason": "inserting",
"mt_backedges": [
{
"type": "Tuple{typeof(<<), Any, Int64}",
"tree": {
"method_instance": {
"method": "transcode(::Type{UInt16}, src::AbstractVector{UInt8}) @ Base strings\\cstring.jl:181",
"method_instance": "MethodInstance for transcode(::Type{UInt16}, ::Base.CodeUnits{UInt8})"
},
"children": [
{
"method_instance": {
"method": "cwstring(s::AbstractString) @ Base strings\\cstring.jl:119",
"method_instance": "MethodInstance for Base.cwstring(::AbstractString)"
},
"children": [
{
"method_instance": {
"method": "memoized_env_lookup(str::AbstractString) @ Base env.jl:8",
"method_instance": "MethodInstance for Base.memoized_env_lookup(::AbstractString)"
},
"children": [
{
"method_instance": {
"method": "_hasenv(s::AbstractString) @ Base env.jl:25",
"method_instance": "MethodInstance for Base._hasenv(::AbstractString)"
},
"children": [
{
"method_instance": {
"method": "in(k::AbstractString, ::Base.KeySet{String, Base.EnvDict}) @ Base env.jl:171",
"method_instance": "MethodInstance for in(::AbstractString, ::Base.KeySet{String, Base.EnvDict})"
},
"children": [
{
"method_instance": {
"method": "haskey(d::AbstractDict, k) @ Base abstractdict.jl:19",
"method_instance": "MethodInstance for haskey(::Base.EnvDict, ::Any)"
},
"children": [
{
"method_instance": {
"method": "versioninfo(io::IO) @ LinearAlgebra Z:\\home\\nsajko\\tmp\\jl\\jl\\windows\\julia-ec424d47ff\\share\\julia\\stdlib\\v1.13\\LinearAlgebra\\src\\LinearAlgebra.jl:774",
"method_instance": "MethodInstance for LinearAlgebra.versioninfo(::IO)"
},
"children": [
{
"method_instance": {
"method": "versioninfo() @ LinearAlgebra Z:\\home\\nsajko\\tmp\\jl\\jl\\windows\\julia-ec424d47ff\\share\\julia\\stdlib\\v1.13\\LinearAlgebra\\src\\LinearAlgebra.jl:774",
"method_instance": "MethodInstance for LinearAlgebra.versioninfo()"
},
"children": [
]
},
{
"method_instance": {
"method": "versioninfo() @ LinearAlgebra Z:\\home\\nsajko\\tmp\\jl\\jl\\windows\\julia-ec424d47ff\\share\\julia\\stdlib\\v1.13\\LinearAlgebra\\src\\LinearAlgebra.jl:774",
"method_instance": "MethodInstance for LinearAlgebra.versioninfo()"
},
"children": [
]
}
]
}
]
}
]
}
]
},
{
"method_instance": {
"method": "access_env(onError::Function, str::AbstractString) @ Base env.jl:27",
"method_instance": "MethodInstance for Base.access_env(::Base.var\"#getindex##11#getindex##12\", ::AbstractString)"
},
"children": [
{
"method_instance": {
"method": "getindex(::Base.EnvDict, k::AbstractString) @ Base env.jl:164",
"method_instance": "MethodInstance for getindex(::Base.EnvDict, ::AbstractString)"
},
"children": [
{
"method_instance": {
"method": "(::LinearAlgebra.var\"#print_var#versioninfo##0\")(io, indent, name) @ LinearAlgebra Z:\\home\\nsajko\\tmp\\jl\\jl\\windows\\julia-ec424d47ff\\share\\julia\\stdlib\\v1.13\\LinearAlgebra\\src\\LinearAlgebra.jl:798",
"method_instance": "MethodInstance for (::LinearAlgebra.var\"#print_var#versioninfo##0\")(::IO, ::String, ::AbstractString)"
},
"children": [
{
"method_instance": {
"method": "versioninfo(io::IO) @ LinearAlgebra Z:\\home\\nsajko\\tmp\\jl\\jl\\windows\\julia-ec424d47ff\\share\\julia\\stdlib\\v1.13\\LinearAlgebra\\src\\LinearAlgebra.jl:774",
"method_instance": "MethodInstance for LinearAlgebra.versioninfo(::IO)"
},
"children": [
]
}
]
}
]
}
]
},
{
"method_instance": {
"method": "access_env(onError::Function, str::AbstractString) @ Base env.jl:27",
"method_instance": "MethodInstance for Base.access_env(::Returns{Symbol}, ::AbstractString)"
},
"children": [
{
"method_instance": {
"method": "get(::Base.EnvDict, k::AbstractString, def) @ Base env.jl:165",
"method_instance": "MethodInstance for get(::Base.EnvDict, ::AbstractString, ::Symbol)"
},
"children": [
{
"method_instance": {
"method": "in(k, v::Base.KeySet) @ Base abstractdict.jl:73",
"method_instance": "MethodInstance for in(::Any, ::Base.KeySet{String, Base.EnvDict})"
},
"children": [
{
"method_instance": {
"method": "haskey(d::AbstractDict, k) @ Base abstractdict.jl:19",
"method_instance": "MethodInstance for haskey(::Base.EnvDict, ::Any)"
},
"children": [
]
}
]
}
]
}
]
}
]
}
]
}
]
}
}
],
"backedges": [
{
"method_instance": {
"method": "<<(x::Integer, c::Int64) @ Base operators.jl:720",
"method_instance": "MethodInstance for <<(::Integer, ::Int64)"
},
"children": [
{
"method_instance": {
"method": "transcode(::Type{UInt16}, src::AbstractVector{UInt8}) @ Base strings\\cstring.jl:181",
"method_instance": "MethodInstance for transcode(::Type{UInt16}, ::Base.CodeUnits{UInt8})"
},
"children": [
]
}
]
}
],
"mt_cache": [
],
"mt_disable": [
]
},
{
"method": ">>(::Main.RawInvalidations.I, ::Int64) @ Main.RawInvalidations \\home\\nsajko\\invalidations\\script\\reproducer.jl:19",
"reason": "inserting",
"mt_backedges": [
{
"type": "Tuple{typeof(>>), Any, Int64}",
"tree": {
"method_instance": {
"method": "transcode(::Type{UInt16}, src::AbstractVector{UInt8}) @ Base strings\\cstring.jl:181",
"method_instance": "MethodInstance for transcode(::Type{UInt16}, ::Base.CodeUnits{UInt8})"
},
"children": [
]
}
}
],
"backedges": [
{
"method_instance": {
"method": ">>(x::Integer, c::Int64) @ Base operators.jl:761",
"method_instance": "MethodInstance for >>(::Integer, ::Int64)"
},
"children": [
{
"method_instance": {
"method": "transcode(::Type{UInt16}, src::AbstractVector{UInt8}) @ Base strings\\cstring.jl:181",
"method_instance": "MethodInstance for transcode(::Type{UInt16}, ::Base.CodeUnits{UInt8})"
},
"children": [
]
}
]
}
],
"mt_cache": [
],
"mt_disable": [
]
}
]
} |
|
Is it possible to figure out why getindex on We could fix there. |
I looked into that already, but couldn't find anything to improve in that direction. It could be fixed by adding a typeassert to the julia> methods(getindex, Tuple{Base.CodeUnits{UInt8}, Int})
# 1 method for generic function "getindex" from Base:
[1] getindex(s::Base.CodeUnits, i::Int64)
@ strings/basic.jl:803
julia> code_typed(getindex, Tuple{Base.CodeUnits{UInt8}, Int}; debuginfo=:source)
1-element Vector{Any}:
CodeInfo(
@ strings/basic.jl:803 within `getindex`
┌ @ Base_compiler.jl:55 within `getproperty`
1 ─│ %1 = builtin Base.getfield(s, :s)::AbstractString
│ └
│ %2 = dynamic Base.codeunit(%1, i)::Any
└── return %2
) => Any
Done already, however I suppose it's better to be thorough and fix on multiple places on the call chain: |
Maybe, I mean the only reason you find these in particular is because they end up getting called by type unstable code that ends up in the sysimage. Otherwise, you would annotate every |
My reasoning is that in the future some other piece of type unstable code could be added, eventually calling
Not quite:
|
51 tasks
|
What exactly is the issue here? If it's code duplication, I could introduce an additional local function like If this seems useless after the LinearAlgebra.jl PR is merged, as far as I understand it should still be useful for released versions of Julia, |
It doesn't to me feel like his function does anything wrong. The only reason you are changing these is because of some badly written code in stdlibs, or?
You can just make the same fix here in stdlib/LinearAlgebra? |
|
|
Oh, you mean to make a PR to the v1.11 backports branch, modifying the LinearAlgebra code there directly? |
Perhaps not wrong, however I guess it can do better, to be more resistant to invalidation? |
This was referenced Apr 25, 2025
58 tasks
That seems bad, to include it (for startup latency; and upgradablity) if true. Should any of it be there, or then only partially (a tiny part of) possibly? And does your PR for it and/or this one affect that situation? Should either or both be backported to 1.12, and does that mean be in 1.12.0? The other one doesn't have a 1.12 backport label. |
This was referenced Aug 19, 2025
DilumAluthge
added a commit
that referenced
this pull request
Sep 5, 2025
Backported PRs: - [x] #54840 <!-- Add boundscheck in speccache_eq to avoid OOB access due to data race --> - [x] #42080 <!-- recommend explicit `using Foo: Foo, ...` in package code (was: "using considered harmful") --> - [x] #58127 <!-- [DOC] Update installation docs: /downloads/ => /install/ --> - [x] #58202 <!-- [release-1.11] malloc: use jl_get_current_task to fix null check --> - [x] #58584 <!-- Make `Ptr` values static-show w/ type-information --> - [x] #58637 <!-- Make late gc lower handle insertelement of alloca use. --> - [x] #58837 <!-- fix null comparisons for non-standard address spaces --> - [x] #57826 <!-- Add a `similar` method for `Type{<:CodeUnits}` --> - [x] #58293 <!-- fix trailing indices stackoverflow in reinterpreted array --> - [x] #58887 <!-- Pkg: Allow configuring can_fancyprint(io::IO) using IOContext --> - [x] #58937 <!-- Fix nthreadpools size in JLOptions --> - [x] #58978 <!-- Fix precompilepkgs warn loaded setting --> - [x] #58998 <!-- Bugfix: Use Base.aligned_sizeof instead of sizeof in Mmap.mmap --> - [x] #59120 <!-- Fix memory order typo in "src/julia_atomics.h" --> - [x] #59170 <!-- Clarify and enhance confusing precompile test --> Need manual backport: - [ ] #56329 <!-- loading: clean up more concurrency issues --> - [ ] #56956 <!-- Add "mea culpa" to foreign module assignment error. --> - [ ] #57035 <!-- linux: workaround to avoid deadlock inside dl_iterate_phdr in glibc --> - [ ] #57089 <!-- Block thread from receiving profile signal with stackwalk lock --> - [ ] #57249 <!-- restore non-freebsd-unix fix for profiling --> - [ ] #58011 <!-- Remove try-finally scope from `@time_imports` `@trace_compile` `@trace_dispatch` --> - [ ] #58062 <!-- remove unnecessary edge from `exp_impl` to `pow` --> - [ ] #58157 <!-- add showing a string to REPL precompile workload --> - [ ] #58209 <!-- Specialize `one` for the `SizedArray` test helper --> - [ ] #58108 <!-- Base.get_extension & Dates.format made public --> - [ ] #58356 <!-- codegen: remove readonly from abstract type calling convention --> - [ ] #58415 <!-- [REPL] more reliable extension loading --> - [ ] #58510 <!-- Don't filter `Core` methods from newly-inferred list --> - [ ] #58110 <!-- relax dispatch for the `IteratorSize` method for `Generator` --> - [ ] #58965 <!-- Fix `hygienic-scope`s in inner macro expansions --> - [ ] #58971 <!-- Fix alignment of failed precompile jobs on CI --> - [ ] #59066 <!-- build: Also pass -fno-strict-aliasing for C++ --> Contains multiple commits, manual intervention needed: - [ ] #55877 <!-- fix FileWatching designs and add workaround for a stat bug on Apple --> - [ ] #56755 <!-- docs: fix scope type of a `struct` to hard --> - [ ] #57809 <!-- Fix fptrunc Float64 -> Float16 rounding through Float32 --> - [ ] #57398 <!-- Make remaining float intrinsics require float arguments --> - [ ] #56351 <!-- Fix `--project=@script` when outside script directory --> - [ ] #57129 <!-- clarify that time_ns is monotonic --> - [ ] #58134 <!-- Note annotated string API is experimental in Julia 1.11 in HISTORY.md --> - [ ] #58401 <!-- check that hashing of types does not foreigncall (`jl_type_hash` is concrete evaluated) --> - [ ] #58435 <!-- Fix layout flags for types that have oddly sized primitive type fields --> - [ ] #58483 <!-- Fix tbaa usage when storing into heap allocated immutable structs --> - [ ] #58512 <!-- Make more types jl_static_show readably --> - [ ] #58012 <!-- Re-enable tab completion of kwargs for large method tables --> - [ ] #58683 <!-- Add 0 predecessor to entry basic block and handle it in inlining --> - [ ] #59112 <!-- Add builtin function name to add methods error --> Non-merged PRs with backport label: - [ ] #59329 <!-- aotcompile: destroy LLVM context after serializing combined module --> - [ ] #58848 <!-- Set array size only when safe to do so --> - [ ] #58535 <!-- gf.c: include const-return methods in `--trace-compile` --> - [ ] #58038 <!-- strings/cstring: `transcode`: prevent Windows sysimage invalidation --> - [ ] #57604 <!-- `@nospecialize` for `string_index_err` --> - [ ] #57366 <!-- Use ptrdiff_t sized offsets for gvars_offsets to allow large sysimages --> - [ ] #56890 <!-- Enable getting non-boxed LLVM type from Julia Type --> - [ ] #56823 <!-- Make version of opaque closure constructor in world --> - [ ] #55958 <!-- also redirect JL_STDERR etc. when redirecting to devnull --> - [ ] #55956 <!-- Make threadcall gc safe --> - [ ] #55534 <!-- Set stdlib sources as read-only during installation --> - [ ] #55499 <!-- propagate the terminal's `displaysize` to the `IOContext` used by the REPL --> - [ ] #55458 <!-- Allow for generically extracting unannotated string --> - [ ] #55457 <!-- Make AnnotateChar equality consider annotations --> - [ ] #55220 <!-- `isfile_casesensitive` fixes on Windows --> - [ ] #53957 <!-- tweak how filtering is done for what packages should be precompiled --> - [ ] #51479 <!-- prevent code loading from lookin in the versioned environment when building Julia --> - [ ] #50813 <!-- More doctests for Sockets and capitalization fix --> - [ ] #50157 <!-- improve docs for `@inbounds` and `Base.@propagate_inbounds` --> --------- Co-authored-by: Kiran Pamnany <kpamnany@users.noreply.github.com> Co-authored-by: adienes <51664769+adienes@users.noreply.github.com> Co-authored-by: Gabriel Baraldi <baraldigabriel@gmail.com> Co-authored-by: Keno Fischer <keno@juliacomputing.com> Co-authored-by: Simeon David Schaub <simeon@schaub.rocks> Co-authored-by: Jameson Nash <vtjnash@gmail.com> Co-authored-by: Alex Arslan <ararslan@comcast.net> Co-authored-by: Fons van der Plas <fonsvdplas@gmail.com> Co-authored-by: Ian Butterworth <i.r.butterworth@gmail.com> Co-authored-by: JonasIsensee <jonas.isensee@web.de> Co-authored-by: Curtis Vogt <curtis.vogt@gmail.com> Co-authored-by: Dilum Aluthge <dilum@aluthge.com> Co-authored-by: DilumAluthgeBot <43731525+DilumAluthgeBot@users.noreply.github.com> Co-authored-by: DilumAluthge <5619885+DilumAluthge@users.noreply.github.com>
59 tasks
28 tasks
|
Close this? |
25 tasks
nsajko
removed
backport 1.11
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Should prevent all sysimage invalidation that happens on 64-bit Windows when running: