Add filesystem func to transform a path to a URI #55454

tecosaur · 2024-08-11T05:00:40Z

In a few places across Base and the stdlib, we emit paths that we like people to be able to click on in their terminal and editor. Up to this point, we have relied on auto-filepath detection, but this does not allow for alternative link text, such as contracted paths.

Doing so (via OSC 8 terminal links for example) requires filepath URI encoding.

This functionality was previously part of a PR modifying stacktrace printing (#51816), but after that became held up for unrelated reasons and another PR appeared that would benefit from this utility (#55335), I've split out this functionality so it can be used before the stacktrace printing PR is resolved.

tecosaur · 2024-08-11T07:52:50Z

This seems good on *nix machines, I could do with a little help working out what should happen with windows drive path components though.

Edit: I think I've worked it out.

base/path.jl

IanButterworth

Should this call expanduser first?

julia> Base.Filesystem.uripath("~/foo.jl")
"file://Ians-MacBook-Pro.local/Users/ian/Documents/GitHub/julia/~/foo.jl"

tecosaur · 2024-08-12T03:29:02Z

Should this call expanduser first?

I'm not sure on this. I wish otherwise but get the impression that most of the Base.Filesystem functions don't auto-expand ~, and so it may be most consistent to not.

tecosaur · 2024-08-12T03:31:08Z

Other than that, it just looks like I need to fix the Windows tests. It looks like uripath doing the right thing though:

  Expression: Base.Filesystem.uripath("$(absdrive)some$(sep)file.txt") == "file://$(host)/$(drive)some/file.txt"
   Evaluated: "file://win2k22-amdci6-4/C%3a/some/file.txt" == "file://win2k22-amdci6-4/C:\\some/file.txt"
               ^ looks good to me                              ^ needs updating

base/path.jl

davidanthoff · 2024-08-13T18:45:49Z

I don't think it should add the hostname into the URI if the user provided path doesn't have one. That is certainly true on Windows, but I also don't think it should be done on Unix/Mac.

On Windows, there are local paths and UNC paths. A URI with a hostname converts to a UNC path. So, the current implementation doesn't round trip: if one starts with a local path, converts to a URI and then back to path, one ends up with a UNC path. Plus, such a UNC path will be invalid, one can't just construct a valid UNC path by concatenating a hostname with a local file path (UNC paths are based on shares).

The percent encoding should use upper-case letters (https://www.rfc-editor.org/rfc/rfc3986#section-2.1).

I would just mimic the VS Code implementation https://github.com/microsoft/vscode-uri/blob/6dec22d7dcc6c63c30343d3a8d56050d0078cb6a/src/uri.ts#L302 :)

It might also be nice to have an optional flag where one can specify whether the path is a windows path or a unix path, so that the platform specific behavior is the default, but one also has access to the other form.

If the hostname is added (which I think it shouldn't be) then it should probably also be encoded?

I don't understand why the encoding is done per path segment? Why not just call encode on the whole path component as a whole?

tecosaur · 2024-08-14T04:55:03Z

I don't think it should add the hostname into the URI if the user provided path doesn't have one. That is certainly true on Windows, but I also don't think it should be done on Unix/Mac.

That is contrary to the impression I have from reading the referenced resources, for example the hostname component is mandatory in RFC1738. Yes, file:/// works on many systems but that's a shorthand in the Freedesktop File URI spec. However, I've seen this recommended against on the basis that both (1) it's a common extension of the RFCs, but not in keeping with them and (2) you can't just assume a link is always displayed on the same machine that has the file, people do use SSH sessions.

The percent encoding should use upper-case letters (https://www.rfc-editor.org/rfc/rfc3986#section-2.1).

Ta.

It might also be nice to have an optional flag where one can specify whether the path is a windows path or a unix path, so that the platform specific behavior is the default, but one also has access to the other form.

I'd be inclined to wait till a use case for this appeared. This is a private API currently, so we're not locking ourselves into anything.

If the hostname is added (which I think it shouldn't be) then it should probably also be encoded?

Ah yep, that was an oversight. Thanks.

I don't understand why the encoding is done per path segment? Why not just call encode on the whole path component as a whole?

Because that's how I thought to do it, that also seems like a nice approach, I'll just add / to the regex ignore list.

davidanthoff · 2024-08-14T07:34:22Z

the hostname component is mandatory in RFC1738

No, that is not correct :) https://www.rfc-editor.org/rfc/rfc1738#section-3.10 is pretty clear that <host> can be the empty string, and then you end up with file:/// according to the spec.

The key thing is that on Windows things will just not work if you put in the hostname when you start with a regular local path. If you start with a path C:\foo\bar.txt on a computer with hostname MYCOMPUTER and use the code in this PR to create a URI you get file://MYCOMPUTER/C%3A/foo/bar.txt. The correct way to turn that into a path then gets you \\MYCOMPUTER\C:\foo\bar.txt, which is a completely different path from what we started with (in fact, it appears like a UNC path but is not a valid one). The right way to do this (on Windows) is to check whether your original path is a local path or a UNC path. If local, leave the hostname empty, if it is a UNC path, parse it into the hostname and the rest of the path and then construct a URI with a hostname and path element. If Julia were to output URIs with a hostname for local paths, none of the links that the existing terminals in say VS Code or the default Windows Terminal create for that will work.

I'm less familiar with Linux/Mac, but it seems to me that given that the URI spec allows for representation of paths with and without a hostname, we should just faithfully convert paths that start without a hostname to a uri without a hostname, and paths with a hostname to a uri with a hostname. That certainly seems to be the way other languages do it.

Oh, and another thing: the set of characters that need to be percent encoded is different for different URI components. I don't remember whether the authority and the path component have the same set, but it probably would be good to check because one might not be able to use exactly the same algorithm for the two parts.

tecosaur · 2024-08-14T08:49:39Z

No, that is not correct :) https://www.rfc-editor.org/rfc/rfc1738#section-3.10 is pretty clear that can be the empty string, and then you end up with file:/// according to the spec.

Ah, so it is! I'd focused on the BNF in section 5, which has:

host           = hostname | hostnumber
fileurl        = "file://" [ host | "localhost" ] "/" fpath

As an aside, I've just noticed we probably want to reference https://www.rfc-editor.org/rfc/rfc8089 instead of RFC1738 (it lists an empty host uri as an example in Appendix B, FYI).

The key thing is that on Windows things will just not work if you put in the hostname when you start with a regular local path. If you start with a path C:\foo\bar.txt on a computer with hostname MYCOMPUTER and use the code in this PR to create a URI you get file://MYCOMPUTER/C%3A/foo/bar.txt. The correct way to turn that into a path then gets you \\MYCOMPUTER\C:\foo\bar.txt, which is a completely different path from what we started with (in fact, it appears like a UNC path but is not a valid one).

Ok, sounds like we might want to treat Windows specially based on what you're saying, but there are two aspects to this I'd like to raise first:

Regardless of UNC-path-ness, the file://MYCOMPUTER/... url seems completely valid to me based on the relevant specs. Or is there something I'm missing?
From a glance at https://learn.microsoft.com/en-us/dotnet/standard/io/file-path-formats#unc-paths, it seems as though \\MYCOMPUTER\C$\foo\bar.txt would be a valid UNC path (as long as gethostname() provides the NetBIOS machine name).

If Julia is supposed to accept UNC paths, it seems like we'd need to be able to split a windows path into three components: server, share, path.

If Julia were to output URIs with a hostname for local paths, none of the links that the existing terminals in say VS Code or the default Windows Terminal create for that will work.

I'll happily take your word on this, but I'd be interested in hearing how testing this goes if you haven't already.

I'm less familiar with Linux/Mac, but it seems to me that given that the URI spec allows for representation of paths with and without a hostname, we should just faithfully convert paths that start without a hostname to a uri without a hostname, and paths with a hostname to a uri with a hostname.

Unix paths don't have a host component the way that Windows UNC paths seem to.

That certainly seems to be the way other languages do it.

Other languages can have libraries that are able to infer whether a hostname component is appropriate or not? I'd be interested to see that.

Anyway, while it sounds like Windows might be a bit more particular, the *nix machines I've used have been perfectly happy with hostname-form file:// URIs. At least for *nix machines then, I'm inclined to take the approach that it's better to have the hostname component and not need it, that need it and not have it.

Oh, and another thing: the set of characters that need to be percent encoded is different for different URI components. I don't remember whether the authority and the path component have the same set, but it probably would be good to check because one might not be able to use exactly the same algorithm for the two parts.

From my reading of the last paragraph of https://www.rfc-editor.org/rfc/rfc3986#section-3.2.2 and https://www.rfc-editor.org/rfc/rfc3986#section-3.3 it seems like percent-encoded octets is appropriate for both, skipping the characters listed in https://www.rfc-editor.org/rfc/rfc3986#section-2.3.

davidanthoff · 2024-08-14T20:32:50Z

file://MYCOMPUTER/... url seems completely valid to me based on the relevant specs

Yes, it is a valid URI that decodes to a UNC path.

it seems as though \\MYCOMPUTER\C$\foo\bar.txt would be a valid UNC path

Yes, those are so called administrative shares. They exist by default if network sharing has been enabled on a machine, and only administrators can access them.

If Julia were to output URIs with a hostname for local paths, none of the links that the existing terminals in say VS Code or the default Windows Terminal create for that will work.

I'll happily take your word on this, but I'd be interested in hearing how testing this goes if you haven't already.

I just tried it in Windows Terminal. It handles file URIs without an authority just fine, and for URIs with an authority it says that those link types aren't supported.

In the VS Code terminal many more things work. file://DANTHOFF-HOME/C$/Users will open the network share, i.e. it turns that into an UNC path and then opens that. file://DANTHOFF-HOME/C:/Users (surprisingly) also works, and it actually does not turn that into a UNC path, but instead seems to recognize that the hostname matches the local hostname, and then just access things via the local file path. To me that seems a bit weird, to be perfectly honest, I think they must have some logic when to turn an URI with a hostname into a local file path vs UNC path that is not entirely clear to me.

I just stumbled over another thing: I think some of the RFCs seem to suggest that the hostname must be a fully qualified domain name or an IP address, not just a hostname?

tecosaur · 2024-08-15T05:25:50Z

Yes, those are so called administrative shares. They exist by default if network sharing has been enabled on a machine, and only administrators can access them.

👍 good to know

I just tried it in Windows Terminal. It handles file URIs without an authority just fine, and for URIs with an authority it says that those link types aren't supported. In the VS Code terminal many more things work [...] I think they must have some logic when to turn an URI with a hostname into a local file path vs UNC path that is not entirely clear to me.

This is very useful to hear.

I just stumbled over another thing: I think some of the RFCs seem to suggest that the hostname must be a fully qualified domain name or an IP address, not just a hostname?

This is a bit muddy. RFC8089 indeed describes it as a fully qualified domain name or IP, but then in Appendix E.3 on UNC paths, talks about using the UNC hostname in place of a fully qualified domain name.

That said, when including the hostname I paid particular attention to the Freedesktop URI spec, which includes:

Hostnames
---------

When generating a file: uri the hostname part, if nonempty, should be
whatever is returned from gethostname(). This means that the name is
canonical for all users on the same machine, so that you can easily
see if the referenced file is on the current machine. Note that
"localhost" or an empty hostname needs to be handled specially, always
meaning the host the uri is being interpreted on.

In some cases gethostname() will happen to be a fully qualified domain name, e.g. on the CI runner one of the tests produces file://amdci4.julia.csail.mit.edu/some%20file%20with%20%5Eodd%25%20chars.

So, the current approach I've taken (see recent force pushes) is to:

Use gethostname() as the hostname on *nix machines
Extract the hostname component from UNC paths on Windows
Use an empty hostname for local paths on Windows

From our conversation and the specs, this seems to me like a good enough approach to try initially.

davidanthoff · 2024-08-15T19:30:05Z

From our conversation and the specs, this seems to me like a good enough approach to try initially.

Agreed!

I stumbled over another interesting question: what to do about WSL (Windows Subsystem for Linux), i.e. if a user runs Julia in WSL on Windows. They will be running a Linux build of Julia, and Julia will essentially run in a Linux VM, but the terminal is running on Windows. The links generated right now won't work in that Windows environment. Windows Terminal does allow for two specific hostnames in file URIs, namely wsl$ and wsl.localhost (microsoft/terminal#14993). In an ideal world we would detect that case, and then output URIs that use one of those hostnames. There is another wrinkle, namely the URI would also have to include the distribution name. From within WSL one can run wslpath -m <PATH> to generate a path that works from Windows to access the WSL file system, and so if one turns that into a file link, everything should work well. In theory ;)

tecosaur · 2024-08-24T06:55:46Z

Hmmm, that is an annoying wrinkle. From a quick read online it seems like checking for the existence of /proc/sys/fs/binfmt_misc/WSLInterop might be the way we should check for WSL. It's what snapd uses, and seems to be used widely beyond it.

There is another wrinkle, namely the URI would also have to include the distribution name. From within WSL one can run wslpath -m to generate a path that works from Windows to access the WSL file system, and so if one turns that into a file link, everything should work well. In theory ;)

If you can give me more detail on what this looks like in practice, then I'll see about doing this 🙂

In a few places across Base and the stdlib, we emit paths that we like people to be able to click on in their terminal and editor. Up to this point, we have relied on auto-filepath detection, but this does not allow for alternative link text, such as contracted paths. Doing so (via OSC 8 terminal links for example) requires filepath URI encoding. This functionality was previously part of a PR modifying stacktrace printing, but after that became held up for unrelated reasons and another PR appeared that would benefit from this utility, I've split out this functionality so it can be used before the stacktrace printing PR is resolved.

tecosaur · 2024-08-24T07:13:30Z

Actually, I think I might have figured it out.

fredrikekre · 2024-08-25T00:50:48Z

base/path.jl

+        encode_uri_component(s) = # RFC3986 Section 2.3
+            replace(s, r"[^A-Za-z0-9\-_.~/]+" => percent_escape)
+        localpath = join(eachsplit(abspath(path), path_separator_re, keepempty=false), '/')
+        host = if ispath("/proc/sys/fs/binfmt_misc/WSLInterop") # WSL sigil


xref #36425

Ah, that's a relevant xref. Checking for /proc/sys/fs/binfmt_misc/WSLInterop looks like a pretty solid way of checking for WSL though, but from the page I linked it seems like it may have been introduced after the issue that's mentioned in that PR as the relevant reference.

davidanthoff · 2024-08-25T16:07:13Z

Running wslpath -m / inside WSL returns a path that includes the hostname and the distribution name. So I guess one way would be to run that once, cache the results and use that? Or maybe find the source for that and see what they are doing...

IanButterworth · 2024-09-05T00:04:35Z

When this is ready it would be good to use it for the generic branch here

julia/stdlib/Profile/src/Profile.jl

Line 827 in bada969

    
           # TODO: convert the path to a generic URI (line numbers are not supported by generic URI)

that just went in in #55335

Also, if there are any improvements to be made to the IDE-specific formats above that might have been garnered from this PR work.

IanButterworth · 2024-09-24T17:26:53Z

I propose we merge this. It seems to me handling WSL edge cases can be future improvements?

tecosaur · 2024-09-24T17:39:54Z

From the docs I've read, the code I've written should handle WSL, but there's nothing like somebody actually testing it.

It can always be improved in follow-up PRs 🙂

* Improve type-stability in SymTridiagonal triu!/tril! (#55646) Changing the final `elseif` branch to an `else` makes it clear that the method definite returns a value, and the returned type is now a `Tridiagonal` instead of a `Union{Nothing, Tridiagonal}` * Reuse size-check function from `lacpy!` in `copytrito!` (#55664) Since there is a size-check function in `lacpy!` that does the same thing, we may reuse it instead of duplicating the check * Update calling-c-and-fortran-code.md: fix ccall parameters (not a tuple) (#55665) * Allow exact redefinition for types with recursive supertype reference (#55380) This PR allows redefining a type when the new type is exactly identical to the previous one (like #17618, #20592 and #21024), even if the type has a reference to itself in its supertype. That particular case used to error (issue #54757), whereas with this PR: ```julia julia> struct Rec <: AbstractVector{Rec} end julia> struct Rec <: AbstractVector{Rec} end # this used to error julia> ``` Fix #54757 by implementing the solution proposed there. Hence, this should also fix downstream Revise bug https://github.com/timholy/Revise.jl/issues/813. --------- Co-authored-by: N5N3 <2642243996@qq.com> * Reroute Symmetric/Hermitian + Diagonal through triangular (#55605) This should fix the `Diagonal`-related issue from https://github.com/JuliaLang/julia/issues/55590, although the `SymTridiagonal` one still remains. ```julia julia> using LinearAlgebra julia> a = Matrix{BigFloat}(undef, 2,2) 2×2 Matrix{BigFloat}: #undef #undef #undef #undef julia> a[1] = 1; a[3] = 1; a[4] = 1 1 julia> a = Hermitian(a) 2×2 Hermitian{BigFloat, Matrix{BigFloat}}: 1.0 1.0 1.0 1.0 julia> b = Symmetric(a) 2×2 Symmetric{BigFloat, Matrix{BigFloat}}: 1.0 1.0 1.0 1.0 julia> c = Diagonal([1,1]) 2×2 Diagonal{Int64, Vector{Int64}}: 1 ⋅ ⋅ 1 julia> a+c 2×2 Hermitian{BigFloat, Matrix{BigFloat}}: 2.0 1.0 1.0 2.0 julia> b+c 2×2 Symmetric{BigFloat, Matrix{BigFloat}}: 2.0 1.0 1.0 2.0 ``` * inference: check argtype compatibility in `abstract_call_opaque_closure` (#55672) * Forward istriu/istril for triangular to parent (#55663) * win: move stack_overflow_warning to the backtrace fiber (#55640) There is not enough stack space remaining after a stack overflow on Windows to allocate the 4k page used by `write` to call the WriteFile syscall. This causes it to hard-crash. But we can simply run this on the altstack implementation, where there is plenty of space. * Check if ct is not null before doing is_addr_on_stack in the macos signal handler. (#55603) Before the check we used to segfault while segfaulting and hang --------- Co-authored-by: Jameson Nash <vtjnash@gmail.com> * Profile.print: color Base/Core & packages. Make paths clickable (#55335) Updated ## This PR ![Screenshot 2024-09-02 at 1 47 23 PM](https://github.com/user-attachments/assets/1264e623-70b2-462a-a595-1db2985caf64) ## master ![Screenshot 2024-09-02 at 1 49 42 PM](https://github.com/user-attachments/assets/14d62fe1-c317-4df5-86e9-7c555f9ab6f1) Todo: - [ ] ~Maybe drop the `@` prefix when coloring it, given it's obviously special when colored~ If someone copy-pasted the profile into an issue this would make it confusing. - [ ] Figure out why `Profile.print(format=:flat)` is truncating before the terminal width is used up - [x] Make filepaths terminal links (even if they're truncated) * better signal handling (#55623) Instead of relying on creating a fake stack frame, and having no signals delivered, kernel bugs, accidentally gc_collect, or other issues occur during the delivery and execution of these calls, use the ability we added recently to emulate a longjmp into a unw_context to eliminate any time where there would exist any invalid states. Secondly, when calling jl_exit_thread0_cb, we used to end up completely smashing the unwind info (with CFI_NOUNWIND), but this makes core files from SIGQUIT much less helpful, so we now have a `fake_stack_pop` function with contains the necessary CFI directives such that a minimal unwind from the debugger will likely still succeed up into the frames that were removed. We cannot do this perfectly on AArch64 since that platform's DWARF spec lacks the ability to do so. On other platforms, this should be possible to implement exactly (subject to libunwind implementation quality). This is currently thus only fully implemented for x86_64 on Darwin Apple. * fix `exct` for mismatched opaque closure call * improve `exct` modeling for opaque closure calls * fix `nothrow` modeling for `invoke` calls * improve `exct` modeling for `invoke` calls * show a bit more detail when finished precompiling (#55660) * subtype: minor clean up for fast path for lhs union and rhs typevar (#55645) Follow up #55413. The error pattern mentioned in https://github.com/JuliaLang/julia/pull/55413#issuecomment-2288384468 care's `∃y`'s ub in env rather than its original ub. So it seems more robust to check the bounds in env directly. The equivalent typevar propagation is lifted from `subtype_var` for the same reason. * Adding `JL_DATA_TYPE` annotation to `_jl_globalref_t` (#55684) `_jl_globalref_t` seems to be allocated in the heap, and there is an object `jl_globalref_type` which indicates that it is in fact, a data type, thus it should be annotated with `JL_DATA_TYPE`?? * Make GEP when loading the PTLS an inbounds one. (#55682) Non inbounds GEPs should only be used when doing pointer arithmethic i.e Ptr or MemoryRef boundscheck. Found when auditing non inbounds GEPs for https://github.com/JuliaLang/julia/pull/55681 * codegen: make boundscheck GEP not be inbounds while the load GEP is inbounds (#55681) Avoids undefined behavior on the boundschecking arithmetic, which is correct only assuming overflow follows unsigned arithmetic wrap around rules. Also add names to the Memory related LLVM instructions to aid debugging Closes: https://github.com/JuliaLang/julia/pull/55674 * Make `rename` public (#55652) Fixes #41584. Follow up of #55503 I think `rename` is a very useful low-level file system operation. Many other programming languages have this function, so it is useful when porting IO code to Julia. One use case is to improve the Zarr.jl package to be more compatible with zarr-python. https://github.com/zarr-developers/zarr-python/blob/0b5483a7958e2ae5512a14eb424a84b2a75dd727/src/zarr/v2/storage.py#L994 uses the `os.replace` function. It would be nice to be able to directly use `Base.rename` as a replacement for `os.replace` to ensure compatibility. Another use case is writing a safe zip file extractor in pure Julia. https://github.com/madler/sunzip/blob/34107fa9e2a2e36e7e72725dc4c58c9ad6179898/sunzip.c#L365 uses the `rename` function to do this in C. Lastly in https://github.com/medyan-dev/MEDYANSimRunner.jl/blob/67d5b42cc599670486d5d640260a95e951091f7a/src/file-saving.jl#L83 I am using `ccall(:jl_fs_rename` to save files, because I have large numbers of Julia processes creating and reading these files at the same time on a distributed file system on a cluster, so I don't want data to become corrupted if one of the nodes crashes (which happens fairly regularly). However `jl_fs_rename` is not public, and might break in a future release. This PR also adds a note to `mv` comparing it to the `mv` command, similar to the note on the `cp` function. * contrib: include private libdir in `ldflags` on macOS (#55687) The private libdir is used on macOS, so it needs to be included in our `ldflags` * Profile.print: Shorten C paths too (#55683) * [LLVMLibUnwindJLL] Update llvmlibunwind to 14.0.6 (#48140) * Add `JL_DATA_TYPE` for `jl_line_info_node_t` and `jl_code_info_t` (#55698) * Canonicalize names of nested functions by keeping a more fine grained counter -- per (module, method name) pair (#53719) As mentioned in https://github.com/JuliaLang/julia/pull/53716, we've been noticing that `precompile` statements lists from one version of our codebase often don't apply cleanly in a slightly different version. That's because a lot of nested and anonymous function names have a global numeric suffix which is incremented every time a new name is generated, and these numeric suffixes are not very stable across codebase changes. To solve this, this PR makes the numeric suffixes a bit more fine grained: every pair of (module, top-level/outermost function name) will have its own counter, which should make nested function names a bit more stable across different versions. This PR applies @JeffBezanson's idea of making the symbol name changes directly in `current-julia-module-counter`. Here is an example: ```Julia julia> function foo(x) function bar(y) return x + y end end foo (generic function with 1 method) julia> f = foo(42) (::var"#bar#foo##0"{Int64}) (generic function with 1 method) ``` * Use `uv_available_parallelism` inside `jl_effective_threads` (#55592) * [LinearAlgebra] Initialise number of BLAS threads with `jl_effective_threads` (#55574) This is a safer estimate than `Sys.CPU_THREADS` to avoid oversubscribing the machine when running distributed applications, or when the Julia process is constrained by external controls (`taskset`, `cgroups`, etc.). Fix #55572 * Artifacts: Improve type-stability (#55707) This improves Artifacts.jl to make `artifact"..."` fully type-stable, so that it can be used with `--trim`. This is a requirement for JLL support w/ trimmed executables. Dependent on https://github.com/JuliaLang/julia/pull/55016 --------- Co-authored-by: Gabriel Baraldi <baraldigabriel@gmail.com> * Remove redundant conversion in structured matrix broadcasting (#55695) The additional construction is unnecessary, as we are already constructing a `Matrix`. Performance: ```julia julia> using LinearAlgebra julia> U = UpperTriangular(rand(1000,1000)); julia> L = LowerTriangular(rand(1000,1000)); julia> @btime $U .+ $L; 1.956 ms (6 allocations: 15.26 MiB) # nightly 1.421 ms (3 allocations: 7.63 MiB) # This PR ``` * [Profile] fix threading issue (#55704) I forgot about the existence of threads, so had hard-coded this to only support one thread. Clearly that is not sufficient though, so use the semaphore here as it is intended to be used. Fixes #55703 --------- Co-authored-by: Ian Butterworth <i.r.butterworth@gmail.com> * delete flaky ranges/`TwicePrecision` test (#55712) Fixes #55710 * Avoid stack overflow in triangular eigvecs (#55497) This fixes a stack overflow in ```julia julia> using LinearAlgebra, StaticArrays julia> U = UpperTriangular(SMatrix{2,2}(1:4)) 2×2 UpperTriangular{Int64, SMatrix{2, 2, Int64, 4}} with indices SOneTo(2)×SOneTo(2): 1 3 ⋅ 4 julia> eigvecs(U) Warning: detected a stack overflow; program state may be corrupted, so further execution might be unreliable. ERROR: StackOverflowError: Stacktrace: [1] eigvecs(A::UpperTriangular{Float32, SMatrix{2, 2, Float32, 4}}) (repeats 79984 times) @ LinearAlgebra ~/.julia/juliaup/julia-nightly/share/julia/stdlib/v1.12/LinearAlgebra/src/triangular.jl:2749 ``` After this, ```julia julia> eigvecs(U) 2×2 Matrix{Float32}: 1.0 1.0 0.0 1.0 ``` * builtins: add `Core.throw_methoderror` (#55705) This allows us to simulate/mark calls that are known-to-fail. Required for https://github.com/JuliaLang/julia/pull/54972/ * Small missing tests for Irrationals (#55657) Looks like a bunch of methods for `Irrational`s are tested but not picked up by coverage... * Implement faster thread local rng for scheduler (#55501) Implement optimal uniform random number generator using the method proposed in https://github.com/swiftlang/swift/pull/39143 based on OpenSSL's implementation of it in https://github.com/openssl/openssl/blob/1d2cbd9b5a126189d5e9bc78a3bdb9709427d02b/crypto/rand/rand_uniform.c#L13-L99 This PR also fixes some bugs found while developing it. This is a replacement for https://github.com/JuliaLang/julia/pull/50203 and fixes the issues found by @IanButterworth with both rngs C rng <img width="1011" alt="image" src="https://github.com/user-attachments/assets/0dd9d5f2-17ef-4a70-b275-1d12692be060"> New scheduler rng <img width="985" alt="image" src="https://github.com/user-attachments/assets/4abd0a57-a1d9-46ec-99a5-535f366ecafa"> ~On my benchmarks the julia implementation seems to be almost 50% faster than the current implementation.~ With oscars suggestion of removing the debiasing this is now almost 5x faster than the original implementation. And almost fully branchless We might want to backport the two previous commits since they technically fix bugs. --------- Co-authored-by: Valentin Churavy <vchuravy@users.noreply.github.com> * Add precompile signatures to Markdown to reduce latency. (#55715) Fixes #55706 that is seemingly a 4472x regression, not just 16x (was my first guess, based on CondaPkg, also fixes or greatly mitigates https://github.com/JuliaPy/CondaPkg.jl/issues/145), and large part of 3x regression for PythonCall. --------- Co-authored-by: Kristoffer Carlsson <kcarlsson89@gmail.com> * Fix invalidations for FileIO (#55593) Fixes https://github.com/JuliaIO/FileIO.jl/issues/396 * Fix various issues with PGO+LTO makefile (#55581) This fixes various issues with the PGO+LTO makefile - `USECCACHE` doesn't work throwing an error at https://github.com/JuliaLang/julia/blob/eb5587dac02d1f6edf486a71b95149139cc5d9f7/Make.inc#L734 This is because setting `CC` and `CCX` by passing them as arguments to `make` prevents `Make.inc` from prepending these variables with `ccache` as `Make.inc` doesn't use override. To workaround this I instead set `USECLANG` and add the toolchain to the `PATH`. - To deal with similar issues for the other make flags, I pass them as environment variables which can be edited in `Make.inc`. - I add a way to build in one go by creating the `all` target, now you can just run `make` and a PGO+LTO build that profiles Julia's build will be generated. - I workaround `PROFRAW_FILES` not being reevaluated after `stage1` builds, this caused the generation of `PROFILE_FILE` to run an outdated command if `stage1` was built and affected the profraw files. This is important when building in one go. - I add a way to run rules like `binary-dist` which are not defined in this makefile with the correct toolchain which for example prevents `make binary-dist` from unnecessarily rebuilding `sys.ji`. - Include `-Wl,--undefined-version` till https://github.com/JuliaLang/julia/issues/54533 gets fixed. These changes need to be copied to the PGO+LTO+BOLT makefile and some to the BOLT makefile in a later pr. --------- Co-authored-by: Zentrik <Zentrik@users.noreply.github.com> * Fix `pkgdir` for extensions (#55720) Fixes https://github.com/JuliaLang/julia/issues/55719 --------- Co-authored-by: Max Horn <241512+fingolfin@users.noreply.github.com> * Avoid materializing arrays in bidiag matmul (#55450) Currently, small `Bidiagonal`/`Tridiagonal` matrices are materialized in matrix multiplications, but this is wasteful and unnecessary. This PR changes this to use a naive matrix multiplication for small matrices, and fall back to the banded multiplication for larger ones. Multiplication by a `Bidiagonal` falls back to a banded matrix multiplication for all sizes in the current implementation, and iterates in a cache-friendly manner for the non-`Bidiagonal` matrix. In certain cases, the matrices were being materialized if the non-structured matrix was small, even if the structured matrix was large. This is changed as well in this PR. Some improvements in performance: ```julia julia> B = Bidiagonal(rand(3), rand(2), :U); A = rand(size(B)...); C = similar(A); julia> @btime mul!($C, $A, $B); 193.152 ns (6 allocations: 352 bytes) # nightly v"1.12.0-DEV.1034" 18.826 ns (0 allocations: 0 bytes) # This PR julia> T = Tridiagonal(rand(99), rand(100), rand(99)); A = rand(2, size(T,2)); C = similar(A); julia> @btime mul!($C, $A, $T); 9.398 μs (8 allocations: 79.94 KiB) # nightly 416.407 ns (0 allocations: 0 bytes) # This PR julia> B = Bidiagonal(rand(300), rand(299), :U); A = rand(20000, size(B,2)); C = similar(A); julia> @btime mul!($C, $A, $B); 33.395 ms (0 allocations: 0 bytes) # nightly 6.695 ms (0 allocations: 0 bytes) # This PR (cache-friendly) ``` Closes https://github.com/JuliaLang/julia/pull/55414 --------- Co-authored-by: Daniel Karrasch <daniel.karrasch@posteo.de> * Fix `@time_imports` extension recognition (#55718) * drop typed GEP calls (#55708) Now that we use LLVM 18, and almost have LLVM 19 support, do cleanup to remove LLVM 15/16 type pointer support. LLVM now slightly prefers that we rewrite our complex GEP to use a simple emit_ptrgep call instead, which is also much simpler for julia to emit also. * minor fixup for JuliaLang/julia#55705 (#55726) * [REPL] prevent silent hang if precompile script async blocks fail (#55685) * Various fixes to byte / bytearray search (#54579) This was originally intended as a targeted fix to #54578, but I ran into a bunch of smaller issues with this code that also needed to be solved and it turned out to be difficult to fix them with small, trivial PRs. I would also like to refactor this whole file, but I want these correctness fixes to be merged first, because a larger refactoring has higher risk of getting stuck without getting reviewed and merged. ## Larger things that needs decisions * The internal union `Base.ByteArray` has been deleted. Instead, the unions `DenseInt8` and `DenseUInt8` have been added. These more comprehensively cover the types that was meant, e.g. `Memory{UInt8}` was incorrectly not covered by the former. As stated in the TODO, the concept of a "memory backed dense byte array" is needed throughout Julia, so this ideally needs to be implemented as a single type and used throughout Base. The fix here is a decent temporary solution. See #53178 #54581 * The `findall` docstring between two arrays was incorrectly not attached to the method - now it is. **Note that this change _changes_ the documentation** since it includes a docstring that was previously missed. Hence, it's an API addition. * Added a new minimal `testhelpers/OffsetDenseArrays.jl` which provide a `DenseVector` with offset axes for testing purposes. ## Trivial fixes * `findfirst(==(Int8(-1)), [0xff])` and similar findlast, findnext and findprev is no longer buggy, see #54578 * `findfirst([0x0ff], Int8[-1])` is similarly no longer buggy, see #54578 * `findnext(==('\xa6'), "æ", 1)` and `findprev(==('\xa6'), "æa", 2)` no longer incorrectly throws an error * The byte-oriented find* functions now work correctly with offset arrays * Fixed incorrect use of `GC.@preserve`, where the pointer was taken before the preserve block. * More of the optimised string methods now also apply to `SubString{String}` Closes #54578 Co-authored-by: Martin Holters <martin.holters@hsu-hh.de> * codegen: deduplicate code for calling a specsig (#55728) I am tired of having 3 gratuitously different versions of this code to maintain. * Fix "Various fixes to byte / bytearray search" (#55734) Fixes the conflict between #54593 and #54579 `_search` returns `nothing` instead of zero as a sentinal in #54579 * Fix `make binary-dist` when using `USE_BINARYBUILDER_LLVM=0` (#55731) `make binary-dist` expects lld to be in usr/tools but it ends up in usr/bin so I copied it into usr/tools. Should fix the scheduled source tests which currently fail at linking. I think this is also broken with `USE_BINARYBUILDER_LLVM=0` and `BUILD_LLD=0`, maybe https://github.com/JuliaLang/julia/commit/ceaeb7b71bc76afaca2f3b80998164a47e30ce33 is the fix? --------- Co-authored-by: Zentrik <Zentrik@users.noreply.github.com> * Precompile the `@time_imports` printing so it doesn't confuse reports (#55729) Makes functions for the report printing that can be precompiled into the sysimage. * codegen: some cleanup of layout computations (#55730) Change Alloca to take an explicit alignment, rather than relying on LLVM to guess our intended alignment from the DataLayout. Eventually we should try to change this code to just get all layout data from julia queries (jl_field_offset, julia_alignment, etc.) instead of relying on creating an LLVM element type for memory and inspecting it (CountTrackedPointers, DataLayout, and so on). * Add some loading / LazyArtifacts precompiles to the sysimage (#55740) Fixes https://github.com/JuliaLang/julia/issues/55725 These help LazyArtifacts mainly but seem beneficial for the sysimage. * Update stable version number in readme to v1.10.5 (#55742) * Add `invokelatest` barrier to `string(...)` in `@assert` (#55739) This change protects `@assert` from invalidations to `Base.string(...)` by adding an `invokelatest` barrier. A common source of invalidations right now is `print(io, join(args...))`. The problem is: 1. Inference concludes that `join(::Any...)` returns `Union{String,AnnotatedString}` 2. The `print` call is union-split to `String` and `AnnotatedString` 3. This code is now invalidated when StyledStrings defines `print(io, ::AnnotatedString)` The invalidation chain for `@assert` is similar: ` @assert 1 == 1` calls into `string(::Expr)` which calls into `print(io, join(args::Any...))`. Unfortunately that leads to the invalidation of almost all `@assert`s without an explicit error message Similar to https://github.com/JuliaLang/julia/pull/55583#issuecomment-2308969806 * Don't show string concatenation error hint with zero arg `+` (#55749) Closes #55745 * Don't leave trailing whitespace when printing do-block expr (#55738) Before, when printing a `do`-block, we'd print a white-space after `do` even if no arguments follow. Now we don't print that space. --------- Co-authored-by: Lilith Orion Hafner <lilithhafner@gmail.com> * Don't pass lSystem to the linker since macos always links it (#55722) This stops it complaing about duplicated libs. For libunwind there isn't much we can do because it's part of lsystem and we also need out own. * define `numerator` and `denominator` for `Complex` (#55694) Fixes #55693 * More testsets for SubString and a few missing tests (#55656) Co-authored-by: Simeon David Schaub <simeon@schaub.rocks> * Reorganize search tests into testsets (#55658) Some of these tests are nearly 10 years old! Organized some of them into testsets just in case one breaks in the future, should make it easier to find the problem. --------- Co-authored-by: Simeon David Schaub <simeon@schaub.rocks> * fix #45494, error in ssa conversion with complex type decl (#55744) We were missing a call to `renumber-assigned-ssavalues` in the case where the declared type is used to assert the type of a value taken from a closure box. * Revert "Avoid materializing arrays in bidiag matmul" (#55737) Reverts JuliaLang/julia#55450. @jishnub suggested reverting this PR to fix #55727. * Add a docs section about loading/precomp/ttfx time tuning (#55569) * Add compat entry for `Base.donotdelete` (#55773) * REPL: precompile in its own module because Main is closed. Add check for unexpected errors. (#55759) * Try to put back previously flakey addmul tests (#55775) Partial revert of #50071, inspired by conversation in https://github.com/JuliaLang/julia/issues/49966#issuecomment-2350935477 Ran the tests 100 times to make sure we're not putting back something that's still flaky. Closes #49966 * Print results of `runtests` with `printstyled` (#55780) This ensures escape characters are used only if `stdout` can accept them. * move null check in `unsafe_convert` of RefValue (#55766) LLVM can optimize out this check but our optimizer can't, so this leads to smaller IR in most cases. * Fix hang in tmerge_types_slow (#55757) Fixes https://github.com/JuliaLang/julia/issues/55751 Co-authored-by: Jameson Nash <jameson@juliacomputing.com> * trace-compile: color recompilation yellow (#55763) Marks recompilation of a method that produced a `precompile` statement as yellow, or if color isn't supported adds a trailing comment: `# recompilation`. The coloring matches the `@time_imports` coloring. i.e. an excerpt of ``` % ./julia --start=no --trace-compile=stderr --trace-compile-timing -e "using InteractiveUtils; @time @time_imports using Plots" ``` ![Screenshot 2024-09-13 at 5 04 24 PM](https://github.com/user-attachments/assets/85bd99e0-586e-4070-994f-2d845be0d9e7) * Use PrecompileTools mechanics to compile REPL (#55782) Fixes https://github.com/JuliaLang/julia/issues/55778 Based on discussion here https://github.com/JuliaLang/julia/issues/55778#issuecomment-2352428043 With this `?reinterpret` feels instant, with only these precompiles at the start. ![Screenshot 2024-09-16 at 9 49 39 AM](https://github.com/user-attachments/assets/20dc016d-c6f7-4870-acd7-0e795dcf541b) * use `inferencebarrier` instead of `invokelatest` for 1-arg `@assert` (#55783) This version would be better as per this comment: <https://github.com/JuliaLang/julia/pull/55739#pullrequestreview-2304360447> I confirmed this still allows us to avoid invalidations reported at JuliaLang/julia#55583. * Inline statically known method errors. (#54972) This replaces the `Expr(:call, ...)` with a call of a new builtin `Core.throw_methoderror` This is useful because it makes very clear if something is a static method error or a plain dynamic dispatch that always errors. Tools such as AllocCheck or juliac can notice that this is not a genuine dynamic dispatch, and prevent it from becoming a false positive compile-time error. Dependent on https://github.com/JuliaLang/julia/pull/55705 --------- Co-authored-by: Cody Tapscott <topolarity@tapscott.me> * Fix shell `cd` error when working dir has been deleted (#41244) root cause: if current dir has been deleted, then pwd() will throw an IOError: pwd(): no such file or directory (ENOENT) --------- Co-authored-by: Ian Butterworth <i.r.butterworth@gmail.com> * codegen: fix bits compare for UnionAll (#55770) Fixes #55768 in two parts: one is making the type computation in emit_bits_compare agree with the parent function and two is not using the optimized egal code for UnionAll kinds, which is different from how the egal code itself works for kinds. * use libuv to measure maxrss (#55806) Libuv has a wrapper around rusage on Unix (and its equivalent on Windows). We should probably use it. * REPL: use atreplinit to change the active module during precompilation (#55805) * 🤖 [master] Bump the Pkg stdlib from 299a35610 to 308f9d32f (#55808) * Improve codegen for `Core.throw_methoderror` and `Core.current_scope` (#55803) This slightly improves our (LLVM) codegen for `Core.throw_methoderror` and `Core.current_scope` ```julia julia> foo() = Core.current_scope() julia> bar() = Core.throw_methoderror(+, nothing) ``` Before: ```llvm ; Function Signature: foo() define nonnull ptr @julia_foo_2488() #0 { top: %0 = call ptr @jl_get_builtin_fptr(ptr nonnull @"+Core.#current_scope#2491.jit") %Builtin_ret = call nonnull ptr %0(ptr nonnull @"jl_global#2492.jit", ptr null, i32 0) ret ptr %Builtin_ret } ; Function Signature: bar() define void @julia_bar_589() #0 { top: %jlcallframe1 = alloca [2 x ptr], align 8 %0 = call ptr @jl_get_builtin_fptr(ptr nonnull @"+Core.#throw_methoderror#591.jit") %jl_nothing = load ptr, ptr @jl_nothing, align 8 store ptr @"jl_global#593.jit", ptr %jlcallframe1, align 8 %1 = getelementptr inbounds ptr, ptr %jlcallframe1, i64 1 store ptr %jl_nothing, ptr %1, align 8 %Builtin_ret = call nonnull ptr %0(ptr nonnull @"jl_global#592.jit", ptr nonnull %jlcallframe1, i32 2) call void @llvm.trap() unreachable } ``` After: ```llvm ; Function Signature: foo() define nonnull ptr @julia_foo_713() #0 { top: %thread_ptr = call ptr asm "movq %fs:0, $0", "=r"() #5 %tls_ppgcstack = getelementptr inbounds i8, ptr %thread_ptr, i64 -8 %tls_pgcstack = load ptr, ptr %tls_ppgcstack, align 8 %current_scope = getelementptr inbounds i8, ptr %tls_pgcstack, i64 -72 %0 = load ptr, ptr %current_scope, align 8 ret ptr %0 } ; Function Signature: bar() define void @julia_bar_1581() #0 { top: %jlcallframe1 = alloca [2 x ptr], align 8 %jl_nothing = load ptr, ptr @jl_nothing, align 8 store ptr @"jl_global#1583.jit", ptr %jlcallframe1, align 8 %0 = getelementptr inbounds ptr, ptr %jlcallframe1, i64 1 store ptr %jl_nothing, ptr %0, align 8 %jl_f_throw_methoderror_ret = call nonnull ptr @jl_f_throw_methoderror(ptr null, ptr nonnull %jlcallframe1, i32 2) call void @llvm.trap() unreachable } ``` * a minor improvement for EA-based `:effect_free`-ness refinement (#55796) * fix #52986, regression in `@doc` of macro without REPL loaded (#55795) fix #52986 * Assume that docstring code with no lang is julia (#55465) * Broadcast binary ops involving strided triangular (#55798) Currently, we evaluate expressions like `(A::UpperTriangular) + (B::UpperTriangular)` using broadcasting if both `A` and `B` have strided parents, and forward the summation to the parents otherwise. This PR changes this to use broadcasting if either of the two has a strided parent. This avoids accessing the parent corresponding to the structural zero elements, as the index might not be initialized. Fixes https://github.com/JuliaLang/julia/issues/55590 This isn't a general fix, as we still sum the parents if neither is strided. However, it will address common cases. This also improves performance, as we only need to loop over one half: ```julia julia> using LinearAlgebra julia> U = UpperTriangular(zeros(100,100)); julia> B = Bidiagonal(zeros(100), zeros(99), :U); julia> @btime $U + $B; 35.530 μs (4 allocations: 78.22 KiB) # nightly 13.441 μs (4 allocations: 78.22 KiB) # This PR ``` * Reland " Avoid materializing arrays in bidiag matmul #55450" (#55777) This relands #55450 and adds tests for the failing case noted in https://github.com/JuliaLang/julia/issues/55727. The `addmul` tests that were failing earlier pass with this change. The issue in the earlier PR was that we were not exiting quickly for `iszero(alpha)` in `_bibimul!` for small matrices, and were computing the result as `C .= A * B * alpha + C * beta`. The problem with this is that if `A * B` contains `NaN`s, this propagates to `C` even if `alpha === 0.0`. This is fixed now, and the result is only computed if `!iszero(alpha)`. * move the test case added in #50174 to test/core.jl (#55811) Also renames the name of the test function to avoid name collision. * [Random] Avoid conversion to `Float32` in `Float16` sampler (#55819) * simplify the fields of `UnionSplitInfo` (#55815) xref: <https://github.com/JuliaLang/julia/pull/54972#discussion_r1766187771> * Add errorhint for nonexisting fields and properties (#55165) I played a bit with error hints and crafted this: ```julia julia> (1+2im).real ERROR: FieldError: type Complex has no field real, available fields: `re`, `im` julia> nothing.xy ERROR: FieldError: type Nothing has no field xy; Nothing has no fields at all. julia> svd(rand(2,2)).VV ERROR: FieldError: type SVD has no field VV, available fields: `U`, `S`, `Vt` Available properties: `V` ``` --------- Co-authored-by: Lilith Orion Hafner <lilithhafner@gmail.com> * Improve printing of several arguments (#55754) Following a discussion on [Discourse](https://discourse.julialang.org/t/string-optimisation-in-julia/119301/10?u=gdalle), this PR tries to improve `print` (and variants) for more than one argument. The idea is that `for` is type-unstable over the tuple `args`, while `foreach` unrolls. --------- Co-authored-by: Steven G. Johnson <stevenj@mit.edu> * Markdown: support `parse(::AbstractString)` (#55747) `Markdown.parse` is documented to accept `AbstractString` but it was implemented by calling `IOBuffer` on the string argument. `IOBuffer`, however, is documented only for `String` arguments. This commit changes the current `parse(::AbstractString)` to `parse(::String)` and implements `parse(::AbstractString)` by converting the argument to `String`. Now, even `LazyString`s can be parsed to Markdown representation. Fixes #55732 * better error for esc outside of macro expansion (#55797) fixes #55788 --------- Co-authored-by: Jeff Bezanson <jeff.bezanson@gmail.com> * allow kronecker product between recursive triangular matrices (#55527) Using the recently introduced recursive `zero` I can remove the specialization to `<:Number` as @dkarrasch wanted to do in #54413. --------- Co-authored-by: Jishnu Bhattacharya <jishnub.github@gmail.com> * [Dates] Make test more robust against non-UTC timezones (#55829) `%M` is the format specifier for the minutes, not the month (which should be `%m`), and it was used twice. Also, on macOS `Libc.strptime` internally calls `mktime` which depends on the local timezone. We now temporarily set `TZ=UTC` to avoid depending on the local timezone. Fix #55827. * 🤖 [master] Bump the Pkg stdlib from 308f9d32f to ef9f76c17 (#55838) * lmul!/rmul! for banded matrices (#55823) This adds fast methods for `lmul!` and `rmul!` between banded matrices and numbers. Performance impact: ```julia julia> T = Tridiagonal(rand(999), rand(1000), rand(999)); julia> @btime rmul!($T, 0.2); 4.686 ms (0 allocations: 0 bytes) # nightly v"1.12.0-DEV.1225" 669.355 ns (0 allocations: 0 bytes) # this PR ``` * Specialize indexing triangular matrices with BandIndex (#55644) With this, certain indexing operations involving a `BandIndex` may be evaluated as constants. This isn't used directly presently, but might allow for more performant broadcasting in the future. With this, ```julia julia> n = 3; T = Tridiagonal(rand(n-1), rand(n), rand(n-1)); julia> @code_warntype ((T,j) -> UpperTriangular(T)[LinearAlgebra.BandIndex(2,j)])(T, 1) MethodInstance for (::var"#17#18")(::Tridiagonal{Float64, Vector{Float64}}, ::Int64) from (::var"#17#18")(T, j) @ Main REPL[12]:1 Arguments #self#::Core.Const(var"#17#18"()) T::Tridiagonal{Float64, Vector{Float64}} j::Int64 Body::Float64 1 ─ %1 = Main.UpperTriangular(T)::UpperTriangular{Float64, Tridiagonal{Float64, Vector{Float64}}} │ %2 = LinearAlgebra.BandIndex::Core.Const(LinearAlgebra.BandIndex) │ %3 = (%2)(2, j)::Core.PartialStruct(LinearAlgebra.BandIndex, Any[Core.Const(2), Int64]) │ %4 = Base.getindex(%1, %3)::Core.Const(0.0) └── return %4 ``` The indexing operation may be evaluated at compile-time, as the band index is constant-propagated. * Replace regex package module checks with actual code checks (#55824) Fixes https://github.com/JuliaLang/julia/issues/55792 Replaces https://github.com/JuliaLang/julia/pull/55822 Improves what https://github.com/JuliaLang/julia/pull/51635 was trying to do i.e. ``` ERROR: LoadError: `using/import Printf` outside of a Module detected. Importing a package outside of a module is not allowed during package precompilation. ``` * fall back to slower stat filesize if optimized filesize fails (#55641) * Use "index" instead of "subscript" to refer to indexing in performance tips (#55846) * privatize annotated string API, take two (#55845) https://github.com/JuliaLang/julia/pull/55453 is stuck on StyledStrings and Base documentation being entangled and there isn't a good way to have the documentation of Base types / methods live in an stdlib. This is a stop gap solution to finally be able to move forwards with 1.11. * 🤖 [master] Bump the Downloads stdlib from 1061ecc to 89d3c7d (#55854) Stdlib: Downloads URL: https://github.com/JuliaLang/Downloads.jl.git Stdlib branch: master Julia branch: master Old commit: 1061ecc New commit: 89d3c7d Julia version: 1.12.0-DEV Downloads version: 1.6.0(It's okay that it doesn't match) Bump invoked by: @KristofferC Powered by: [BumpStdlibs.jl](https://github.com/JuliaLang/BumpStdlibs.jl) Diff: https://github.com/JuliaLang/Downloads.jl/compare/1061ecc377a053fce0df94e1a19e5260f7c030f5...89d3c7dded535a77551e763a437a6d31e4d9bf84 ``` $ git log --oneline 1061ecc..89d3c7d 89d3c7d fix cancelling upload requests (#259) df33406 gracefully cancel a request (#256) ``` Co-authored-by: Dilum Aluthge <dilum@aluthge.com> * docs: Small edits to noteworthy differences (#55852) - The first line edit changes it so that the Julia example goes first, not the Python example, keeping with the general flow of the lines above. - The second adds a "the" that is missing. * Add filesystem func to transform a path to a URI (#55454) In a few places across Base and the stdlib, we emit paths that we like people to be able to click on in their terminal and editor. Up to this point, we have relied on auto-filepath detection, but this does not allow for alternative link text, such as contracted paths. Doing so (via OSC 8 terminal links for example) requires filepath URI encoding. This functionality was previously part of a PR modifying stacktrace printing (#51816), but after that became held up for unrelated reasons and another PR appeared that would benefit from this utility (#55335), I've split out this functionality so it can be used before the stacktrace printing PR is resolved. * constrain the path argument of `include` functions to `AbstractString` (#55466) Each `Module` defined with `module` automatically gets an `include` function with two methods. Each of those two methods takes a file path as its last argument. Even though the path argument is unconstrained by dispatch, it's documented as constrained with `::AbstractString`: https://docs.julialang.org/en/v1.11-dev/base/base/#include Furthermore, I think that any invocation of `include` with a non-`AbstractString` path will necessarily throw a `MethodError` eventually. Thus this change should be harmless. Adding the type constraint to the path argument is an improvement because any possible exception would be thrown earlier than before. Apart from modules defined with `module`, the same issue is present with the anonymous modules created by `evalfile`, which is also addressed. Sidenote: `evalfile` seems to be completely untested apart from the test added here. Co-authored-by: Florian <florian.atteneder@gmail.com> * Mmap: fix grow! for non file IOs (#55849) Fixes https://github.com/JuliaLang/julia/issues/54203 Requires #55641 Based on https://github.com/JuliaLang/julia/pull/55641#issuecomment-2334162489 cc. @JakeZw @ronisbr --------- Co-authored-by: Jameson Nash <vtjnash@gmail.com> * codegen: split gc roots from other bits on stack (#55767) In order to help avoid memory provenance issues, and better utilize stack space (somewhat), and use FCA less, change the preferred representation of an immutable object to be a pair of `<packed-data,roots>` values. This packing requires some care at the boundaries and if the expected field alignment exceeds that of a pointer. The change is expected to eventually make codegen more flexible at representing unions of values with both bits and pointer regions. Eventually we can also have someone improve the late-gc-lowering pass to take advantage of this increased information accuracy, but currently it will not be any better than before at laying out the frame. * Refactoring to be considered before adding MMTk * Removing jl_gc_notify_image_load, since it's a new function and not part of the refactoring * Moving gc_enable code to gc-common.c * Addressing PR comments * Push resolution of merge conflict * Removing jl_gc_mark_queue_obj_explicit extern definition from scheduler.c * Don't need the getter function since it's possible to use jl_small_typeof directly * WIP: Adding support for MMTk/Immix * Refactoring to be considered before adding MMTk * Adding fastpath allocation * Fixing removed newlines * Refactoring to be considered before adding MMTk * Adding a few comments; Moving some functions to be closer together * Fixing merge conflicts * Applying changes from refactoring before adding MMTk * Update TaskLocalRNG docstring according to #49110 (#55863) Since #49110, which is included in 1.10 and 1.11, spawning a task no longer advances the parent task's RNG state, so this statement in the docs was incorrect. * Root globals in toplevel exprs (#54433) This fixes #54422, the code here assumes that top level exprs are always rooted, but I don't see that referenced anywhere else, or guaranteed, so conservatively always root objects that show up in code. * codegen: fix alignment typos (#55880) So easy to type jl_datatype_align to get the natural alignment instead of julia_alignment to get the actual alignment. This should fix the Revise workload. Change is visible with ``` julia> code_llvm(Random.XoshiroSimd.forkRand, (Random.TaskLocalRNG, Base.Val{8})) ``` * Fix some corner cases of `isapprox` with unsigned integers (#55828) * 🤖 [master] Bump the Pkg stdlib from ef9f76c17 to 51d4910c1 (#55896) * Profile: fix order of fields in heapsnapshot & improve formatting (#55890) * Profile: Improve generation of clickable terminal links (#55857) * inference: add missing `TypeVar` handling for `instanceof_tfunc` (#55884) I thought these sort of problems had been addressed by d60f92c, but it seems some were missed. Specifically, `t.a` and `t.b` from `t::Union` could be `TypeVar`, and if they are passed to a subroutine or recursed without being unwrapped or rewrapped, errors like JuliaLang/julia#55882 could occur. This commit resolves the issue by calling `unwraptv` in the `Union` handling within `instanceof_tfunc`. I also found a similar issue inside `nfields_tfunc`, so that has also been fixed, and test cases have been added. While I haven't been able to make up a test case specifically for the fix in `instanceof_tfunc`, I have confirmed that this commit certainly fixes the issue reported in JuliaLang/julia#55882. - fixes JuliaLang/julia#55882 * Install terminfo data under /usr/share/julia (#55881) Just like all other libraries, we don't want internal Julia files to mess with system files. Introduced by https://github.com/JuliaLang/julia/pull/55411. * expose metric to report reasons why full GCs were triggered (#55826) Additional GC observability tool. This will help us to diagnose why some of our servers are triggering so many full GCs in certain circumstances. * Revert "Improve printing of several arguments" (#55894) Reverts JuliaLang/julia#55754 as it overrode some performance heuristics which appeared to be giving a significant gain/loss in performance: Closes https://github.com/JuliaLang/julia/issues/55893 * Do not trigger deprecation warnings in `Test.detect_ambiguities` and `Test.detect_unbound_args` (#55869) #55868 * do not intentionally suppress errors in precompile script from being reported or failing the result (#55909) I was slightly annoying that the build was set up to succeed if this step failed, so I removed the error suppression and fixed up the script slightly * Remove eigvecs method for SymTridiagonal (#55903) The fallback method does the same, so this specialized method isn't necessary * add --trim option for generating smaller binaries (#55047) This adds a command line option `--trim` that builds images where code is only included if it is statically reachable from methods marked using the new function `entrypoint`. Compile-time errors are given for call sites that are too dynamic to allow trimming the call graph (however there is an `unsafe` option if you want to try building anyway to see what happens). The PR has two other components. One is changes to Base that generally allow more code to be compiled in this mode. These changes will either be merged in separate PRs or moved to a separate part of the workflow (where we will build a custom system image for this purpose). The branch is set up this way to make it easy to check out and try the functionality. The other component is everything in the `juliac/` directory, which implements a compiler driver script based on this new option, along with some examples and tests. This will eventually become a package "app" that depends on PackageCompiler and provides a CLI for all of this stuff, so it will not be merged here. To try an example: ``` julia contrib/juliac.jl --output-exe hello --trim test/trimming/hello.jl ``` When stripped the resulting executable is currently about 900kb on my machine. Also includes a lot of work by @topolarity --------- Co-authored-by: Gabriel Baraldi <baraldigabriel@gmail.com> Co-authored-by: Tim Holy <tim.holy@gmail.com> Co-authored-by: Cody Tapscott <topolarity@tapscott.me> * fix rawbigints OOB issues (#55917) Fixes issues introduced in #50691 and found in #55906: * use `@inbounds` and `@boundscheck` macros in rawbigints, for catching OOB with `--check-bounds=yes` * fix OOB in `truncate` * prevent loading other extensions when precompiling an extension (#55589) The current way of loading extensions when precompiling an extension very easily leads to cycles. For example, if you have more than one extension and you happen to transitively depend on the triggers of one of your extensions you will immediately hit a cycle where the extensions will try to load each other indefinitely. This is an issue because you cannot directly influence your transitive dependency graph so from this p.o.v the current system of loading extension is "unsound". The test added here checks this scenario and we can now precompile and load it without any warnings or issues. Would have made https://github.com/JuliaLang/julia/issues/55517 a non issue. Fixes https://github.com/JuliaLang/julia/issues/55557 --------- Co-authored-by: KristofferC <kristoffer.carlsson@juliacomputing.com> * TOML: Avoid type-pirating `Base.TOML.Parser` (#55892) Since stdlibs can be duplicated but Base never is, `Base.require_stdlib` makes type piracy even more complicated than it normally would be. To adapt, this changes `TOML.Parser` to be a type defined by the TOML stdlib, so that we can define methods on it without committing type-piracy and avoid problems like Pkg.jl#4017 Resolves https://github.com/JuliaLang/Pkg.jl/issues/4017#issuecomment-2377589989 * [FileWatching] fix PollingFileWatcher design and add workaround for a stat bug What started as an innocent fix for a stat bug on Apple (#48667) turned into a full blown investigation into the design problems with the libuv backend for PollingFileWatcher, and writing my own implementation of it instead which could avoid those singled-threaded concurrency bugs. * [FileWatching] fix FileMonitor similarly and improve pidfile reliability Previously pidfile used the same poll_interval as sleep to detect if this code made any concurrency mistakes, but we do not really need to do that once FileMonitor is fixed to be reliable in the presence of parallel concurrency (instead of using watch_file). * [FileWatching] reorganize file and add docs * Add `--trace-dispatch` (#55848) * relocation: account for trailing path separator in depot paths (#55355) Fixes #55340 * change compiler to be stackless (#55575) This change ensures the compiler uses very little stack, making it compatible with running on any arbitrary system stack size and depths much more reliably. It also could be further modified now to easily add various forms of pause-able/resumable inference, since there is no implicit state on the stack--everything is local and explicit now. Whereas before, less than 900 frames would crash in less than a second: ``` $ time ./julia -e 'f(::Val{N}) where {N} = N <= 0 ? 0 : f(Val(N - 1)); f(Val(1000))' Warning: detected a stack overflow; program state may be corrupted, so further execution might be unreliable. Internal error: during type inference of f(Base.Val{1000}) Encountered stack overflow. This might be caused by recursion over very long tuples or argument lists. [23763] signal 6: Abort trap: 6 in expression starting at none:1 __pthread_kill at /usr/lib/system/libsystem_kernel.dylib (unknown line) Allocations: 1 (Pool: 1; Big: 0); GC: 0 Abort trap: 6 real 0m0.233s user 0m0.165s sys 0m0.049s ```` Now: it is effectively unlimited, as long as you are willing to wait for it: ``` $ time ./julia -e 'f(::Val{N}) where {N} = N <= 0 ? 0 : f(Val(N - 1)); f(Val(50000))' info: inference of f(Base.Val{50000}) from f(Base.Val{N}) where {N} exceeding 2500 frames (may be slow). info: inference of f(Base.Val{50000}) from f(Base.Val{N}) where {N} exceeding 5000 frames (may be slow). info: inference of f(Base.Val{50000}) from f(Base.Val{N}) where {N} exceeding 10000 frames (may be slow). info: inference of f(Base.Val{50000}) from f(Base.Val{N}) where {N} exceeding 20000 frames (may be slow). info: inference of f(Base.Val{50000}) from f(Base.Val{N}) where {N} exceeding 40000 frames (may be slow). real 7m4.988s $ time ./julia -e 'f(::Val{N}) where {N} = N <= 0 ? 0 : f(Val(N - 1)); f(Val(1000))' real 0m0.214s user 0m0.164s sys 0m0.044s $ time ./julia -e '@noinline f(::Val{N}) where {N} = N <= 0 ? GC.safepoint() : f(Val(N - 1)); f(Val(5000))' info: inference of f(Base.Val{5000}) from f(Base.Val{N}) where {N} exceeding 2500 frames (may be slow). info: inference of f(Base.Val{5000}) from f(Base.Val{N}) where {N} exceeding 5000 frames (may be slow). real 0m8.609s user 0m8.358s sys 0m0.240s ``` * optimizer: simplify the finalizer inlining pass a bit (#55934) Minor adjustments have been made to the algorithm of the finalizer inlining pass. Previously, it required that the finalizer registration dominate all uses, but this is not always necessary as far as the finalizer inlining point dominates all the uses. So the check has been relaxed. Other minor fixes have been made as well, but their importance is low. * Limit `@inbounds` to indexing in the dual-iterator branch in `copyto_unaliased!` (#55919) This simplifies the `copyto_unalised!` implementation where the source and destination have different `IndexStyle`s, and limits the `@inbounds` to only the indexing operation. In particular, the iteration over `eachindex(dest)` is not marked as `@inbounds` anymore. This seems to help with performance when the destination uses Cartesian indexing. Reduced implementation of the branch: ```julia function copyto_proposed!(dest, src) axes(dest) == axes(src) || throw(ArgumentError("incompatible sizes")) iterdest, itersrc = eachindex(dest), eachindex(src) for (destind, srcind) in zip(iterdest, itersrc) @inbounds dest[destind] = src[srcind] end dest end function copyto_current!(dest, src) axes(dest) == axes(src) || throw(ArgumentError("incompatible sizes")) iterdest, itersrc = eachindex(dest), eachindex(src) ret = iterate(iterdest) @inbounds for a in src idx, state = ret::NTuple{2,Any} dest[idx] = a ret = iterate(iterdest, state) end dest end function copyto_current_limitinbounds!(dest, src) axes(dest) == axes(src) || throw(ArgumentError("incompatible sizes")) iterdest, itersrc = eachindex(dest), eachindex(src) ret = iterate(iterdest) for isrc in itersrc idx, state = ret::NTuple{2,Any} @inbounds dest[idx] = src[isrc] ret = iterate(iterdest, state) end dest end ``` ```julia julia> a = zeros(40000,4000); b = rand(size(a)...); julia> av = view(a, UnitRange.(axes(a))...); julia> @btime copyto_current!($av, $b); 617.704 ms (0 allocations: 0 bytes) julia> @btime copyto_current_limitinbounds!($av, $b); 304.146 ms (0 allocations: 0 bytes) julia> @btime copyto_proposed!($av, $b); 240.217 ms (0 allocations: 0 bytes) julia> versioninfo() Julia Version 1.12.0-DEV.1260 Commit 4a4ca9c8152 (2024-09-28 01:49 UTC) Build Info: Official https://julialang.org release Platform Info: OS: Linux (x86_64-linux-gnu) CPU: 8 × Intel(R) Core(TM) i5-10310U CPU @ 1.70GHz WORD_SIZE: 64 LLVM: libLLVM-18.1.7 (ORCJIT, skylake) Threads: 1 default, 0 interactive, 1 GC (on 8 virtual cores) Environment: JULIA_EDITOR = subl ``` I'm not quite certain why the proposed implementation here (`copyto_proposed!`) is even faster than `copyto_current_limitinbounds!`. In any case, `copyto_proposed!` is easier to read, so I'm not complaining. This fixes https://github.com/JuliaLang/julia/issues/53158 * Strong zero in Diagonal triple multiplication (#55927) Currently, triple multiplication with a `LinearAlgebra.BandedMatrix` sandwiched between two `Diagonal`s isn't associative, as this is implemented using broadcasting, which doesn't assume a strong zero, whereas the two-term matrix multiplication does. ```julia julia> D = Diagonal(StepRangeLen(NaN, 0, 3)); julia> B = Bidiagonal(1:3, 1:2, :U); julia> D * B * D 3×3 Matrix{Float64}: NaN NaN NaN NaN NaN NaN NaN NaN NaN julia> (D * B) * D 3×3 Bidiagonal{Float64, Vector{Float64}}: NaN NaN ⋅ ⋅ NaN NaN ⋅ ⋅ NaN julia> D * (B * D) 3×3 Bidiagonal{Float64, Vector{Float64}}: NaN NaN ⋅ ⋅ NaN NaN ⋅ ⋅ NaN ``` This PR ensures that the 3-term multiplication is evaluated as a sequence of two-term multiplications, which fixes this issue. This also improves performance, as only the bands need to be evaluated now. ```julia julia> D = Diagonal(1:1000); B = Bidiagonal(1:1000, 1:999, :U); julia> @btime $D * $B * $D; 656.364 μs (11 allocations: 7.63 MiB) # v"1.12.0-DEV.1262" 2.483 μs (12 allocations: 31.50 KiB) # This PR ``` * Fix dispatch on `alg` in Float16 Hermitian eigen (#55928) Currently, ```julia julia> using LinearAlgebra julia> A = Hermitian(reshape(Float16[1:16;], 4, 4)); julia> eigen(A).values |> typeof Vector{Float16} (alias for Array{Float16, 1}) julia> eigen(A, LinearAlgebra.QRIteration()).values |> typeof Vector{Float32} (alias for Array{Float32, 1}) ``` This PR moves the specialization on the `eltype` to an internal method, so that firstly all `alg`s dispatch to that method, and secondly, there are no ambiguities introduce by specializing the top-level `eigen`. The latter currently causes test failures in `StaticArrays` (https://github.com/JuliaArrays/StaticArrays.jl/actions/runs/11092206012/job/30816955210?pr=1279), and should be fixed by this PR. * Remove specialized `ishermitian` method for `Diagonal{<:Real}` (#55948) The fallback method for `Diagonal{<:Number}` handles this already by checking that the `diag` is real, so we don't need this additional specialization. * Fix logic in `?` docstring example (#55945) * fix `unwrap_macrocalls` (#55950) The implementation of `unwrap_macrocalls` has assumed that what `:macrocall` wraps is always an `Expr` object, but that is not necessarily correct: ```julia julia> Base.@assume_effects :nothrow @show 42 ERROR: LoadError: TypeError: in typeassert, expected Expr, got a value of type Int64 Stacktrace: [1] unwrap_macrocalls(ex::Expr) @ Base ./expr.jl:906 [2] var"@assume_effects"(__source__::LineNumberNode, __module__::Module, args::Vararg{Any}) @ Base ./expr.jl:756 in expression starting at REPL[1]:1 ``` This commit addresses this issue. * make faster BigFloats (#55906) We can coalesce the two required allocations for the MFPR BigFloat API design into one allocation, hopefully giving a easy performance boost. It would have been slightly easier and more efficient if MPFR BigFloat was already a VLA instead of containing a pointer here, but that does not prevent the optimization. * Add propagate_inbounds_meta to atomic genericmemory ops (#55902) `memoryref(mem, i)` will otherwise emit a boundscheck. ``` ; │ @ /home/vchuravy/WorkstealingQueues/src/CLL.jl:53 within `setindex_atomic!` @ genericmemory.jl:329 ; │┌ @ boot.jl:545 within `memoryref` %ptls_field = getelementptr inbounds i8, ptr %tls_pgcstack, i64 16 %ptls_load = load ptr, ptr %ptls_field, align 8 %"box::GenericMemoryRef" = call noalias nonnull align 8 dereferenceable(32) ptr @ijl_gc_small_alloc(ptr %ptls_load, i32 552, i32 32, i64 23456076646928) #9 %"box::GenericMemoryRef.tag_addr" = getelementptr inbounds i64, ptr %"box::GenericMemoryRef", i64 -1 store atomic i64 23456076646928, ptr %"box::GenericMemoryRef.tag_addr" unordered, align 8 store ptr %memoryref_data, ptr %"box::GenericMemoryRef", align 8 %.repack8 = getelementptr inbounds { ptr, ptr }, ptr %"box::GenericMemoryRef", i64 0, i32 1 store ptr %memoryref_mem, ptr %.repack8, align 8 call void @ijl_bounds_error_int(ptr nonnull %"box::GenericMemoryRef", i64 %7) unreachable ``` For the Julia code: ```julia function Base.setindex_atomic!(buf::WSBuffer{T}, order::Symbol, val::T, idx::Int64) where T @inbounds Base.setindex_atomic!(buf.buffer, order, val,((idx - 1) & buf.mask) + 1) end ``` from https://github.com/gbaraldi/WorkstealingQueues.jl/blob/0ebc57237cf0c90feedf99e4338577d04b67805b/src/CLL.jl#L41 * fix rounding mode in construction of `BigFloat` from pi (#55911) The default argument of the method was outdated, reading the global default rounding directly, bypassing the `ScopedValue` stuff. * fix `nonsetable_type_hint_handler` (#55962) The current implementation is wrong, causing it to display inappropriate hints like the following: ```julia julia> s = Some("foo"); julia> s[] = "bar" ERROR: MethodError: no method matching setindex!(::Some{String}, ::String) The function `setindex!` exists, but no method is defined for this combination of argument types. You attempted to index the type String, rather than an instance of the type. Make sure you create the type using its constructor: d = String([...]) rather than d = String Stacktrace: [1] top-level scope @ REPL[2]:1 ``` * REPL: make UndefVarError aware of imported modules (#55932) * fix test/staged.jl (#55967) In particular, the implementation of `overdub_generator54341` was dangerous. This fixes it up. * Explicitly store a module's location (#55963) Revise wants to know what file a module's `module` definition is in. Currently it does this by looking at the source location for the implicitly generated `eval` method. This is terrible for two reasons: 1. The method may not exist if the module is a baremodule (which is not particularly common, which is probably why we haven't seen it). 2. The fact that the implicitly generated `eval` method has this location information is an implementation detail that I'd like to get rid of (#55949). This PR adds explicit file/line info to `Module`, so that Revise doesn't have to use the hack anymore. * mergewith: add single argument example to docstring (#55964) I ran into this edge case. I though it should be documented. --------- Co-authored-by: Lilith Orion Hafner <lilithhafner@gmail.com> * [build] avoid libedit linkage and align libccalllazy* SONAMEs (#55968) While building the 1.11.0-rc4 in Homebrew[^1] in preparation for 1.11.0 release (and to confirm Sequoia successfully builds) I noticed some odd linkage for our Linux builds, which included of: 1. LLVM libraries were linking to `libedit.so`, e.g. ``` Dynamic Section: NEEDED libedit.so.0 NEEDED libz.so.1 NEEDED libzstd.so.1 NEEDED libstdc++.so.6 NEEDED libm.so.6 NEEDED libgcc_s.so.1 NEEDED libc.so.6 NEEDED ld-linux-x86-64.so.2 SONAME libLLVM-16jl.so ``` CMakeCache.txt showed ``` //Use libedit if available. LLVM_ENABLE_LIBEDIT:BOOL=ON ``` Which might be overriding `HAVE_LIBEDIT` at https://github.com/JuliaLang/llvm-project/blob/julia-release/16.x/llvm/cmake/config-ix.cmake#L222-L225. So just added `LLVM_ENABLE_LIBEDIT` 2. Wasn't sure if there was a reason for this but `libccalllazy*` had mismatched SONAME: ```console ❯ objdump -p lib/julia/libccalllazy* | rg '\.so' lib/julia/libccalllazybar.so: file format elf64-x86-64 NEEDED ccalllazyfoo.so SONAME ccalllazybar.so lib/julia/libccalllazyfoo.so: file format elf64-x86-64 SONAME ccalllazyfoo.so ``` Modifying this, but can drop if intentional. --- [^1]: https://github.com/Homebrew/homebrew-core/pull/192116 * Add missing `copy!(::AbstractMatrix, ::UniformScaling)` method (#55970) Hi everyone! First PR to Julia here. It was noticed in a Slack thread yesterday that `copy!(A, I)` doesn't work, but `copyto!(A, I)` does. This PR adds the missing method for `copy!(::AbstractMatrix, ::UniformScaling)`, which simply defers to `copyto!`, and corresponding tests. I added a `compat` notice for Julia 1.12. --------- Co-authored-by: Lilith Orion Hafner <lilithhafner@gmail.com> * Add forward progress update to NEWS.md (#54089) Closes #40009 which was left open because of the needs news tag. --------- Co-authored-by: Ian Butterworth <i.r.butterworth@gmail.com> * Fix an intermittent test failure in `core` test (#55973) The test wants to assert that `Module` is not resolved in `Main`, but other tests do resolve this identifier, so the test can fail depending on test order (and I've been seeing such failures on CI recently). Fix that by running the test in a fresh subprocess. * fix comma logic in time_print (#55977) Minor formatting fix * optimizer: fix up the inlining algorithm to use correct `nargs`/`isva` (#55976) It appears that inlining.jl was not updated in JuliaLang/julia#54341. Specifically, using `nargs`/`isva` from `mi.def::Method` in `ir_prepare_inlining!` causes the following error to occur: ```julia function generate_lambda_ex(world::UInt, source::LineNumberNode, argnames, spnames, @nospecialize body) stub = Core.GeneratedFunctionStub(identity, Core.svec(argnames...), Core.svec(spnames...)) return stub(world, source, body) end function overdubbee54341(a, b) return a + b end const overdubee_codeinfo54341 = code_lowered(overdubbee54341, Tuple{Any, Any})[1] function overdub_generator54341(world::UInt, source::LineNumberNode, selftype, fargtypes) if length(fargtypes) != 2 return generate_lambda_ex(world, source, (:overdub54341, :args), (), :(error("Wrong number of arguments"))) else return copy(overdubee_codeinfo54341) end end @eval function overdub54341(args...) $(Expr(:meta, :generated, overdub_generator54341)) $(Expr(:meta, :generated_only)) end topfunc(x) = overdub54341(x, 2) ``` ```julia julia> topfunc(1) Internal error: during type inference of topfunc(Int64) Encountered unexpected error in runtime: BoundsError(a=Array{Any, 1}(dims=(2,), mem=Memory{Any}(8, 0x10632e780)[SSAValue(2), SSAValue(3), #<null>, #<null>, #<null>, #<null>, #<null>, #<null>]), i=(3,)) throw_boundserror at ./essentials.jl:14 getindex at ./essentials.jl:909 [inlined] ssa_substitute_op! at ./compiler/ssair/inlining.jl:1798 ssa_substitute_op! at ./compiler/ssair/inlining.jl:1852 ir_inline_item! at ./compiler/ssair/inlining.jl:386 ... ``` This commit updates the abstract interpretation and inlining algorithm to use the `nargs`/`isva` values held by `CodeInfo`. Similar modifications have also been made to EscapeAnalysis.jl. @nanosoldier `runbenchmarks("inference", vs=":master")` * Add `.zed` directory to `.gitignore` (#55974) Similar to the `vscode` config directory, we may ignore the `zed` directory as well. * typeintersect: reduce unneeded allocations from `merge_env` `merge_env` and `final_merge_env` could be skipped for emptiness test or if we know there's only 1 valid Union state. * typeintersect: trunc env before nested `intersect_all` if valid. This only covers the simplest cases. We might want a full dependence analysis and keep env length minimum in the future. * `@time` actually fix time report commas & add tests (#55982) https://github.com/JuliaLang/julia/pull/55977 looked simple but wasn't quite right because of a bad pattern in the lock conflicts report section. So fix and add tests. * adjust EA to JuliaLang/julia#52527 (#55986) `Ent…

* Add filesystem func to transform a path to a URI (#55454) In a few places across Base and the stdlib, we emit paths that we like people to be able to click on in their terminal and editor. Up to this point, we have relied on auto-filepath detection, but this does not allow for alternative link text, such as contracted paths. Doing so (via OSC 8 terminal links for example) requires filepath URI encoding. This functionality was previously part of a PR modifying stacktrace printing (#51816), but after that became held up for unrelated reasons and another PR appeared that would benefit from this utility (#55335), I've split out this functionality so it can be used before the stacktrace printing PR is resolved. * constrain the path argument of `include` functions to `AbstractString` (#55466) Each `Module` defined with `module` automatically gets an `include` function with two methods. Each of those two methods takes a file path as its last argument. Even though the path argument is unconstrained by dispatch, it's documented as constrained with `::AbstractString`: https://docs.julialang.org/en/v1.11-dev/base/base/#include Furthermore, I think that any invocation of `include` with a non-`AbstractString` path will necessarily throw a `MethodError` eventually. Thus this change should be harmless. Adding the type constraint to the path argument is an improvement because any possible exception would be thrown earlier than before. Apart from modules defined with `module`, the same issue is present with the anonymous modules created by `evalfile`, which is also addressed. Sidenote: `evalfile` seems to be completely untested apart from the test added here. Co-authored-by: Florian <florian.atteneder@gmail.com> * Mmap: fix grow! for non file IOs (#55849) Fixes https://github.com/JuliaLang/julia/issues/54203 Requires #55641 Based on https://github.com/JuliaLang/julia/pull/55641#issuecomment-2334162489 cc. @JakeZw @ronisbr --------- Co-authored-by: Jameson Nash <vtjnash@gmail.com> * codegen: split gc roots from other bits on stack (#55767) In order to help avoid memory provenance issues, and better utilize stack space (somewhat), and use FCA less, change the preferred representation of an immutable object to be a pair of `<packed-data,roots>` values. This packing requires some care at the boundaries and if the expected field alignment exceeds that of a pointer. The change is expected to eventually make codegen more flexible at representing unions of values with both bits and pointer regions. Eventually we can also have someone improve the late-gc-lowering pass to take advantage of this increased information accuracy, but currently it will not be any better than before at laying out the frame. * Refactoring to be considered before adding MMTk * Removing jl_gc_notify_image_load, since it's a new function and not part of the refactoring * Moving gc_enable code to gc-common.c * Addressing PR comments * Push resolution of merge conflict * Removing jl_gc_mark_queue_obj_explicit extern definition from scheduler.c * Don't need the getter function since it's possible to use jl_small_typeof directly * WIP: Adding support for MMTk/Immix * Refactoring to be considered before adding MMTk * Adding fastpath allocation * Fixing removed newlines * Refactoring to be considered before adding MMTk * Adding a few comments; Moving some functions to be closer together * Fixing merge conflicts * Applying changes from refactoring before adding MMTk * Update TaskLocalRNG docstring according to #49110 (#55863) Since #49110, which is included in 1.10 and 1.11, spawning a task no longer advances the parent task's RNG state, so this statement in the docs was incorrect. * Root globals in toplevel exprs (#54433) This fixes #54422, the code here assumes that top level exprs are always rooted, but I don't see that referenced anywhere else, or guaranteed, so conservatively always root objects that show up in code. * codegen: fix alignment typos (#55880) So easy to type jl_datatype_align to get the natural alignment instead of julia_alignment to get the actual alignment. This should fix the Revise workload. Change is visible with ``` julia> code_llvm(Random.XoshiroSimd.forkRand, (Random.TaskLocalRNG, Base.Val{8})) ``` * Fix some corner cases of `isapprox` with unsigned integers (#55828) * 🤖 [master] Bump the Pkg stdlib from ef9f76c17 to 51d4910c1 (#55896) * Profile: fix order of fields in heapsnapshot & improve formatting (#55890) * Profile: Improve generation of clickable terminal links (#55857) * inference: add missing `TypeVar` handling for `instanceof_tfunc` (#55884) I thought these sort of problems had been addressed by d60f92c, but it seems some were missed. Specifically, `t.a` and `t.b` from `t::Union` could be `TypeVar`, and if they are passed to a subroutine or recursed without being unwrapped or rewrapped, errors like JuliaLang/julia#55882 could occur. This commit resolves the issue by calling `unwraptv` in the `Union` handling within `instanceof_tfunc`. I also found a similar issue inside `nfields_tfunc`, so that has also been fixed, and test cases have been added. While I haven't been able to make up a test case specifically for the fix in `instanceof_tfunc`, I have confirmed that this commit certainly fixes the issue reported in JuliaLang/julia#55882. - fixes JuliaLang/julia#55882 * Install terminfo data under /usr/share/julia (#55881) Just like all other libraries, we don't want internal Julia files to mess with system files. Introduced by https://github.com/JuliaLang/julia/pull/55411. * expose metric to report reasons why full GCs were triggered (#55826) Additional GC observability tool. This will help us to diagnose why some of our servers are triggering so many full GCs in certain circumstances. * Revert "Improve printing of several arguments" (#55894) Reverts JuliaLang/julia#55754 as it overrode some performance heuristics which appeared to be giving a significant gain/loss in performance: Closes https://github.com/JuliaLang/julia/issues/55893 * Do not trigger deprecation warnings in `Test.detect_ambiguities` and `Test.detect_unbound_args` (#55869) #55868 * do not intentionally suppress errors in precompile script from being reported or failing the result (#55909) I was slightly annoying that the build was set up to succeed if this step failed, so I removed the error suppression and fixed up the script slightly * Remove eigvecs method for SymTridiagonal (#55903) The fallback method does the same, so this specialized method isn't necessary * add --trim option for generating smaller binaries (#55047) This adds a command line option `--trim` that builds images where code is only included if it is statically reachable from methods marked using the new function `entrypoint`. Compile-time errors are given for call sites that are too dynamic to allow trimming the call graph (however there is an `unsafe` option if you want to try building anyway to see what happens). The PR has two other components. One is changes to Base that generally allow more code to be compiled in this mode. These changes will either be merged in separate PRs or moved to a separate part of the workflow (where we will build a custom system image for this purpose). The branch is set up this way to make it easy to check out and try the functionality. The other component is everything in the `juliac/` directory, which implements a compiler driver script based on this new option, along with some examples and tests. This will eventually become a package "app" that depends on PackageCompiler and provides a CLI for all of this stuff, so it will not be merged here. To try an example: ``` julia contrib/juliac.jl --output-exe hello --trim test/trimming/hello.jl ``` When stripped the resulting executable is currently about 900kb on my machine. Also includes a lot of work by @topolarity --------- Co-authored-by: Gabriel Baraldi <baraldigabriel@gmail.com> Co-authored-by: Tim Holy <tim.holy@gmail.com> Co-authored-by: Cody Tapscott <topolarity@tapscott.me> * fix rawbigints OOB issues (#55917) Fixes issues introduced in #50691 and found in #55906: * use `@inbounds` and `@boundscheck` macros in rawbigints, for catching OOB with `--check-bounds=yes` * fix OOB in `truncate` * prevent loading other extensions when precompiling an extension (#55589) The current way of loading extensions when precompiling an extension very easily leads to cycles. For example, if you have more than one extension and you happen to transitively depend on the triggers of one of your extensions you will immediately hit a cycle where the extensions will try to load each other indefinitely. This is an issue because you cannot directly influence your transitive dependency graph so from this p.o.v the current system of loading extension is "unsound". The test added here checks this scenario and we can now precompile and load it without any warnings or issues. Would have made https://github.com/JuliaLang/julia/issues/55517 a non issue. Fixes https://github.com/JuliaLang/julia/issues/55557 --------- Co-authored-by: KristofferC <kristoffer.carlsson@juliacomputing.com> * TOML: Avoid type-pirating `Base.TOML.Parser` (#55892) Since stdlibs can be duplicated but Base never is, `Base.require_stdlib` makes type piracy even more complicated than it normally would be. To adapt, this changes `TOML.Parser` to be a type defined by the TOML stdlib, so that we can define methods on it without committing type-piracy and avoid problems like Pkg.jl#4017 Resolves https://github.com/JuliaLang/Pkg.jl/issues/4017#issuecomment-2377589989 * [FileWatching] fix PollingFileWatcher design and add workaround for a stat bug What started as an innocent fix for a stat bug on Apple (#48667) turned into a full blown investigation into the design problems with the libuv backend for PollingFileWatcher, and writing my own implementation of it instead which could avoid those singled-threaded concurrency bugs. * [FileWatching] fix FileMonitor similarly and improve pidfile reliability Previously pidfile used the same poll_interval as sleep to detect if this code made any concurrency mistakes, but we do not really need to do that once FileMonitor is fixed to be reliable in the presence of parallel concurrency (instead of using watch_file). * [FileWatching] reorganize file and add docs * Add `--trace-dispatch` (#55848) * relocation: account for trailing path separator in depot paths (#55355) Fixes #55340 * change compiler to be stackless (#55575) This change ensures the compiler uses very little stack, making it compatible with running on any arbitrary system stack size and depths much more reliably. It also could be further modified now to easily add various forms of pause-able/resumable inference, since there is no implicit state on the stack--everything is local and explicit now. Whereas before, less than 900 frames would crash in less than a second: ``` $ time ./julia -e 'f(::Val{N}) where {N} = N <= 0 ? 0 : f(Val(N - 1)); f(Val(1000))' Warning: detected a stack overflow; program state may be corrupted, so further execution might be unreliable. Internal error: during type inference of f(Base.Val{1000}) Encountered stack overflow. This might be caused by recursion over very long tuples or argument lists. [23763] signal 6: Abort trap: 6 in expression starting at none:1 __pthread_kill at /usr/lib/system/libsystem_kernel.dylib (unknown line) Allocations: 1 (Pool: 1; Big: 0); GC: 0 Abort trap: 6 real 0m0.233s user 0m0.165s sys 0m0.049s ```` Now: it is effectively unlimited, as long as you are willing to wait for it: ``` $ time ./julia -e 'f(::Val{N}) where {N} = N <= 0 ? 0 : f(Val(N - 1)); f(Val(50000))' info: inference of f(Base.Val{50000}) from f(Base.Val{N}) where {N} exceeding 2500 frames (may be slow). info: inference of f(Base.Val{50000}) from f(Base.Val{N}) where {N} exceeding 5000 frames (may be slow). info: inference of f(Base.Val{50000}) from f(Base.Val{N}) where {N} exceeding 10000 frames (may be slow). info: inference of f(Base.Val{50000}) from f(Base.Val{N}) where {N} exceeding 20000 frames (may be slow). info: inference of f(Base.Val{50000}) from f(Base.Val{N}) where {N} exceeding 40000 frames (may be slow). real 7m4.988s $ time ./julia -e 'f(::Val{N}) where {N} = N <= 0 ? 0 : f(Val(N - 1)); f(Val(1000))' real 0m0.214s user 0m0.164s sys 0m0.044s $ time ./julia -e '@noinline f(::Val{N}) where {N} = N <= 0 ? GC.safepoint() : f(Val(N - 1)); f(Val(5000))' info: inference of f(Base.Val{5000}) from f(Base.Val{N}) where {N} exceeding 2500 frames (may be slow). info: inference of f(Base.Val{5000}) from f(Base.Val{N}) where {N} exceeding 5000 frames (may be slow). real 0m8.609s user 0m8.358s sys 0m0.240s ``` * optimizer: simplify the finalizer inlining pass a bit (#55934) Minor adjustments have been made to the algorithm of the finalizer inlining pass. Previously, it required that the finalizer registration dominate all uses, but this is not always necessary as far as the finalizer inlining point dominates all the uses. So the check has been relaxed. Other minor fixes have been made as well, but their importance is low. * Limit `@inbounds` to indexing in the dual-iterator branch in `copyto_unaliased!` (#55919) This simplifies the `copyto_unalised!` implementation where the source and destination have different `IndexStyle`s, and limits the `@inbounds` to only the indexing operation. In particular, the iteration over `eachindex(dest)` is not marked as `@inbounds` anymore. This seems to help with performance when the destination uses Cartesian indexing. Reduced implementation of the branch: ```julia function copyto_proposed!(dest, src) axes(dest) == axes(src) || throw(ArgumentError("incompatible sizes")) iterdest, itersrc = eachindex(dest), eachindex(src) for (destind, srcind) in zip(iterdest, itersrc) @inbounds dest[destind] = src[srcind] end dest end function copyto_current!(dest, src) axes(dest) == axes(src) || throw(ArgumentError("incompatible sizes")) iterdest, itersrc = eachindex(dest), eachindex(src) ret = iterate(iterdest) @inbounds for a in src idx, state = ret::NTuple{2,Any} dest[idx] = a ret = iterate(iterdest, state) end dest end function copyto_current_limitinbounds!(dest, src) axes(dest) == axes(src) || throw(ArgumentError("incompatible sizes")) iterdest, itersrc = eachindex(dest), eachindex(src) ret = iterate(iterdest) for isrc in itersrc idx, state = ret::NTuple{2,Any} @inbounds dest[idx] = src[isrc] ret = iterate(iterdest, state) end dest end ``` ```julia julia> a = zeros(40000,4000); b = rand(size(a)...); julia> av = view(a, UnitRange.(axes(a))...); julia> @btime copyto_current!($av, $b); 617.704 ms (0 allocations: 0 bytes) julia> @btime copyto_current_limitinbounds!($av, $b); 304.146 ms (0 allocations: 0 bytes) julia> @btime copyto_proposed!($av, $b); 240.217 ms (0 allocations: 0 bytes) julia> versioninfo() Julia Version 1.12.0-DEV.1260 Commit 4a4ca9c8152 (2024-09-28 01:49 UTC) Build Info: Official https://julialang.org release Platform Info: OS: Linux (x86_64-linux-gnu) CPU: 8 × Intel(R) Core(TM) i5-10310U CPU @ 1.70GHz WORD_SIZE: 64 LLVM: libLLVM-18.1.7 (ORCJIT, skylake) Threads: 1 default, 0 interactive, 1 GC (on 8 virtual cores) Environment: JULIA_EDITOR = subl ``` I'm not quite certain why the proposed implementation here (`copyto_proposed!`) is even faster than `copyto_current_limitinbounds!`. In any case, `copyto_proposed!` is easier to read, so I'm not complaining. This fixes https://github.com/JuliaLang/julia/issues/53158 * Strong zero in Diagonal triple multiplication (#55927) Currently, triple multiplication with a `LinearAlgebra.BandedMatrix` sandwiched between two `Diagonal`s isn't associative, as this is implemented using broadcasting, which doesn't assume a strong zero, whereas the two-term matrix multiplication does. ```julia julia> D = Diagonal(StepRangeLen(NaN, 0, 3)); julia> B = Bidiagonal(1:3, 1:2, :U); julia> D * B * D 3×3 Matrix{Float64}: NaN NaN NaN NaN NaN NaN NaN NaN NaN julia> (D * B) * D 3×3 Bidiagonal{Float64, Vector{Float64}}: NaN NaN ⋅ ⋅ NaN NaN ⋅ ⋅ NaN julia> D * (B * D) 3×3 Bidiagonal{Float64, Vector{Float64}}: NaN NaN ⋅ ⋅ NaN NaN ⋅ ⋅ NaN ``` This PR ensures that the 3-term multiplication is evaluated as a sequence of two-term multiplications, which fixes this issue. This also improves performance, as only the bands need to be evaluated now. ```julia julia> D = Diagonal(1:1000); B = Bidiagonal(1:1000, 1:999, :U); julia> @btime $D * $B * $D; 656.364 μs (11 allocations: 7.63 MiB) # v"1.12.0-DEV.1262" 2.483 μs (12 allocations: 31.50 KiB) # This PR ``` * Fix dispatch on `alg` in Float16 Hermitian eigen (#55928) Currently, ```julia julia> using LinearAlgebra julia> A = Hermitian(reshape(Float16[1:16;], 4, 4)); julia> eigen(A).values |> typeof Vector{Float16} (alias for Array{Float16, 1}) julia> eigen(A, LinearAlgebra.QRIteration()).values |> typeof Vector{Float32} (alias for Array{Float32, 1}) ``` This PR moves the specialization on the `eltype` to an internal method, so that firstly all `alg`s dispatch to that method, and secondly, there are no ambiguities introduce by specializing the top-level `eigen`. The latter currently causes test failures in `StaticArrays` (https://github.com/JuliaArrays/StaticArrays.jl/actions/runs/11092206012/job/30816955210?pr=1279), and should be fixed by this PR. * Remove specialized `ishermitian` method for `Diagonal{<:Real}` (#55948) The fallback method for `Diagonal{<:Number}` handles this already by checking that the `diag` is real, so we don't need this additional specialization. * Fix logic in `?` docstring example (#55945) * fix `unwrap_macrocalls` (#55950) The implementation of `unwrap_macrocalls` has assumed that what `:macrocall` wraps is always an `Expr` object, but that is not necessarily correct: ```julia julia> Base.@assume_effects :nothrow @show 42 ERROR: LoadError: TypeError: in typeassert, expected Expr, got a value of type Int64 Stacktrace: [1] unwrap_macrocalls(ex::Expr) @ Base ./expr.jl:906 [2] var"@assume_effects"(__source__::LineNumberNode, __module__::Module, args::Vararg{Any}) @ Base ./expr.jl:756 in expression starting at REPL[1]:1 ``` This commit addresses this issue. * make faster BigFloats (#55906) We can coalesce the two required allocations for the MFPR BigFloat API design into one allocation, hopefully giving a easy performance boost. It would have been slightly easier and more efficient if MPFR BigFloat was already a VLA instead of containing a pointer here, but that does not prevent the optimization. * Add propagate_inbounds_meta to atomic genericmemory ops (#55902) `memoryref(mem, i)` will otherwise emit a boundscheck. ``` ; │ @ /home/vchuravy/WorkstealingQueues/src/CLL.jl:53 within `setindex_atomic!` @ genericmemory.jl:329 ; │┌ @ boot.jl:545 within `memoryref` %ptls_field = getelementptr inbounds i8, ptr %tls_pgcstack, i64 16 %ptls_load = load ptr, ptr %ptls_field, align 8 %"box::GenericMemoryRef" = call noalias nonnull align 8 dereferenceable(32) ptr @ijl_gc_small_alloc(ptr %ptls_load, i32 552, i32 32, i64 23456076646928) #9 %"box::GenericMemoryRef.tag_addr" = getelementptr inbounds i64, ptr %"box::GenericMemoryRef", i64 -1 store atomic i64 23456076646928, ptr %"box::GenericMemoryRef.tag_addr" unordered, align 8 store ptr %memoryref_data, ptr %"box::GenericMemoryRef", align 8 %.repack8 = getelementptr inbounds { ptr, ptr }, ptr %"box::GenericMemoryRef", i64 0, i32 1 store ptr %memoryref_mem, ptr %.repack8, align 8 call void @ijl_bounds_error_int(ptr nonnull %"box::GenericMemoryRef", i64 %7) unreachable ``` For the Julia code: ```julia function Base.setindex_atomic!(buf::WSBuffer{T}, order::Symbol, val::T, idx::Int64) where T @inbounds Base.setindex_atomic!(buf.buffer, order, val,((idx - 1) & buf.mask) + 1) end ``` from https://github.com/gbaraldi/WorkstealingQueues.jl/blob/0ebc57237cf0c90feedf99e4338577d04b67805b/src/CLL.jl#L41 * fix rounding mode in construction of `BigFloat` from pi (#55911) The default argument of the method was outdated, reading the global default rounding directly, bypassing the `ScopedValue` stuff. * fix `nonsetable_type_hint_handler` (#55962) The current implementation is wrong, causing it to display inappropriate hints like the following: ```julia julia> s = Some("foo"); julia> s[] = "bar" ERROR: MethodError: no method matching setindex!(::Some{String}, ::String) The function `setindex!` exists, but no method is defined for this combination of argument types. You attempted to index the type String, rather than an instance of the type. Make sure you create the type using its constructor: d = String([...]) rather than d = String Stacktrace: [1] top-level scope @ REPL[2]:1 ``` * REPL: make UndefVarError aware of imported modules (#55932) * fix test/staged.jl (#55967) In particular, the implementation of `overdub_generator54341` was dangerous. This fixes it up. * Explicitly store a module's location (#55963) Revise wants to know what file a module's `module` definition is in. Currently it does this by looking at the source location for the implicitly generated `eval` method. This is terrible for two reasons: 1. The method may not exist if the module is a baremodule (which is not particularly common, which is probably why we haven't seen it). 2. The fact that the implicitly generated `eval` method has this location information is an implementation detail that I'd like to get rid of (#55949). This PR adds explicit file/line info to `Module`, so that Revise doesn't have to use the hack anymore. * mergewith: add single argument example to docstring (#55964) I ran into this edge case. I though it should be documented. --------- Co-authored-by: Lilith Orion Hafner <lilithhafner@gmail.com> * [build] avoid libedit linkage and align libccalllazy* SONAMEs (#55968) While building the 1.11.0-rc4 in Homebrew[^1] in preparation for 1.11.0 release (and to confirm Sequoia successfully builds) I noticed some odd linkage for our Linux builds, which included of: 1. LLVM libraries were linking to `libedit.so`, e.g. ``` Dynamic Section: NEEDED libedit.so.0 NEEDED libz.so.1 NEEDED libzstd.so.1 NEEDED libstdc++.so.6 NEEDED libm.so.6 NEEDED libgcc_s.so.1 NEEDED libc.so.6 NEEDED ld-linux-x86-64.so.2 SONAME libLLVM-16jl.so ``` CMakeCache.txt showed ``` //Use libedit if available. LLVM_ENABLE_LIBEDIT:BOOL=ON ``` Which might be overriding `HAVE_LIBEDIT` at https://github.com/JuliaLang/llvm-project/blob/julia-release/16.x/llvm/cmake/config-ix.cmake#L222-L225. So just added `LLVM_ENABLE_LIBEDIT` 2. Wasn't sure if there was a reason for this but `libccalllazy*` had mismatched SONAME: ```console ❯ objdump -p lib/julia/libccalllazy* | rg '\.so' lib/julia/libccalllazybar.so: file format elf64-x86-64 NEEDED ccalllazyfoo.so SONAME ccalllazybar.so lib/julia/libccalllazyfoo.so: file format elf64-x86-64 SONAME ccalllazyfoo.so ``` Modifying this, but can drop if intentional. --- [^1]: https://github.com/Homebrew/homebrew-core/pull/192116 * Add missing `copy!(::AbstractMatrix, ::UniformScaling)` method (#55970) Hi everyone! First PR to Julia here. It was noticed in a Slack thread yesterday that `copy!(A, I)` doesn't work, but `copyto!(A, I)` does. This PR adds the missing method for `copy!(::AbstractMatrix, ::UniformScaling)`, which simply defers to `copyto!`, and corresponding tests. I added a `compat` notice for Julia 1.12. --------- Co-authored-by: Lilith Orion Hafner <lilithhafner@gmail.com> * Add forward progress update to NEWS.md (#54089) Closes #40009 which was left open because of the needs news tag. --------- Co-authored-by: Ian Butterworth <i.r.butterworth@gmail.com> * Fix an intermittent test failure in `core` test (#55973) The test wants to assert that `Module` is not resolved in `Main`, but other tests do resolve this identifier, so the test can fail depending on test order (and I've been seeing such failures on CI recently). Fix that by running the test in a fresh subprocess. * fix comma logic in time_print (#55977) Minor formatting fix * optimizer: fix up the inlining algorithm to use correct `nargs`/`isva` (#55976) It appears that inlining.jl was not updated in JuliaLang/julia#54341. Specifically, using `nargs`/`isva` from `mi.def::Method` in `ir_prepare_inlining!` causes the following error to occur: ```julia function generate_lambda_ex(world::UInt, source::LineNumberNode, argnames, spnames, @nospecialize body) stub = Core.GeneratedFunctionStub(identity, Core.svec(argnames...), Core.svec(spnames...)) return stub(world, source, body) end function overdubbee54341(a, b) return a + b end const overdubee_codeinfo54341 = code_lowered(overdubbee54341, Tuple{Any, Any})[1] function overdub_generator54341(world::UInt, source::LineNumberNode, selftype, fargtypes) if length(fargtypes) != 2 return generate_lambda_ex(world, source, (:overdub54341, :args), (), :(error("Wrong number of arguments"))) else return copy(overdubee_codeinfo54341) end end @eval function overdub54341(args...) $(Expr(:meta, :generated, overdub_generator54341)) $(Expr(:meta, :generated_only)) end topfunc(x) = overdub54341(x, 2) ``` ```julia julia> topfunc(1) Internal error: during type inference of topfunc(Int64) Encountered unexpected error in runtime: BoundsError(a=Array{Any, 1}(dims=(2,), mem=Memory{Any}(8, 0x10632e780)[SSAValue(2), SSAValue(3), #<null>, #<null>, #<null>, #<null>, #<null>, #<null>]), i=(3,)) throw_boundserror at ./essentials.jl:14 getindex at ./essentials.jl:909 [inlined] ssa_substitute_op! at ./compiler/ssair/inlining.jl:1798 ssa_substitute_op! at ./compiler/ssair/inlining.jl:1852 ir_inline_item! at ./compiler/ssair/inlining.jl:386 ... ``` This commit updates the abstract interpretation and inlining algorithm to use the `nargs`/`isva` values held by `CodeInfo`. Similar modifications have also been made to EscapeAnalysis.jl. @nanosoldier `runbenchmarks("inference", vs=":master")` * Add `.zed` directory to `.gitignore` (#55974) Similar to the `vscode` config directory, we may ignore the `zed` directory as well. * typeintersect: reduce unneeded allocations from `merge_env` `merge_env` and `final_merge_env` could be skipped for emptiness test or if we know there's only 1 valid Union state. * typeintersect: trunc env before nested `intersect_all` if valid. This only covers the simplest cases. We might want a full dependence analysis and keep env length minimum in the future. * `@time` actually fix time report commas & add tests (#55982) https://github.com/JuliaLang/julia/pull/55977 looked simple but wasn't quite right because of a bad pattern in the lock conflicts report section. So fix and add tests. * adjust EA to JuliaLang/julia#52527 (#55986) `EnterNode.catch_dest` can now be `0` after the `try`/`catch` elision feature implemented in JuliaLang/julia#52527, and we actually need to adjust `EscapeAnalysis.compute_frameinfo` too. * Improvements to JITLink Seeing what this will look like, since it has a number of features (delayed compilation, concurrent compilation) that are starting to become important, so it would be nice to switch to only supporting one common implementation of memory management. Refs #50248 I am expecting https://github.com/llvm/llvm-project/issues/63236 may cause some problems, since we reconfigured some CI machines to minimize that issue, but it is still likely relevant. * rewrite catchjmp asm to use normal relocations instead of manual editing * add logic to prefer loading modules that are already loaded (#55908) Iterate over the list of existing loaded modules for PkgId whenever loading a new module for PkgId, so that we will use that existing build_id content if it otherwise passes the other stale_checks. * Apple: fix bus error on smaller readonly file in unix (#55859) Enables the fix for #28245 in #44354 for Apple now that the Julia bugs are fixed by #55641 and #55877. Closes #28245 * Add `Float16` to `Base.HWReal` (#55929) * docs: make mod an operator (#55988) * InteractiveUtils: add `@trace_compile` and `@trace_dispatch` (#55915) * Profile: document heap snapshot viewing tools (#55743) * [REPL] Fix #55850 by using `safe_realpath` instead of `abspath` in `projname` (#55851) * optimizer: enable load forwarding with the `finalizer` elision (#55991) When the finalizer elision pass is used, load forwarding is not performed currently, regardless of whether the pass succeeds or not. But this is not necessary, and by keeping the `setfield!` call, we can safely forward `getfield` even if finalizer elision is tried. * Avoid `stat`-ing stdlib path if it's unreadable (#55992) * doc: manual: cmd: fix Markdown in table entry for `--trim` (#55979) * Avoid conversions to `Float64` in non-literal powers of `Float16` (#55994) Co-authored-by: Alex Arslan <ararslan@comcast.net> * Remove unreachable error branch in memset calls (and in repeat) (#55985) Some places use the pattern memset(A, v, length(A)), which requires a conversion UInt(length(A)). This is technically fallible, but can't actually fail when A is a Memory or Array. Remove the dead error branch by casting to UInt instead. Similarly, in repeat(x, r), r is first checked to be nonnegative, then converted to UInt, then used in multiple calls where it is converted to UInt each time. Here, only do it once. * fix up docstring of `mod` (#56000) * fix typos (#56008) these are all in markdown files Co-authored-by: spaette <spaette@users.noreply.github.com> * Vectorise random vectors of `Float16` (#55997) * Clarify `div` docstring for floating-point input (#55918) Closes #55837 This is a variant of the warning found in the `fld` docstring clarifying floating-point behaviour. * improve getproperty(Pairs) warnings (#55989) - Only call `depwarn` if the field is `itr` or `data`; otherwise let the field error happen as normal - Give a more specific deprecation warning. * Document type-piracy / type-leakage restrictions for `require_stdlib` (#56005) I was a recent offender in https://github.com/JuliaLang/Pkg.jl/issues/4017#issuecomment-2377589989 This PR tries to lay down some guidelines for the behavior that stdlibs and the callers of `require_stdlib` must adhere to to avoid "duplicate stdlib" bugs These bugs are particularly nasty because they are experienced semi-rarely and under pretty specific circumstances (they only occur when `require_stdlib` loads another copy of a stdlib, often in a particular order and/or with a particular state of your pre-compile / loading cache) so they may make it a long way through a pre-release cycle without an actionable bug report. * [LinearAlgebra] Remove unreliable doctests (#56011) The exact textual representation of the output of these doctests depend on the specific kernel used by the BLAS backend, and can vary between versions of OpenBLAS (as it did in #41973), or between different CPUs, which makes these doctests unreliable. Fix #55998. * cleanup functions of Hermitian matrices (#55951) The functions of Hermitian matrices are a bit of a mess. For example, if we have a Hermitian matrix `a` with negative eigenvalues, `a^0.5` doesn't produce the `Symmetric` wrapper, but `sqrt(a)` does. On the other hand, if we have a positive definite `b`, `b^0.5` will be `Hermitian`, but `sqrt(b)` will be `Symmetric`: ```julia using LinearAlgebra a = Hermitian([1.0 2.0;2.0 1.0]) a^0.5 sqrt(a) b = Hermitian([2.0 1.0; 1.0 2.0]) b^0.5 sqrt(b) ``` This sort of arbitrary assignment of wrappers happens with pretty much all functions defined there. There's also some oddities, such as `cis` being the only function defined for `SymTridiagonal`, even though all `eigen`-based functions work, and `cbrt` being the only function not defined for complex Hermitian matrices. I did a cleanup: I defined all functions for `SymTridiagonal` and `Hermitian{<:Complex}`, and always assigned the appropriate wrapper, preserving the input one when possible. There's an inconsistency remaining that I didn't fix, that only `sqrt` and `log` accept a tolerance argument, as changing that is probably breaking. There were also hardly any tests that I could find (only `exp`, `log`, `cis`, and `sqrt`). I'm happy to add them if it's desired. * Fix no-arg `ScopedValues.@with` within a scope (#56019) Fixes https://github.com/JuliaLang/julia/issues/56017 * LinearAlgebra: make matprod_dest public (#55537) Currently, in a matrix multiplication `A * B`, we use `B` to construct the destination. However, this may not produce the optimal destination type, and is essentially single-dispatch. Letting packages specialize `matprod_dest` would help us obtain the optimal type by dispatching on both the arguments. This may significantly improve performance in the matrix multiplication. As an example: ```julia julia> using LinearAlgebra, FillArrays, SparseArrays julia> F = Fill(3, 10, 10); julia> s = sprand(10, 10, 0.1); julia> @btime $F * $s; 15.225 μs (10 allocations: 4.14 KiB) julia> typeof(F * s) SparseMatrixCSC{Float64, Int64} julia> nnz(F * s) 80 julia> VERSION v"1.12.0-DEV.1074" ``` In this case, the destination is a sparse matrix with 80% of its elements filled and being set one-by-one, which is terrible for performance. Instead, if we specialize `matprod_dest` to return a dense destination, we may obtain ```julia julia> LinearAlgebra.matprod_dest(F::FillArrays.AbstractFill, S::SparseMatrixCSC, ::Type{T}) where {T} = Matrix{T}(undef, size(F,1), size(S,2)) julia> @btime $F * $s; 754.632 ns (2 allocations: 944 bytes) julia> typeof(F * s) Matrix{Float64} ``` Potentially, this may be improved further by specializing `mul!`, but this is a 20x improvement just by choosing the right destination. Since this is being made public, we may want to bikeshed on an appropriate name for the function. * Sockets: Warn when local network access not granted. (#56023) Works around https://github.com/JuliaLang/julia/issues/56022 * Update test due to switch to intel syntax by default in #48103 (#55993) * add require_lock call to maybe_loaded_precompile (#56027) If we expect this to be a public API (https://github.com/timholy/Revise.jl for some reason is trying to access this state), we should lock around it for consistency with the other similar functions. Needed for https://github.com/timholy/Revise.jl/pull/856 * fix `power_by_squaring`: use `promote` instead of type inference (#55634) Fixes #53504 Fixes #55633 * Don't show keymap `@error` for hints (#56041) It's too disruptive to show errors for hints. The error will still be shown if tab is pressed. Helps issues like https://github.com/JuliaLang/julia/issues/56037 * Refactoring to be considered before adding MMTk * Removing jl_gc_notify_image_load, since it's a new function and not part of the refactoring * Moving gc_enable code to gc-common.c * Addressing PR comments * Push resolution of merge conflict * Removing jl_gc_mark_queue_obj_explicit extern definition from scheduler.c * Don't need the getter function since it's possible to use jl_small_typeof directly * Remove extern from free_stack declaration in julia_internal.h * Putting everything that is common GC tls into gc-tls-common.h * Typo * Adding gc-tls-common.h to Makefile as a public header * Removing gc-tls-common fields from gc-tls-mmtk.h * Fix typo in sockets tests. (#56038) * EA: use `is_mutation_free_argtype` for the escapability check (#56028) EA has been using `isbitstype` for type-level escapability checks, but a better criterion (`is_mutation_free`) is available these days, so we would like to use that instead. * effects: fix `Base.@_noub_meta` (#56061) This had the incorrect number of arguments to `Expr(:purity, ...)` causing it to be silently ignored. * effects: improve `:noub_if_noinbounds` documentation (#56060) Just a small touch-up * Disallow assigning asymmetric values to SymTridiagonal (#56068) Currently, we can assign an asymmetric value to a `SymTridiagonal`, which goes against what `setindex!` is expected to do. This is because `SymTridiagonal` symmetrizes the values along the diagonal, so setting a diagonal entry to an asymmetric value would lead to a subsequent `getindex` producing a different result. ```julia julia> s = SMatrix{2,2}(1:4); julia> S = SymTridiagonal(fill(s,4), fill(s,3)) 4×4 SymTridiagonal{SMatrix{2, 2, Int64, 4}, Vector{SMatrix{2, 2, Int64, 4}}}: [1 3; 3 4] [1 3; 2 4] ⋅ ⋅ [1 2; 3 4] [1 3; 3 4] [1 3; 2 4] ⋅ ⋅ [1 2; 3 4] [1 3; 3 4] [1 3; 2 4] ⋅ ⋅ [1 2; 3 4] [1 3; 3 4] julia> S[1,1] = s 2×2 SMatrix{2, 2, Int64, 4} with indices SOneTo(2)×SOneTo(2): 1 3 2 4 julia> S[1,1] == s false julia> S[1,1] 2×2 Symmetric{Int64, SMatrix{2, 2, Int64, 4}} with indices SOneTo(2)×SOneTo(2): 1 3 3 4 ``` After this PR, ```julia julia> S[1,1] = s ERROR: ArgumentError: cannot set a diagonal entry of a SymTridiagonal to an asymmetric value ``` * Remove unused matrix type params in diag methods (#56048) These parameters are not used in the method, and are unnecessary for dispatch. * LinearAlgebra: diagzero for non-OneTo axes (#55252) Currently, the off-diagonal zeros for a block-`Diagonal` matrix is computed using `diagzero`, which calls `zeros` for the sizes of the elements. This returns an `Array`, unless one specializes `diagzero` for the custom `Diagonal` matrix type. This PR defines a `zeroslike` function that dispatches on the axes of the elements, which lets packages specialize on the axes to return custom `AbstractArray`s. Choosing to specialize on the `eltype` avoids the need to specialize on the container, and allows packages to return appropriate types for custom axis types. With this, ```julia julia> LinearAlgebra.zeroslike(::Type{S}, ax::Tuple{SOneTo, Vararg{SOneTo}}) where {S<:SMatrix} = SMatrix{map(length, ax)...}(ntuple(_->zero(eltype(S)), prod(length, ax))) julia> D = Diagonal(fill(SMatrix{2,3}(1:6), 2)) 2×2 Diagonal{SMatrix{2, 3, Int64, 6}, Vector{SMatrix{2, 3, Int64, 6}}}: [1 3 5; 2 4 6] ⋅ ⋅ [1 3 5; 2 4 6] julia> D[1,2] # now an SMatrix 2×3 SMatrix{2, 3, Int64, 6} with indices SOneTo(2)×SOneTo(3): 0 0 0 0 0 0 julia> LinearAlgebra.zeroslike(::Type{S}, ax::Tuple{SOneTo, Vararg{SOneTo}}) where {S<:MMatrix} = MMatrix{map(length, ax)...}(ntuple(_->zero(eltype(S)), prod(length, ax))) julia> D = Diagonal(fill(MMatrix{2,3}(1:6), 2)) 2×2 Diagonal{MMatrix{2, 3, Int64, 6}, Vector{MMatrix{2, 3, Int64, 6}}}: [1 3 5; 2 4 6] ⋅ ⋅ [1 3 5; 2 4 6] julia> D[1,2] # now an MMatrix 2×3 MMatrix{2, 3, Int64, 6} with indices SOneTo(2)×SOneTo(3): 0 0 0 0 0 0 ``` The reason this can't be the default behavior is that we are not guaranteed that there exists a `similar` method that accepts the combination of axes. This is why we have to fall back to using the sizes, unless a specialized method is provided by a package. One positive outcome of this is that indexing into such a block-diagonal matrix will now usually be type-stable, which mitigates https://github.com/JuliaLang/julia/issues/45535 to some extent (although it doesn't resolve the issue). I've also updated the `getindex` for `Bidiagonal` to use `diagzero`, instead of the similarly defined `bidiagzero` function that it was using. Structured block matrices may now use `diagzero` uniformly to generate the zero elements. * Multi-argument `gcdx(a, b, c...)` (#55935) Previously, `gcdx` only worked for two arguments - but the underlying idea extends to any (nonzero) number of arguments. Similarly, `gcd` already works for 1, 2, 3+ arguments. This PR implements the 1 and 3+ argument versions of `gcdx`, following the [wiki page](https://en.wikipedia.org/wiki/Extended_Euclidean_algorithm#The_case_of_more_than_two_numbers) for the Extended Euclidean algorithm. * Refactoring to be considered before adding MMTk * Removing jl_gc_notify_image_load, since it's a new function and not part of the refactoring * Moving gc_enable code to gc-common.c * Addressing PR comments * Push resolution of merge conflict * Removing jl_gc_mark_queue_obj_explicit extern definition from scheduler.c * Don't need the getter function since it's possible to use jl_small_typeof directly * Remove extern from free_stack declaration in julia_internal.h * Putting everything that is common GC tls into gc-tls-common.h * Typo * Adding gc-tls-common.h to Makefile as a public header * Adding jl_full_sweep_reasons since timing.jl depends on it * Fixing issue with jl_full_sweep_reasons (missing constants) * fix `_growbeg!` unncessary resizing (#56029) This was very explicitly designed such that if there was a bunch of extra space at the end of the array, we would copy rather than allocating, but by making `newmemlen` be at least `overallocation(memlen)` rather than `overallocation(len)`, this branch was never hit. found by https://github.com/JuliaLang/julia/issues/56026 * REPL: hide any prints to stdio during `complete_line` (#55959) * teach llvm-alloc-helpers about `gc_loaded` (#56030) combined with https://github.com/JuliaLang/julia/pull/55913, the compiler is smart enough to fully remove ``` function f() m = Memory{Int}(undef, 3) @inbounds m[1] = 2 @inbounds m[2] = 2 @inbounds m[3] = 4 @inbounds return m[1] + m[2] + m[3] end ``` * mpfr: prevent changing precision (#56049) Changing precision requires reallocating the data field, which is better done by making a new BigFloat (since they are conceptually immutable anyways). Also do a bit a cleanup while here. Closes #56044 * stackwalk: fix jl_thread_suspend_and_get_state race (#56047) There was a missing re-assignment of old = -1; at the end of that loop which means in the ABA case, we accidentally actually acquire the lock on the thread despite not actually having stopped the thread; or in the counter-case, we try to run through this logic with old==-1 on the next iteration, and that isn't valid either (jl_thread_suspend_and_get_state should return failure and the loop will abort too early). Fix #56046 * irrationals: restrict assume effects annotations to known types (#55886) Other changes: * replace `:total` with the less powerful `:foldable` * add an `<:Integer` dispatch constraint on the `rationalize` method, closes #55872 * replace `Rational{<:Integer}` with just `Rational`, they're equal Other issues, related to `BigFloat` precision, are still present in irrationals.jl, to be fixed by followup PRs, including #55853. Fixes #55874 * update `hash` doc string: `widen` not required any more (#55867) Implementing `widen` isn't a requirement any more, since #26022. * Merge `diag` methods for triangular matrices (#56086) * slightly improve inference in precompilation code (#56084) Avoids the ``` 11: signature Tuple{typeof(convert), Type{String}, Any} triggered MethodInstance for Base.Precompilation.ExplicitEnv(::String) (84 children) ``` shown in https://github.com/JuliaLang/julia/issues/56080#issuecomment-2404765120 Co-authored-by: KristofferC <kristoffer.carlsson@juliacomputing.com> * avoid defining `convert(Vector{String}, ...)` in LibGit2 (#56082) This is a weird conversion function to define. Seems cleaner to use the iteration interface for this. Also avoids some invalidations (https://github.com/JuliaLang/julia/issues/56080#issuecomment-2404765120) Co-authored-by: KristofferC <kristoffer.carlsson@juliacomputing.com> * array: inline `convert` where possible (#56034) This improves a common scenario, where someone wants to `push!` a poorly-typed object onto a well-typed Vector. For example: ```julia const NT = @NamedTuple{x::Int,y::Any} foo(v::Vector{NT}, x::Int, @nospecialize(y)) = push!(v, (; x, y)) ``` The `(; x, y)` is slightly poorly-typed here. It could have any type for its `.y` field before it is converted inside the `push!` to a NamedTuple with `y::Any` Without this PR, the dispatch for this `push!` cannot be inferred: ```julia julia> code_typed(foo, (Vector{NT}, Int, Any))[1] CodeInfo( 1 ─ ... │ %4 = %new(%3, x, y)::NamedTuple{(:x, :y), <:Tuple{Int64, Any}} │ %5 = Main.push!(v, %4)::Vector{@NamedTuple{x::Int64, y}} └── return %5 ) => Vector{@NamedTuple{x::Int64, y}} ``` With this PR, the above dynamic call is fully statically resolved and inlined (and therefore `--trim` compatible) * Remove some unnecessary `real` specializations for structured matrices (#56083) The `real(::AbstractArray{<:Rea})` fallback method should handle these cases correctly. * Combine `diag` methods for `SymTridiagonal` (#56014) Currently, there are two branches, one for an `eltype` that is a `Number`, and the other that deals with generic `eltype`s. They do similar things, so we may combine these, and use branches wherever necessary to retain the performance. We also may replace explicit materialized arrays by generators in `copyto!`. Overall, this improves performance in `diag` for matrices of matrices, whereas the performance in the common case of matrices of numbers remains unchanged. ```julia julia> using StaticArrays, LinearAlgebra julia> s = SMatrix{2,2}(1:4); julia> S = SymTridiagonal(fill(s,100), fill(s,99)); julia> @btime diag($S); 1.292 μs (5 allocations: 7.16 KiB) # nightly, v"1.12.0-DEV.1317" 685.012 ns (3 allocations: 3.19 KiB) # This PR ``` This PR also allows computing the `diag` for more values of the band index `n`: ```julia julia> diag(S,99) 1-element Vector{SMatrix{2, 2, Int64, 4}}: [0 0; 0 0] ``` This would work as long as `getindex` works for the `SymTridiagonal` for that band, and the zero element may be converted to the `eltype`. * fix `Vararg{T,T} where T` crashing `code_typed` (#56081) Not sure this is the right place to fix this error, perhaps `match.spec_types` should always be a tuple of valid types? fixes #55916 --------- Co-authored-by: Jameson Nash <vtjnash@gmail.com> * [libblastrampoline_jll] Upgrade to v5.11.1 (#56094) v5.11.1 is a patch release with a couple of RISC-V fixes. * Revert "REPL: hide any prints to stdio during `complete_line`" (#56102) * Remove warning from c when binding is ambiguous (#56103) * make `Base.ANSIIterator` have a concrete field (#56088) Avoids the invalidation ``` backedges: 1: superseding sizeof(s::AbstractString) @ Base strings/basic.jl:177 with MethodInstance for sizeof(::AbstractString) (75 children) ``` shown in https://github.com/JuliaLang/julia/issues/56080#issuecomment-2404765120. Co-authored-by: KristofferC <kristoffer.carlsson@juliacomputing.com> * Subtype: some performance tuning. (#56007) The main motivation of this PR is to fix #55807. dc689fe8700f70f4a4e2dbaaf270f26b87e79e04 tries to remove the slow `may_contain_union_decision` check by re-organizing the code path. Now the fast path has been removed and most of its optimization has been integrated into the preserved slow path. Since the slow path stores all inner ∃ decisions on the outer most R stack, there might be overflow risk. aee69a41441b4306ba3ee5e845bc96cb45d9b327 should fix that concern. The reported MWE now becomes ```julia 0.000002 seconds 0.000040 seconds (105 allocations: 4.828 KiB, 52.00% compilation time) 0.000023 seconds (105 allocations: 4.828 KiB, 49.36% compilation time) 0.000026 seconds (105 allocations: 4.828 KiB, 50.38% compilation time) 0.000027 seconds (105 allocations: 4.828 KiB, 54.95% compilation time) 0.000019 seconds (106 allocations: 4.922 KiB, 49.73% compilation time) 0.000024 seconds (105 allocations: 4.828 KiB, 52.24% compilation time) ``` Local bench also shows that 72855cd slightly accelerates `OmniPackage.jl`'s loading ```julia julia> @time using OmniPackage # v1.11rc4 20.525278 seconds (25.36 M allocations: 1.606 GiB, 8.48% gc time, 12.89% compilation time: 77% of which was recompilation) # v1.11rc4+aee69a4+72855cd 19.527871 seconds (24.92 M allocations: 1.593 GiB, 8.88% gc time, 15.13% compilation time: 82% of which was recompilation) ``` * rearrange jl_delete_thread to be thread-safe (#56097) Prior to this, especially on macOS, the gc-safepoint here would cause the process to segfault as we had already freed the current_task state. Rearrange this code so that the GC interactions (except for the atomic store to current_task) are all handled before entering GC safe, and then signaling the thread is deleted (via setting current_task = NULL, published by jl_unlock_profile_wr to other threads) is last. ``` ERROR: Exception handler triggered on unmanaged thread. Process 53827 stopped * thread #5, stop reason = EXC_BAD_ACCESS (code=2, address=0x100018008) frame #0: 0x0000000100b74344 libjulia-internal.1.12.0.dylib`jl_delete_thread [inlined] jl_gc_state_set(ptls=0x000000011f8b3200, state='\x02', old_state=<unavailable>) at julia_threads.h:272:9 [opt] 269 assert(old_state != JL_GC_CONCURRENT_COLLECTOR_THREAD); 270 jl_atomic_store_release(&ptls->gc_state, state); 271 if (state == JL_GC_STATE_UNSAFE || old_state == JL_GC_STATE_UNSAFE) -> 272 jl_gc_safepoint_(ptls); 273 return old_state; 274 } 275 STATIC_INLINE int8_t jl_gc_state_save_and_set(jl_ptls_t ptls, Target 0: (julia) stopped. (lldb) up frame #1: 0x0000000100b74320 libjulia-internal.1.12.0.dylib`jl_delete_thread [inlined] jl_gc_state_save_and_set(ptls=0x000000011f8b3200, state='\x02') at julia_threads.h:278:12 [opt] 275 STATIC_INLINE int8_t jl_gc_state_save_and_set(jl_ptls_t ptls, 276 int8_t state) 277 { -> 278 return jl_gc_state_set(ptls, state, jl_atomic_load_relaxed(&ptls->gc_state)); 279 } 280 #ifdef __clang_gcanalyzer__ 281 // these might not be a safepoint (if they are no-op safe=>safe transitions), but we have to assume it could be (statically) (lldb) frame #2: 0x0000000100b7431c libjulia-internal.1.12.0.dylib`jl_delete_thread(value=0x000000011f8b3200) at threading.c:537:11 [opt] 534 ptls->root_task = NULL; 535 jl_free_thread_gc_state(ptls); 536 // then park in safe-region -> 537 (void)jl_gc_safe_enter(ptls); 538 } ``` (test incorporated into https://github.com/JuliaLang/julia/pull/55793) * OpenBLAS: Use dynamic architecture support on AArch64. (#56107) We already do so on Yggdrasil, so this just makes both source and binary builds behave similarly. Closes https://github.com/JuliaLang/julia/issues/56075 * IRShow: label builtin / intrinsic / dynamic calls in `code_typed` (#56036) This makes it much easier to spot dynamic dispatches * 🤖 [master] Bump the Pkg stdlib from 51d4910c1 to fbaa2e337 (#56124) * Fix type instability of closures capturing types (2) (#40985) Instead of closures lowering to `typeof` for the types of captured fields, this introduces a new function `_typeof_captured_variable` that returns `Type{T}` if `T` is a type (w/o free typevars). - replaces/closes #35970 - fixes #23618 --------- Co-authored-by: Takafumi Arakaki <aka.tkf@gmail.com> Co-authored-by: Shuhei Kadowaki <aviatesk@gmail.com> * Remove debug error statement from Makefile. (#56127) * align markdown table (#56122) @gbaraldi `#51197` @spaette `#56008` fix innocuous malalignment of table after those pulls were merged * Improve IOBuffer docs (#56024) Based on the discussion in #55978, I have tried to clarify the documentation of `IOBuffer`. * Comment out url and fix typo in stackwalk.c (#56131) Introduced in #55623 * libgit2: Always use the bundled PCRE library. (#56129) This is how Yggdrasil builds the library. * Update JLL build versions (#56133) This commit encompasses the following changes: - Updating the JLL build version for Clang, dSFMT, GMP, LibUV, LibUnwind, LLD, LLVM, libLLVM, MbedTLS, MPFR, OpenBLAS, OpenLibm, p7zip, PCRE2, SuiteSparse, and Zlib. - Updating CompilerSupportLibraries to v1.2.0. The library versions contained in this release of CSL don't differ from v1.1.1, the only difference is that v1.2.0 includes FreeBSD AArch64. - Updating nghttp2 from 1.60.0 to 1.63.0. See [here](https://github.com/nghttp2/nghttp2/releases) for changes between these versions. - Adding `aarch64-unknown-freebsd` to the list of triplets to check when refreshing checksums. Note that dependencies that link to MbedTLS (Curl, LibSSH2, LibGit2) are excluded here. They'll be updated once a resolution is reached for the OpenSSL switching saga. Once that happens, FreeBSD AArch64 should be able to be built without any dependency source builds. * typo in `Compiler.Effects` doc string: `checkbounds` -> `boundscheck` (#56140) Follows up on #56060 * HISTORY: fix missing links (#56137) * OpenBLAS: Fix cross-compilation detection for source build. (#56139) We may be cross-compiling Linux-to-Linux, in which case `BUILD_OS` == `OS`, so look at `XC_HOST` to determine whether we're cross compiling. * `diag` for `BandedMatrix`es for off-limit bands (#56065) Currently, one can only obtain the `diag` for a `BandedMatrix` (such as a `Diagonal`) when the band index is bounded by the size of the matrix. This PR relaxes this requirement to match the behavior for arrays, where `diag` returns an empty vector for a large band index instead of throwing an error. ```julia julia> D = Diagonal(ones(4)) 4×4 Diagonal{Float64, Vector{Float64}}: 1.0 ⋅ ⋅ ⋅ ⋅ 1.0 ⋅ ⋅ ⋅ ⋅ 1.0 ⋅ ⋅ ⋅ ⋅ 1.0 julia> diag(D, 10) Float64[] julia> diag(Array(D), 10) Float64[] ``` Something similar for `SymTridiagonal` is being done in https://github.com/JuliaLang/julia/pull/56014 * Port progress bar improvements from Pkg (#56125) Includes changes from https://github.com/JuliaLang/Pkg.jl/pull/4038 and https://github.com/JuliaLang/Pkg.jl/pull/4044. Co-authored-by: Kristoffer Carlsson <kcarlsson89@gmail.com> * Add support for LLVM 19 (#55650) Co-authored-by: Zentrik <Zentrik@users.noreply.github.com> * 🤖 [master] Bump the Pkg stdlib from fbaa2e337 to 27c1b1ee5 (#56146) * HISTORY entry for deletion of `length(::Stateful)` (#55861) xref #47790 xref #51747 xref #54953 xref #55858 * ntuple: ensure eltype is always `Int` (#55901) Fixes #55790 * Improve remarks of the alloc opt pass slightly. (#55995) The Value printer LLVM uses just prints the kind of instruction so it just shows call. --------- Co-authored-by: Oscar Smith <oscardssmith@gmail.com> * Implement Base.fd() for TCPSocket, UDPSocket, and TCPServer (#53721) This is quite handy if you want to pass off the file descriptor to a C library. I also added a warning to the `fd()` docstring to warn folks about duplicating the file descriptor first. * Fix `JULIA_CPU_TARGET` being propagated to workers precompiling stdlib pkgimages (#54093) Apparently (thanks ChatGPT) each line in a makefile is executed in a separate shell so adding an `export` line on one line does not propagate to the next line. * Merge tr methods for triangular matrices (#56154) Since the methods do identical things, we don't need multiple of these. * Reduce duplication in triangular indexing methods (#56152) This uses an orthogonal design to reduce code duplication in the indexing methods for triangular matrices. * update LLVM docs (#56162) dump with raw=true so you don't get random erorrs, and show how to run single modules. --------- Co-authored-by: Valentin Churavy <vchuravy@users.noreply.github.com> Co-authored-by: Mosè Giordano <765740+giordano@users.noreply.github.com> Co-authored-by: Jameson Nash <vtjnash@gmail.com> * Fix zero elements for block-matrix kron involving Diagonal (#55941) Currently, it's assumed that the zero element is identical for the matrix, but this is not necessary if the elements are matrices themselves and have different sizes. This PR ensures that `kron` for a `Diagonal` has the correct zero elements. Current: ```julia julia> D = Diagonal(1:2) 2×2 Diagonal{Int64, UnitRange{Int64}}: 1 ⋅ ⋅ 2 julia> B = reshape([ones(2,2), ones(3,2), ones(2,3), ones(3,3)], 2, 2); julia> size.(kron(D, B)) 4×4 Matrix{Tuple{Int64, Int64}}: (2, 2) (2, 3) (2, 2) (2, 2) (3, 2) (3, 3) (2, 2) (2, 2) (2, 2) (2, 2) (2, 2) (2, 3) (2, 2) (2, 2) (3, 2) (3, 3) ``` This PR ```julia julia> size.(kron(D, B)) 4×4 Matrix{Tuple{Int64, Int64}}: (2, 2) (2, 3) (2, 2) (2, 3) (3, 2) (3, 3) (3, 2) (3, 3) (2, 2) (2, 3) (2, 2) (2, 3) (3, 2) (3, 3) (3, 2) (3, 3) ``` Note the differences e.g. in the `CartesianIndex(4,1)`, `CartesianIndex(3,2)` and `CartesianIndex(3,3)` elements. * Call `MulAddMul` instead of multiplication in _generic_matmatmul! (#56089) Fix https://github.com/JuliaLang/julia/issues/56085 by calling a newly created `MulAddMul` object that only wraps the `alpha` (with `beta` set to `false`). This avoids the explicit multiplication if `alpha` is known to be `isone`. * improve `allunique`'s type stability (#56161) Caught by https://github.com/aviatesk/JET.jl/issues/667. * Add invalidation barriers for `displaysize` and `implicit_typeinfo` (#56159) These are invalidated by our own stdlibs (Dates and REPL) unfortunately so we need to put this barrier in. This fix is _very_ un-satisfying, because it doesn't do anything to solve this problem for downstream libraries that use e.g. `displaysize`. To fix that, I think we need a way to make sure callers get these invalidation barriers by default... * Fix markdown list in installation.md (#56165) Documenter.jl requires all trailing list content to follow the same indentation as the header. So, in the current view (https://docs.julialang.org/en/v1/manual/installation/#Command-line-arguments) the list appears broken. * [Random] Add more comments and a helper function in Xoshiro code (#56144) Follow up to #55994 and #55997. This should basically be a non-functional change and I see no performance difference, but the comments and the definition of a helper function should make the code easier to follow (I initially struggled in #55997) and extend to other types. * add objects to concisely specify initialization PerProcess: once per process PerThread: once per thread id PerTask: once per task object * add precompile support for recording fields to change Somewhat generalizes our support for changing Ptr to C_NULL. Not particularly fast, since it is just using the builtins implementation of setfield, and delaying the actual stores, but it should suffice. * improve OncePer implementation Address reviewer feedback, add more fixes and more tests, rename to add Once prefix. * fix use-after-free in test (detected in win32 CI) * Make loading work when stdlib deps are missing in the manifest (#56148) Closes https://github.com/JuliaLang/julia/issues/56109 Simulating a bad manifest by having `LibGit2_jll` missing as a dep of `LibGit2` in my default env, say because the manifest was generated by a different julia version or different master julia commit. ## This PR, it just works ``` julia> using Revise julia> ``` i.e. ``` % JULIA_DEBUG=loading ./julia --startup-file=no julia> using Revise ... ┌ Debug: Stdlib LibGit2 [76f85450-5226-5b5a-8eaa-529ad045b433] is trying to load `LibGit2_jll` │ which is not listed as a dep in the load path manifests, so resorting to search │ in the stdlib Project.tomls for true deps └ @ Base loading.jl:387 ┌ Debug: LibGit2 [76f85450-5226-5b5a-8eaa-529ad045b433] indeed depends on LibGit2_jll in project /Users/ian/Documents/GitHub/julia/usr/share/julia/stdlib/v1.12/LibGit2/Project.toml └ @ Base loading.jl:395 ... julia> ``` ## Master ``` julia> using Revise Info Given Revise was explicitly requested, output will be shown live ERROR: LoadError: ArgumentError: Package LibGit2 does not have LibGit2_jll in its dependencies: - Note that the following manifests in the load path were resolved with a potentially different DEV version of the current version, which may be the cause of the error. Try to re-resolve them in the current version, or consider deleting them if that fails: /Users/ian/.julia/environments/v1.12/Manifest.toml - You may have a partially installed environment. Try `Pkg.instantiate()` to ensure all packages in the environment are installed. - Or, if you have LibGit2 checked out for development and have added LibGit2_jll as a dependency but haven't updated your primary environment's manifest file, try `Pkg.resolve()`. - Otherwise you may need to report an issue with LibGit2 ... ``` * Remove llvm-muladd pass and move it's functionality to to llvm-simdloop (#55802) Closes https://github.com/JuliaLang/julia/issues/55785 I'm not sure if we want to backport this like this. Because that removes some functionality (the pass itself). So LLVM.jl and friends might need annoying version code. We can maybe keep the code there and just not run the pass in a backport. * Fix implicit `convert(String, ...)` in several places (#56174) This removes several `convert(String, ...)` from this code, which really shouldn't be something we invalidate on in the first place (see https://github.com/JuliaLang/julia/issues/56173) but this is still an improvement in code quality so let's take it. * Change annotations to use a NamedTuple (#55741) Due to popular demand, the type of annotations is to be changed from a `Tuple{UnitRange{Int}, Pair{Symbol, Any}}` to a `NamedTuple{(:region, :label, :value), Tuple{UnitRange{Int}, Symbol, Any}}`. This requires the expected code churn to `strings/annotated.jl`, and some changes to the StyledStrings and JuliaSyntaxHighlighting libraries. Closes #55249 and closes #55245. * Getting rid of mmtk_julia.c in the binding and moving it to gc-mmtk.c * Trying to organize and label the code in gc-mmtk.c * Remove redundant `convert` in `_setindex!` (#56178) Follow up to #56034, ref: https://github.com/JuliaLang/julia/pull/56034#discussion_r1798573573. --------- Co-authored-by: Cody Tapscott <84105208+topolarity@users.noreply.github.com> * Improve type inference of Artifacts.jl (#56118) This also has some changes that move platform selection to compile time together with https://github.com/JuliaPackaging/JLLWrappers.jl/commit/45cc04963f3c99d4eb902f97528fe16fc37002cc, move the platform selection to compile time. (this helps juliac a ton) * Initial support for RISC-V (#56105) Rebase and extension of @alexfanqi's initial work on porting Julia to RISC-V. Requires LLVM 19. Tested on a VisionFive2, built with: ```make MARCH := rv64gc_zba_zbb MCPU := sifive-u74 USE_BINARYBUILDER:=0 DEPS_GIT = llvm override LLVM_VER=19.1.1 override LLVM_BRANCH=julia-release/19.x override LLVM_SHA1=julia-release/19.x ``` ```julia-repl ❯ ./julia _ _ _ _(_)_ | Documentation: https://docs.julialang.org (_) | (_) (_) | _ _ _| |_ __ _ | Type "?" for help, "]?" for Pkg help. | | | | | | |/ _` | | | | |_| | | | (_| | | Version 1.12.0-DEV.1374 (2024-10-14) _/ |\__'_|_|_|\__'_| | riscv/25092a3982* (fork: 1 commits, 0 days) |__/ | julia> versioninfo(; verbose=true) Julia Version 1.12.0-DEV.1374 Commit 25092a3982* (2024-10-14 09:57 UTC) Platform Info: OS: Linux (riscv64-unknown-linux-gnu) uname: Linux 6.11.3-1-riscv64 #1 SMP Debian 6.11.3-1 (2024-10-10) riscv64 unknown CPU: …

tecosaur force-pushed the uripath branch 2 times, most recently from e4fee58 to 8b36796 Compare August 11, 2024 05:17

tecosaur added the filesystem Underlying file system and functions that use it label Aug 11, 2024

IanButterworth reviewed Aug 11, 2024

View reviewed changes

base/path.jl Show resolved Hide resolved

tecosaur force-pushed the uripath branch 3 times, most recently from 1b26082 to 52e65c8 Compare August 11, 2024 18:01

IanButterworth mentioned this pull request Aug 11, 2024

Profile.print: color Base/Core & packages. Make paths clickable #55335

Merged

3 tasks

IanButterworth approved these changes Aug 11, 2024

View reviewed changes

tecosaur force-pushed the uripath branch 3 times, most recently from f2a879e to 222e891 Compare August 12, 2024 16:19

KristofferC reviewed Aug 12, 2024

View reviewed changes

base/path.jl Outdated Show resolved Hide resolved

tecosaur force-pushed the uripath branch 2 times, most recently from 220782d to 797849a Compare August 12, 2024 17:27

tecosaur force-pushed the uripath branch 3 times, most recently from 1eec7f5 to ee0559d Compare August 14, 2024 13:08

tecosaur force-pushed the uripath branch from ee0559d to 207a5c9 Compare August 15, 2024 15:00

tecosaur force-pushed the uripath branch 3 times, most recently from a181f00 to cd21570 Compare August 24, 2024 07:08

tecosaur force-pushed the uripath branch from cd21570 to 9e1084e Compare August 24, 2024 07:13

fredrikekre reviewed Aug 25, 2024

View reviewed changes

IanButterworth approved these changes Sep 24, 2024

View reviewed changes

tecosaur mentioned this pull request Sep 24, 2024

Profile: Improve generation of clickable terminal links #55857

Merged

IanButterworth merged commit a06a801 into JuliaLang:master Sep 24, 2024
7 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add filesystem func to transform a path to a URI #55454

Add filesystem func to transform a path to a URI #55454

tecosaur commented Aug 11, 2024 •

edited

Loading

tecosaur commented Aug 11, 2024 •

edited

Loading

IanButterworth left a comment

tecosaur commented Aug 12, 2024

tecosaur commented Aug 12, 2024

davidanthoff commented Aug 13, 2024

tecosaur commented Aug 14, 2024 •

edited

Loading

davidanthoff commented Aug 14, 2024

tecosaur commented Aug 14, 2024

davidanthoff commented Aug 14, 2024

tecosaur commented Aug 15, 2024 •

edited

Loading

davidanthoff commented Aug 15, 2024

tecosaur commented Aug 24, 2024

tecosaur commented Aug 24, 2024

fredrikekre Aug 25, 2024

tecosaur Aug 25, 2024

davidanthoff commented Aug 25, 2024

IanButterworth commented Sep 5, 2024

IanButterworth commented Sep 24, 2024

tecosaur commented Sep 24, 2024

Add filesystem func to transform a path to a URI #55454

Add filesystem func to transform a path to a URI #55454

Conversation

tecosaur commented Aug 11, 2024 • edited Loading

tecosaur commented Aug 11, 2024 • edited Loading

IanButterworth left a comment

Choose a reason for hiding this comment

tecosaur commented Aug 12, 2024

tecosaur commented Aug 12, 2024

davidanthoff commented Aug 13, 2024

tecosaur commented Aug 14, 2024 • edited Loading

davidanthoff commented Aug 14, 2024

tecosaur commented Aug 14, 2024

davidanthoff commented Aug 14, 2024

tecosaur commented Aug 15, 2024 • edited Loading

davidanthoff commented Aug 15, 2024

tecosaur commented Aug 24, 2024

tecosaur commented Aug 24, 2024

fredrikekre Aug 25, 2024

Choose a reason for hiding this comment

tecosaur Aug 25, 2024

Choose a reason for hiding this comment

davidanthoff commented Aug 25, 2024

IanButterworth commented Sep 5, 2024

IanButterworth commented Sep 24, 2024

tecosaur commented Sep 24, 2024

tecosaur commented Aug 11, 2024 •

edited

Loading

tecosaur commented Aug 11, 2024 •

edited

Loading

tecosaur commented Aug 14, 2024 •

edited

Loading

tecosaur commented Aug 15, 2024 •

edited

Loading