Better documentation of stdlibs

[carstenbauer]

A lot of functionality has been moved from Base to stdlibs. However, for people who haven't been part of this process (especially beginners) it is not really transparent which stdlibs exist and what their purpose is.

The documentation doesn't seem to contain a list of all stdlibs with short descriptions. I think such a list would be very useful as a compact overview of all stdlibs.

The only kind of overview is the navigation pane. There, however, names do not match actual stdlib names and short descriptions are missing.

1. Do people agree that such a list would be good to have?
2. Where should such a table go?

[fredrikekre]

I think it could be good to have somewhere in the beginning, where we describe what Base includes, and then describe the stdlibs with links to their sections.

[monsij]

@fredrikekre @crstnbr Is this somewhat related to #30469 ?

[ViralBShah]

I think the documentation now does a pretty good job of Base vs. stdlib documentation. The doc coverage is also much higher than when this issue was filed. Can we close?

[carstenbauer]

@ViralBShah While I agree that the documentation has been improved, I still can't find a table (or something similar) in the Manual section that points to the available stdlibs. (The stdlib sections in the navigation pain also still don't match the actual stdlib names. But I think that a proper listing in the Manual section is much better/more important.)

[ViralBShah]

Yes, true. If you have a moment, would you be able to provide a PR with such a table?

[adigitoleo]

I guess the _jll modules should be excluded from the list? Or put in their own section?

Unless I'm missing something, getting the list is quite simple: readdir(Sys.STDLIB).

Or, without the _jll wrappers:

filter(x -> match(r"_jll$", x) === nothing, readdir(Sys.STDLIB))

Just a matter of pretty printing them into a list with links, might have a go tomorrow.

[pbouffard]

Came here when trying to find a single canonical URL to link a Julia newbie to when first mentioning "the standard library". I couldn't find a way to link to a whole section of the docs - I wonder if there could be a page at the start of the Standard Library section called something like "Introduction" that could be linked to. That could have a brief description as in the earlier comment, ie.

The Julia standard library contains additional, commonly used packages that are installed alongside the Julia runtime by default. To use a standard library package, it is first necessary to load the package with a [`using`](@ref) statement. 

The automatic table of the packages themselves would be great to have right there but as I say I'm looking more for a link that answers "what is the Julia Standard Library" more than "what is in the Julia Standard Library.

The first mention of "standard library" in https://docs.julialang.org/en/v1/#man-introduction can link there too.

[adigitoleo]

I have managed to get something working with documenter.jl, the stdlib modules are just in a simple markdown list rather than a table. I'll open a PR shortly, let me know if the section headers look OK or if there is any other feedback. I thought I'd just put this section at the end of the landing page (index.html) but am open to moving it somewhere else.

@pbouffard You will get links for the two new headings as seen in the following screenshot:

[pbouffard]

@pbouffard You will get links for the two new headings as seen in the following screenshot:

Fantastic, thanks!

Would this get backported at all? It would be nice (for me and colleagues anyway) to have this for Julia 1.9.

[carstenbauer]

If this qualifies for a backport - I have no idea if it does - I would expect that it would only go into 1.10 since it is the new LTS.

[adigitoleo]

I've asked about backporting in the PR. I've also mentioned that the original request included short descriptions of each stdlib package, which I haven't added. It would be more appropriate to add those as module-level docstrings of the stdlib packages themselves, so that they are maintained alongside the actual package.

[waldyrious]

Great work @adigitoleo! It was indeed a sorely lacking piece of docs.

let me know if the section headers look OK

It might be worth pointing out that Core and Base (and Main) are described as "standard modules" in the Base > Essentials page of the docs. I wonder if it would be better to reuse that nomenclature in the newly added section header.

Additionally, allow me to share notes I collected for myself a while ago when I was looking for precisely such an overview. There's no guarantee of correctness, but hopefully they can be useful to your PR:

• Some of julia's functionality comes from the standard modules Main, Core and Base (at the base/ directory of the julia repo), as well as various unpackaged submodules.
• Other parts of julia's functionality come from a Standard Library of packaged modules.
• Of these, some are developed directly within the stdlib directory of the julia repo; these are the subdirectories under the stdlib/ directory of the main julia repository, and are listed in the STDLIBS variable in stdlib/Makefile.
  • There are also various *_jll directories, which are appended to the STDLIBS variable but appear to not be actual modules. They seem to be wrappers for external, non-julia libraries like libuv, MPFR, curl, zlib, 7zip, LLVM, etc.
  • Full list, as of v1.9:
    • Artifacts
    • Base64
    • CRC32c
    • Dates
    • Distributed
    • FileWatching
    • Future
    • InteractiveUtils
    • LazyArtifacts
    • LibGit2
    • Libdl
    • LinearAlgebra
    • Logging
    • Markdown
    • Mmap
    • Printf
    • Profile
    • REPL
    • Random
    • Serialization
    • SharedArrays
    • Sockets
    • TOML
    • Test
    • UUIDs
    • Unicode
• Others live in separate repositories outside of the julia repo, and typically even outside the JuliaLang org; they correspond to the *.version files under stdlib/, and are listed in the STDLIBS_EXT variable in stdlib/Makefile. Several of these are (though ideally all of them should be) identified in their readme as part of the Julia stdlib.
  • Those under JuliaLang org (found with this search, which is tweaked to exclude JuliaLang/Example.jl and JuliaLang/julia), as of v1.9, are:
    • JuliaLang/Pkg.jl
    • JuliaLang/Downloads.jl
    • JuliaLang/NetworkOptions.jl
    • (Not sure why these are kept here instead of in the relevant orgs...)
  • Those under other organizations (found with this search, which is tweaked to exclude the JuliaLang org), as of v1.9, are:
    • JuliaWeb/LibCURL.jl
    • JuliaStats/Statistics.jl
    • JuliaSparse/SparseArrays.jl
    • JuliaData/DelimitedFiles.jl
    • JuliaIO/Tar.jl
    • JuliaSparse/SuiteSparse.jl
    • JuliaCrypto/SHA.jl
    • JuliaIO/ArgTools.jl
  • JuliaData/DelimitedFiles.jl in particular, is singled out in the 1.9 highlights blog post as the first stdlib to be upgradable.

Feel free to use any part of the above in case it helps!

[adigitoleo]

Thanks, @waldyrious, I've added I will add a commit to add mention of these things:

as well as various unpackaged submodules.

I have called them "useful" submodules instead of using the adjective "unpackaged" which while being correct may have unintended negative connotations ("unmaintained"). The distinction between versioned stdlib modules and unversioned Base and Core modules should hopefully already be clear from the following h3 subsection.

Full list

This made me realise that the list in the current docs sidebar doesn't use the canonical names ("Random -> "Random Numbers"). So really there is no duplication, the list I'm providing gives the exact names to be used with using/import.

and are listed in the STDLIBS_EXT variable

Ah, I was missing these! They don't appear in the docs sidebar. I wonder if it would be confusing to add them to this listing then, but I feel like I should. After all, those modules are "installed alongside the Julia runtime" by default, and so fit the bill. The links should go to URLs of the respective package docs. Maybe I'll separate the STDLIBS and STDLIBS_EXT listings under distinct headers to clarify. I probably won't worry about distinguishing those that are maintained by the org from those that are not.

Edit: Nope, they are all there, even in the sidebar. STDLIBS and STDLIBS_EXT from the Makefile all get directories in Sys.STDLIB, so all good. I don't think I'll bother with the distinction in that case.

[waldyrious]

I have called them "useful" submodules instead of using the adjective "unpackaged" which while being correct may have unintended negative connotations ("unmaintained").

Yeah, good point. I think it would be nice to take the opportunity to highlight the difference between a module and a package, but I understand if you consider that to be beyond the scope of the current issue/PR.

So really there is no duplication

I'm sorry if I gave the impression that there was duplication! I just listed the various "types" of stdlib modules to clarify what it means for a module to be part of the standard library, in terms of where the source code comes from (where it is developed), how it's integrated into Julia, etc. I do think the distinction is worth documenting, although perhaps this could be considered developer documentation; in that case, I wonder if it would make sense to include at least a hint about it, say somewhere around here.

Also, I think it would make sense to mention the actually external libraries (STDLIBS_EXT notwithstanding), i.e. the *_jll wrappers, since those too are bundled functionality that comes with Julia by default.

[adigitoleo]

highlight the difference between a module and a package

I had assumed this would already be covered somewhere, but the best I could find quickly right now was here. Even if it is not yet in the manual, I feel like "What is the difference between a module and a package?" could probably belong in the FAQ section, rather than the landing page. I think I'd have to check the exact differences again myself to be honest, so I'd rather set it down for another PR.

I'm sorry if I gave the impression that there was duplication!

No it's fine, while I was writing the PR I thought "wait, why am I doing this if it is actually all in the sidebar?" but your list actually made me realise that a list of the canonical names was not present anywhere 😄

I think it was informative to find out that there are some simple env vars to look for in the makefile that show which stdlibs are developed out-of-tree. I agree that this is probably developer documentation, maybe related to the pacakge vs module point. I hesitate with this only because e.g. STDLIBS_EXT is not recorded in the runtime anywhere (that I can find) and I'm not sure how much sense it makes to ossify that list into the manual.

For the _jll, I agree, I'd want to at least add a note that says that Julia includes various wrappers for venerable (not sure if that's the right word) libraries that are written in other languages. And maybe just point to Sys.STDLIB as the place to look for all of those? Or if I do include an explicit listing, maybe in some other section that relates to writing wrappers?

[waldyrious]

venerable (not sure if that's the right word) libraries

I might go instead with "established" or something along those lines.

And maybe just point to Sys.STDLIB as the place to look for all of those? Or if I do include an explicit listing, maybe in some other section that relates to writing wrappers?

IMHO a list would be more friendly to the reader, than sending them to consult the source code. I don't think it's necessary to write about how to produce such wrappers (as part of this PR, I mean — of course it would be a welcome addition), but just mentioning which ones do exist would be useful information in the context of the stdlib overview.

[adigitoleo]

Now with stdlib JLL listing:

[adigitoleo]

When it is merged BinaryBuilder docs should change the link which currently goes to the source folder:

Edit: Bottom of this page: https://docs.binarybuilder.org/stable/jll/