Support installing manpages (and potentially other files)

[joshtriplett]

I use Cargo to build a command-line tool written in Rust. I've also written an accompanying manpage, command.1. I'd like to tell Cargo about manpages via Cargo.toml, so that cargo install will install them to the appropriate location (gzip-compressed and installed to ${root}/share/man/man1/command.1.gz, where the section 1 in man1 gets extracted from the first character of the manpage extension).

Note that once Cargo has a mechanism for a post-build script, some packages may wish to use that to build the manpage. For instance, a package might parse --help output to generate a manpage, process some other document language to produce a manpage, or use help2man-like functionality from an argument parser. However, for a first pass, I'd just like to have support for installing manpages already written directly in manpage format.

[joshtriplett]

Thinking about this, I think that with the specification hashed out, this becomes an E-easy issue. Given that, I'd like to propose a more precise specification for how Cargo should handle this:

cargo install should automatically detect manpages in a man subdirectory adjacent to Cargo.toml. Manpages should match the glob man/*.[1-9]. Given a manpage man/command.N, cargo install should compress it with gzip (with no timestamp and no embedded filename, as though with gzip -9n), and install it to share/man/manN/command.N.gz relative to the install root.

Optionally, cargo install could gain a new option --no-man to disable installing manpages.

(I think it makes sense to use convention-over-configuration for this. If someone has a use case for specifying additional manpages by filename or directory name, we can always add a new field to Cargo.toml at that point.)

[davidszotten]

just to note that bash (or other shell) completion might be another reasonable "potential other file" for a binary to ship

[alexcrichton]

@joshtriplett yeah I agree that once we have a specification for this it's an easy bug to knock off, but unfortunately the specification is the hard part. I don't think we'll want to extend Cargo to install just man pages (if at all) as as @davidszotten mentioned there's other things like bash completions, perhaps documentation, and even files like assets in games.

We don't want Cargo to become a replacement package manager for the system or other "proper" installation methods, so I'm hesitant to make progress on this as well. The purpose of cargo install was a cross-platform method of quickly sharing code with others using Rust.

This probably wants to be discussed a little more broadly before trying to tackle each of the sub-issues associated.

[joshtriplett]

@alexcrichton I can understand that, but on the other hand, cargo doesn't quite yet support the constraints of distro package managers either, though folks are working on that. And in any case, distribution package managers don't support installing into $HOME, while cargo does.

I definitely wouldn't want to use cargo to install anything systemwide into /usr; however, man will also automatically find manpages installed into ~/.local/share/man/manN/command.N , so I'd like to have this support in cargo for installing into $HOME.

For that matter, many distribution package build systems just have the package install itself into a temporary directory via $DESTDIR, and then tweak and package up that temporary directory. So, if cargo knows about various assets to install, the distribution package build system can just use that.

[alexcrichton]

Yeah right now we just don't want to open the floodgates to a whole litany of various post-installation steps of Cargo just yet. It may be worth prototyping this as a new subcommand (perhaps leveraging package.metadata to store configuration) before moving into Cargo as well.

[vu3rdd]

We could create a .cargo/share/man directory where the cargo install can install the manpage into, depending on the section (inferred from the name of the man page - foo.N). And then, as part of the installation of rust/cargo toolchain, we could add .cargo/share/man into the MANPATH environment variable.

How does that sound?

[jugglerchris]

I have a use case where I want to install extra non-manpage files, but need the binary to know the path to them somehow (ie when installed needs to point to the install location, otherwise to somewhere in the source tree).
I might be up for prototyping a cargo subcommand.

[joshtriplett]

We could create a .cargo/share/man directory where the cargo install can install the manpage into, depending on the section (inferred from the name of the man page - foo.N). And then, as part of the installation of rust/cargo toolchain, we could add .cargo/share/man into the MANPATH environment variable.

You don't actually need to add $TARGET/share/man to $MANPATH if $PATH already contains $TARGET/bin; man will automatically look for ../share/man relative to directories on your $PATH.

But otherwise, cargo install installing manpages named foo.N into $TARGET/share/man/manN/ sounds perfect.

[cardoe]

I'd like to see this relative to the --root argument to cargo install so in the default case it goes to $HOME/.cargo/share/man so that it just works for people that have $HOME/.cargo/bin in their path. But that's cause I'm lazy and use cargo install --root=${D}/usr && rm /usr/.crates.toml for packaging.

[casey]

I propose that we tackle this issue in two steps:

1. Decide on a standard way for crate authors to ship man pages with their crate. I think that man/*.[1-9] in the crate root would be a very reasonable way to go.

2. Later decide if we want to extend cargo to actually install man pages and the details of how that would work.

I think that part 1 would be easy and uncontroversial, since there isn't any code to write or behavior to decide on. Also, it would would be very useful on its own for downstream packagers of rust binaries. For example, tools like https://github.com/mmstick/cargo-deb could be extended to include man pages in the .deb, if present.

[joshtriplett]

@casey That sounds good to me. I'd be happy to enable that in debcargo as well.

[cardoe]

On 2/10/17 10:04 PM, Casey Rodarmor wrote: I propose that we tackle this issue in two steps: 1. Decide on a standard way for crate authors to ship man pages as part of their project. I think that |man/*.[1-9]| in the crate root would a very reasonable way to go. 2. Later decide if we want to extend cargo to actually install man pages and the details of how that would work. I think that part 1 would be easy and uncontroversial, since there isn't any code to write or behavior to decide on. Also, it would would be very useful on its own for downstream packagers of rust binaries. For example, tools like https://github.com/mmstick/cargo-deb could be extended to include man pages in the |.deb|, if present. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#2729 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAAih5WT_A-xnOqPpDHk0_k6xfXpgspUks5rbTM4gaJpZM4IkmVI>.

FWIW, Gentoo and Yocto already do this with man pages appearing in man/*.[1-9] because that's how I ship my crates.

…

-- Doug Goldstein

[alexreg]

If we could extend this beyond man pages to general data files, that would be great.

[lilyball]

I think manpages are the most useful thing to handle first, and should be tackled before handling arbitrary other files.

Also, while a man/foo.N convention to auto-detect manpages is fine, Cargo.toml will need a way to explicitly specify the location for manpages as well, as there are other conventions in use (e.g. oghamn/exa uses contrib/man/exa.1, and I've seen share/man/man1/foo.1 in other projects).

[joshtriplett]

@kballard That seems reasonable. Convention first, but also a Cargo.toml key that accepts either a string or list of strings; the strings can specify either directory names (install all manpages in the directory) or filenames (install the specified manpage).

[BartMassey]

In my opinion, once we've gone down the road of having cargo install and a $HOME/.cargo, it is unreasonable not to be able to have the install put arbitrary needed assets in there. I would hate to see this limited to just manpages. The distro package manager is not really a viable substitute for one's own programs under development and in use on one's own machines...

[alexreg]

I fully concur with @BartMassey on the above.

[hasufell]

This is a must-have feature. You might also want to check how cabal handles this:

• in the package definition there is a data-files setting
• you can get the data locations via generated module functions in the code, so it doesn't really matter if it's installed into $HOME, still residing in the build dir or already system-wide (if applicable)

Automagic detection of manpages is fishy, imo. Instead, there should be wildcard support in the corresponding config syntax.

Also, doing this properly and increasing compatibility with package managers will also involve having more fine-grained control over the installation destination (not just --root), because otherwise it will make crate devs hardcode weird and incompatible suffix directories.

I wouldn't want to give full control over the installation procedure to the crate dev. It should not be possible to define arbitrary installation destinations. There must be a set of things, most likely corresponding to the FHS, like --mandir and --datadir, which can be extended carefully, if needed. But this format must be decided upon first in a generic way. As such, I agree with @BartMassey ...it will just make it worse if this is handled per use case.

Or to put it simple: this feature is not a substitute for package managers at all, instead it increases package manager compatibility (if done right).

[alexreg]

@alexcrichton Could we kindly get your thoughts on the recent developments in this discussion? 😊

[alexcrichton]

Adding support for this was explicitly avoided in the initial RFC for this feature. To me that means that adding support will likely entail a further RFC with rationale, detailed design, etc.

[alexreg]

@alexcrichton I didn't see any rationale given there, however. Is there? This seems to be a much in-demand feature... perhaps someone here (more qualified than me) could take up the task of writing an RFC though.

[zeenix]

@alex reading through that link your provided, it sounds to me that installation and uninstallation of data is considered to be only useful for deployments/distros and not devs. As a maintainer/developer of system D-Bus services (Geoclue and gps-share), I need to install (and sometimes also uninstall) files from the source/build directory to test changes etc. It would be very annoying for folks like myself to have to create custom scripts or rpm packages for just being able to install files.

[remexre]

I think that installing to a $DESTDIR (which just so happens to default to ~/.cargo) should be the priority -- creating actual distro packages de facto requires this. I think @hasufell has the right idea, maybe have something like:

# globs supported for all paths
# paths are relative to workspace root directory

[assets]
# defaults to ["man/*.[0-9]"]
# install to mandir
manpages = ["example.1"]

# assets.data defaults to []
# assets.data.FOO = { path = "BAR" } installs BAR to datadir/FOO
[assets.data."icon.png"]
path = "img/icon.png"

[rrthomas]

This depends on the man implementation; it is certainly true for man-db.

This is true for man-1.6g as well (which is what macOS uses). I'm not sure what other implementations are in common use, are there any that you know of that does not have this behavior?

My knowledge does not extend much beyond GNU, sorry.

[lilyball]

As a test, I fired up an Alpine Linux Docker image and ran apk add mandoc man-pages. The resulting man command does not appear to infer paths. man man gives me documentation that states that it is from the POSIX Programmer's Manual, and it includes the following paragraph:

   The historical MANPATH variable is not included in POSIX because no
   attempt is made to specify naming conventions for reference page files,
   nor even to mandate that they are files at all. On some implementations
   they could be a true database, a hypertext file, or even fixed strings
   within the man executable. The standard developers considered the
   portability of reference pages to be outside their scope of work.
   However, users should be aware that MANPATH is implemented on a number
   of historical systems and that it can be used to tailor the search
   pattern for reference pages from the various categories (utilities,
   functions, file formats, and so on) when the system administrator
   reveals the location and conventions for reference pages on the system.

This doesn't actually seem to be the correct manpage for mandoc though. I checked online and found https://mandoc.bsd.lv/man/man.1.html which does state that MANPATH is supported.

So basically, POSIX does not include MANPATH support in its specification, but it's supported by man-1.6g, man-db, and mandoc. Of those, man-1.6g and man-db infer search path from PATH whereas mandoc does not. All three of these implementations will print the current search path when invoked as man -w, which allows for testing whether path inference works if desired. I do not know offhand of any other man implementations to check.

All that said, I'm still of the opinion that anyone running a man implementation that does not infer paths is responsible for updating their own MANPATH (or /etc/man.conf) as appropriate.

[gasche]

For the record, I have a MANPATH on my Fedora machine that appears to have been refined by nvm and opam. Looking further, it looks like nvm (a Ruby package manager) adds to MANPATH when it is already defined (see the documentation) while opam (an OCaml package manager) adds to MANPATH on Linux, but not on OSX/BSD systems (see opamEnv.ml).

[gasche]

Personally I think that making any progress on this issue would be better than stalling because there is no perfect solution. If the idea to place manpages relative to bin is good enough for most systems, it sounds like a fairly good choice, especially given that:

• the location it suggests is the natural one in ~/.cargo
• users that have other man programs can always set MANPATH manually, which is very easy if there is a standard path for cargo installs.

[lilyball]

Nix also only sets MANPATH if it's already set in its profile script (and it also only does this for single-user installs; the multi-user install profile doesn't even bother to set MANPATH at all).

[BartMassey]

Nix also only sets MANPATH if it's already set in its profile script

I think this suggestion is reasonable: Rustup checks the environment for a MANPATH and offers to extend it if so. I'm also OK with staying out of the MANPATH business: like I say, I don't let Rustup touch my very ancient and convoluted dotfiles anyhow, so it won't affect me personally.

Anyhow, sounds to me like we can try to move forward now if somebody wants to do the manpage patches to Cargo. Messing with Rustup can wait until later, if ever.

[devanoneth]

Great to hear we have made some progress in understanding here. FWIW, brew (de facto macOS package manager) also extends MANPATH. We'd be aligning well with many other package managers.

[vilgotf]

Following XDG on linux and placing man pages under $XDG_DATA_HOME/man would be what I (as a linux user) would expect cargo to do, see #1734

[BartMassey]

@vilgotf Point taken. However, given that Cargo is already placing executables in $HOME/.cargo/bin, I think $HOME/.cargo/man makes the most sense after the discussion above. You'd be welcome to symlink $HOME/.cargo/man to $XDG_DATA_HOME/man if you like (on Rust 1.58.1 or later, of course 😅).

[gasche]

I think we shouldn't tie the two discussions together ((1) installing manpages and (2) supporting the XDG specification). It's already hard enough to make progress on (1) (this issue will soon celebrate its 6th birthday), and it does not make sense to use XDG_DATA_HOME/man with all other files going to ~/.cargo.

I think the proper way to handle this is for interested people to push both discussions independently, without interfering with each other (after checking that doing one is not a fundamental blocker for the other), following the obvious "first merge wins" principle:

• if cargo changes to respect XDG by default first, we can change the man-install logic to respect it on XDG-using systems
• if cargo changes to install manpages by default, the in-progress XDG-support work can include a new change to the manpage logic (in addition to the other changes to cargo install directories).

[pmetzger]

I will agree with @gasche above and say again: the perfect is the enemy of the good. I suggest a very simple initial solution that gets slowly improved beats a never completed perfect solution. Even if the initial mechanism is extremely imperfect, it can be improved with experience.

[remyabel2]

Every project does things differently. For example, for exa I have to do:

pandoc --standalone -f markdown -t man $HOME/.cargo/registry/src/github.com-1ecc6299db9ec823/exa-0.10.1/man/exa_colors.5.md > ~/.local/share/man/man5/exa_colors.5

Whereas zoxide already has it in native format and I can just copy it over. And bat has a create.sh script that takes *.1.in files and creates *.1 files. Furthermore, it seems like some projects have man page generation disabled right now because of some issue with clap.

Typically the way that this would be handled on the distro package management side is that they generate the man pages via XYZ build steps and specify the manifest of files that are expected to be generated. For cargo the onus is obviously on the developer as they write the build scripts, I suspect that because everyone does things differently there will be situations where cargo will fail to properly install man pages. At this point though cargo is now just replicating the functionality that is already well handled by distro package managers, I suspect a more agnostic approach is to do what people have been doing for years with Makefiles and install assets to standard directories. Binaries go in $PREFIX/bin, assets/documentation/man pages go in $PREFIX/share and so on and so forth.

I am +1 on not overcomplicating things and just doing XDG_DATA_HOME/man. The sooner the better to avoid the need for hacky workarounds.

[matklad]

As a drive-by summary, it seems like there are some under-explored opportunities to improve the situation, which do not require coordination from Cargo, and which are not blocked on anything:

• Support installing manpages (and potentially other files) #2729 (comment), document "suggested convention" for where in the repository man pages and such should be found. The value here is to cause various binaries to converge on the same convention, which removes mental overhead if you do packaging of many crates "by hand", and makes this potentially automatable. Example to learn from here is cargo-xtask, a polyfill for cargo workflows which, while a hack, did align the ecosystem somewhat.

  Action Item: consider all use-cases surfaced so far, write a spec which serves them, publish the spec as "advisory document", advertise the doc here, on reddit, and on irlo/urlo, and let the community iterate from a well-defined starting point.

• Support installing manpages (and potentially other files) #2729 (comment) document "suggested metadata format in Cargo.toml", which opens further avenues for automation by, eg, distro packagers or a custom cargo command

• Support installing manpages (and potentially other files) #2729 (comment) implement a custom cargo subcommand which take advantage of the above conventions to provide cargo install-like experience with manpages. With such a tool, a readme for hypothertical Rust cli utility called fabulous would look like this:

  # You can use `cargo install` to install the binary
  # Note: this won't install man pages or completions
  $ cargo install fabulous
  
  # To install with shell completions, use `cargo-install2`
  $ cargo install cargo-install2 && cargo install2 fabulous

  Existence of such cargo-install2 would make the experience of users of rust utilities better, and it would be a strong data point for eventual RFC and inclusion the functionality into cargo proper.

  Action Item: implement cargo install2, advertise that.

• As oftentimes manpages and completions are generated programmatically via a "post-build" script, documenting "suggested convention" for this is useful for all the same reasons. (to spitball, the convention might be "run cargo run --manifest-path ./build-dist/Cargo.toml to produce a ./dist folder, which would contain all the files to be packaged in a binary distribution")

cc @rotty, it seems that what you are doing with https://gitlab.com/rotty/cargo-parcel is basically this :-) The only thing missing is advertising the thing (eg, on this very issue), so that the community can run with the idea

[herzenschein]

Hi. I'm from KDE exploring the potential of using Rust for our apps.

This sort of issue is a dealbreaker for GUI desktops apps on Linux. The inability to install arbitrary data files to specified XDG directories means desktop apps cannot install the following using standalone Cargo:

• manpages / docs (the title of this issue)
• icons
• metainfo / appdata, which results in:
  • no descriptions in app stores
  • no changelogs in app stores
• desktop files, which results in:
  • no window icons on Wayland (because XDG Shell demands association between binary and desktop file for this)
  • no task manager icons on Wayland (for the same reason)
  • no additional application actions
• QML files (needed for our QtQuick apps) (Edit: after some further checking, apparently cxx-qt embeds QML files into the executable, so this install path is no longer needed)

Just installing a binary is not enough.

just and cargo-make are too free form for this and require reimplementing common needs for most GUI apps. The closest thing to what GUI apps need that I've found is cargo-parcel, which provides:

• installing data files together with the binary
• install prefix logic (needed to allow, for example, installing in /usr, ~/.local or other relevant prefixes)
• at least the XDG base dir for binaries (which is still limited and not ideal, we need prefix/share/{applications,metainfo,icons,man} etc, issue)

Which is almost what we need, so I'd like to bump visibility for this project here.

GNOME has worked around this issue for years by simply using Meson on top of Cargo to install files, and COSMIC has been using Just and reimplementing the needed XDG dirs + prefix logic for each project in their justfiles. I have no idea how the other Rust GUI toolkits have been managing this sort of thing.

Whether we end up having support for installing files this way via Cargo or cargo-parcel, this functionality would surely help both KDE and COSMIC (GNOME would probably just keep doing their thing supporting and contributing to Meson). I just wanted to express why this sort of functionality is needed.

As for how this is done on the distribution side of things on Linux, take a look at these examples from Arch (just because they're more easily introspectable):

• plasma-workspace using CMake
• eye-of-gnome using Meson
• cosmic-text-editor using Just

Basically we need an install step that already follows the XDG basedir for our downstream packagers (preferably with variables to override behavior), otherwise we end up forcing them to use a lot of install -Dm<permission> calls and similar.

Edit: for crossreferencing, this is tangentially related to #1734.