[RFC] Installation and sharing of resources

[mosteo]

I'm playing with an hypothetical new alr install that relies on gprinstall. It seems to work well. What I'm currently doing is:

• gprinstalling by default to ${CRATE_ROOT}/alire/prefix.
• Exporting that part as ALIRE_PREFIX in our environment.

I tried the for Artifacts ("path") use (...) of GPR files and also works as expected. So, an executable run within our alr printenv can locate shared resources via ALIRE_PREFIX.

But, all in all, I have not much experience with installation and gprinstall. I have some fuzzy ideas were I'd like to hear your feedback @Fabien-Chouteau (and interested parties, e.g. @stcarrez):

• Should we add something explicit to the manifest about exported resources?
  • There seems to be no need if relying on gprinstall.
  • But it could be useful for e.g. the automatic generation of documentation.
  • If so, it could be a simple flag,
  • or it could be a list of the paths being installed.
    • Would we in that case want to take over from gprinstall and install resources ourselves? Or at least verify?

My stance on this right now lends to not duplicating the work of gprinstall. At most have an export-resources flag, or even peek into project files for Artifacts clauses (that could be too hackish if there are cases involved).

Separate issue: at present, gprinstall is only called through alr install but:

• Might we want to call gprinstall after each gprbuild (it seems fast enough)?
  • We could in this case export a couple extra paths for binaries/libraries in alr printenv
  • Crates would not need to export paths explicitly for executables they build.
    • Our current situation is probably not ideal wrt to build profiles possibly generating executables in different places.

In this regard, I suspect that gprinstall is probably very solid and will simplify our lives down the road. We are currently doing everything in-tree, which I'm wary of (except for edition were we want the actual sources and not some installed headers -- I've been bitten by this in the past and it was a pain).

[mosteo]

Down the road, alr install <crate> might serve to deploy crates to a common default user-wide location, a la cargo install, e.g. ~/.config/alire/prefix, even if we were to use gprinstall internally by default during build.

[mosteo]

I have pushed 94749df that gprinstalls any project that's gprbuilt, and it seems to work as expected.

A side effect is that now binaries appear both in-tree and in the internal installation prefix. So alr run should look for executables only in one of those places (currently only in the installation prefix).

[Fabien-Chouteau]

I still don't have I a clear idea of what is needed, and when :)
In my opinion we should wait for a more "complete" design before adding an alr install command.

In terms of resources there are two situation as far as I can see: During development, where crates can be located anywhere (because of pins), and after the install where tools, libs, and resources will be in a common location.

I don't think that doing gprinstall after every builds, or before every run, is an option that will scale. Installation can move huge amounts of data around.

So we need a way to make things work in both dev and install scenarios.

Something that we discussed during FOSDEM was to use an environment variable to define the location of resources for each crate. For instance we could have as part of the alr environment:

• CLIC_PREFIX=/tmp/my_crate/alire/cache/dependencies/clic_0.2.0_5a1f3232
• ADA_TOML_PREFIX=/tmp/my_crate/alire/cache/dependencies/ada_toml_0.2.0_f07de487
• etc.

Each crate that needs to access resources could then use these env variables to get resources path. And we can develop a resources crate to make that task easier (see below).

Now what to do after installation? The *_PREFIX envs will not be defined. An option is to guess the prefix from the directory where the executable is located. GNATcoll has a function for that (Works on Linux, Windows and macOS only see here ). This is what we do for startup-gen here. A resources crate could also make that easier.

Looking at other ecosystems:

• Python provide modules to handle resources.
• There was a discussion on Cargo for man pages, so not something used by the exe/lib itself from 2016 to last month...
  • Looks like cargo install explicitly avoids the subject
  • The discussion doesn't lead to any implementation
• In the Rust discussion someone points to Caba : data-files which seems to be based on generated code.
• Rust as an interesting resource crate. But I am not sure we can replicate that in Ada, and having all the resources hard-coded inside the executable is not a a good option for everything. I don't see any solutions for Rust.

A side effect is that now binaries appear both in-tree and in the internal installation prefix. So alr run should look for executables only in one of those places (currently only in the installation prefix).

Maybe alr run should only look in bin/.

generic
   Crate_Name : String;
package Resources is
   
   function Prefix_Path return String;
   
   function Resource_Path (Relative_Path : String) return String;
   
end Resources;

with Ada.Text_IO;
with Resources;

procedure Plop is
   package Res is new Resources ("plop");
begin
   Ada.Text_IO.Put_Line (Res.Prefix_Path);
   Ada.Text_IO.Put_Line (Res.Resource_Path ("share/plop/template/plop.tmplt"));
   null;
end Plop;

with GNATcoll.Utils;
with GNAT.OS_Lib;
with GNAT.Strings;

with Ada.Characters.Handling;

package body Resources is

   -----------------
   -- Prefix_Path --
   -----------------

   function Prefix_Path return String is
      use Ada.Characters.Handling;
      use GNAT.Strings;

      Env_Prefix : GNAT.Strings.String_Access :=
        GNAT.OS_Lib.Getenv (To_Upper (Crate_Name) & "_PREFIX");

   begin
      if Env_Prefix /= null then
         return Result : String (1 .. Env_Prefix.all'Length) do
            Result := Env_Prefix.all;
            GNAT.Strings.Free (Env_Prefix);
         end return;

      else
         --  Fallback to a path relative to the executable
         return GNATCOLL.Utils.Executable_Location;
      end if;
   end Prefix_Path;

   -------------------
   -- Resource_Path --
   -------------------

   function Resource_Path (Relative_Path : String) return String is
   begin
      return Prefix_Path & "/" & Relative_Path;
   end Resource_Path;

end Resources;

The dependency on GNATcoll seems overkill here...

[mosteo]

Okay, important information all around.

So, on the one hand releases can already export an env var via manifest to point to their resources during development. So a follow-up question is if we want to standardize the naming of such variables or leave that to crate documentation.

As for alr install, if we use it like cargo install to gprinstall on demand to a default/configurable prefix, I don't see much more to it. I'm possibly myopic about this. I wouldn't try to track installations or reuse such prefixes for dependencies (since e.g. it can't support (I think) multiple versions of the same crate).

This discussion leaves out reuse of large dependencies, that to me is a separate can of worms. I wouldn't do any of that through alr install, to me alr install is putting things out there totally out of alr further business.

What I'm now clear is that using grpinstall as a default step during build is not a sound idea, so I'll remove that from the PR and let it rest while we muse ideas.

[Fabien-Chouteau]

@stcarrez we really need you opinion on this :)
Does the Resources system with a mix of environment variable and exec based path make sens to you?

[stcarrez]

As far as I'm concerned, there are two aspects to consider

• crate installation
• development installation

I really like the idea of having a alr install that installs the crate outside of alr. It makes sense for libraries but also for final programs that are composed of a binary but also composed of other files.

Using gprinstall is fine but too restrictive when you have to perform specific tasks during the installation. Sometimes I need to merge files or generate a file from a template. The gprinstall Artifact does not allow to run an external program that allows such customization. A simple customization could be a simple sed or more complex.

When gprinstall cannot be used, a crate specific installation command similar to the build actions would be nice.

For a development installation the Ada Resources package is interesting but it will not help me.
In fact I'm not even sure it will be usable in real case because you will end up having a binary that is linked to the directory where it was compiled.

For AWA, the framework is using some simple configuration file with a multi-path search mechanisms to find files in possibly several directories. To make it usable with Alire, I would need:

1. either to have all the AWA and its components data files (XML, CSS, JS, ...) installed in a well known location. It could be a set of config, db, and web directories located at the root of the crate directory (or in a well known place).
2. or have a configuration file that is configured with a list of search paths that depend on the components used.

For development, 2) is easier because files are not copied.

For example, I would need some post build command that generates somewhere something that looks like:

root_dir=./
app_search_dirs=#{root_dir}/cache/dependencies/ada-asf_1.5.0_a4a50d40/;#{root_dir}/cache/dependencies/ada-ado_2.2.1_6f7e8c07;#{root_dir}/cache/dependencies/awa_2.4.0_bd345545/awa;#{root_dir}/cache/dependencies/awa_2.4.0_bd345545/awa/plugins/awa-mail;#{root_dir}/cache/dependencies/awa_2.4.0_bd345545/awa/plugins/awa-wiki;...

Such generation must not be made be Alire because that's too specific to the crate which are used.
But, may be a post build command that is executed could do the trick provided that it has enough information about its crate and other crates as well (the environment variables would be fine).

[Fabien-Chouteau]

Thanks @onox

Isn't gprinstall mostly useful to install (shared) libraries (gpr, ads, and so files) in the pre-alire era?

gprinstall can install any kind of project (bin, static lib, shared lib, etc.), see example below. I don't see it a pre-alire era since Alire relies heavily on gprbuild already.

For my use case it would need to be able to copy binaries and the files (shaders and textures) in some specific folders to some folders (bin/, share/) relative to some prefix. /usr/bin/install would be sufficient for this, but I'm not sure about how things should work on Windows.

This is what I do for startup-gen, see artifact in GPR here and corresponding folders here.

gprinstall makes this work on any platform (Linux, Windows, macOS, etc.).

[Fabien-Chouteau]

Thanks @stcarrez

For a development installation the Ada Resources package is interesting but it will not help me.
In fact I'm not even sure it will be usable in real case because you will end up having a binary that is linked to the directory where it was compiled.

That's not my intent, so maybe my example was not clear.
During development Resources will use the <CRATENAME>_PREFIX environment variable. This will be typically set by Alire. So Resources behavior will not depend on the directory where it is compiled, but the directory where the sources are located.

After installation, Resources will use a relative path from the directory where the executable is installed.

The constraint is that executable and dependencies resources have to be installed in the same prefix dir.

For AWA, the framework is using some simple configuration file with a multi-path search mechanisms to find files in possibly several directories. To make it usable with Alire, I would need:

This sounds compatible with what I have in mind.

Sometimes I need to merge files or generate a file from a template. The gprinstall Artifact does not allow to run an external program that allows such customization. A simple customization could be a simple sed or more complex.

I am worried that this will lead us to re-invent the wheel. Alire has some hooks to run scripts/programs at given points (post-fetch, pre-build, post-build, etc.), I think that for your use case it will be a matter of pre-install hook and then gprinstall.

[stcarrez]

I am worried that this will lead us to re-invent the wheel. Alire has some hooks to run scripts/programs at given points (post-fetch, pre-build, post-build, etc.), I think that for your use case it will be a matter of pre-install hook and then gprinstall.

Yes, I agree. Alire must not do it but it must allow me to do it. If the hooks have access to the environment variables when it is executed, then such setup could be implemented by some script or tool.

[Fabien-Chouteau]

Hello all,

I started a prototype of what the resources crate could look like: https://github.com/alire-project/resources

It is based on the whereami project that provides the same feature for Windows, Linux, macOS, BSD, etc.
And there is a way to get location of a shared library so even if the shared libs and executables are not installed under the same prefix, the will each get their own resources.

[Fabien-Chouteau]

Hi all,

The solution described above is now available in the development branch of Alire (3e9da2f and 303de0e) and the associated Resources crate here: https://github.com/alire-project/resources

To give it a try:

$ alr init --bin test
$ cd test
$ alr with resources --use=https://github.com/alire-project/resources

and:

with Resources;
with Ada.Text_IO; use Ada.Text_IO;
with Test_Config;

procedure Test is
   package My_Resources is new Resources (Test_Config.Crate_Name);
begin
   Put_Line ("Prefix_Path: " & My_Resources.Prefix_Path);
end Test;

@stcarrez it would be nice to have you feedback, in particular because you requested the feature.

[mosteo]

So, to summarize my current understanding of these matters, an a few thoughts on related features:

• Resource sharing is solved by a combo of environment and executable location detection, made more convenient through the use of the resources crate.
• alr install could be used to run gprinstall on the local workspace to a default user prefix.
• alr install <crate> could be used to download a regular crate to a temporary, build and install it.
  • For binary crates:
    • Skip build
    • gprinstall an optional project file that defines artifacts for the appropriate places
  • Add new pre-install|post-install hooks for any custom installation steps.
• Sharing of large dependencies could be done with e.g. alr with --shared. That's a separate feature, as these crates would be used as regular dependencies.

In this way, the new alr install has a clear meaning of gprinstalling to a prefix, and essentially nothing already existing changes.

[Fabien-Chouteau]

What's not clear to me yet is what will happen when I want to install tools that depend on two different version of a library.
There is also the question of incompatible crate configuration values.

[mosteo]

What's not clear to me yet is what will happen when I want to install tools that depend on two different version of a library. There is also the question of incompatible crate configuration values.

This is really tricky. I think it merits its own discussion beyond resources. I will open a new issue focused on installation of binaries and libraries.

[mosteo]

I'd say that resources are on the right track, so I'll close this one now, and I invite you all to reopen or open more focused issues down the road.