Redesign link creation for installer of CLI tools

[hgy59]

Some times ago I introduced SPK_COMMANDS and SPK_LINKS makefile variables to create links in /usr/local/bin and other destinations (see #3852).

It is now time for a redesign and to use the officially documented "Resource Config" with so called usr-local-linker resources.

Benefits

• it is officially documented and supported
• links are created on service start and removed, when service is stopped
• to recreate broken links a package restart is sufficient (no re-installation required)
• It is supported since DSM ? (at least 6.0) and does not need different implementation for DSM7

Downside:

• The SPK_LINKS feature to create links with different names is not supported (to use the new way, all links with different names must be installed/created in the package target folder and linked from there to /usr/local/...).

Brainstorming:

• create resource file from SPK_COMMANDS regarding the different folders (bin, sbin, share, lib, ...)
• SPK_LINKS can be used to create links with different names in the package target folder (i.e. mc -> mc-utf8, vim8 -> vim-utf8, ...)
• the resource for data-shares defined in wizards sould be generated by the framework too (already implemented on dsm7 branch)
• The dsm7 implementation is incomplete,
  • all links defined by SPK_COMMAND are defined in /usr/local/bin. The target folder is ignored and links to sbin, etc, lib, ... are not supported
  • SPK_LINKS is not (yet) implemented
• the tool jq to generate json from bash scripts is too limited to implement multiple usr/local folders from SPK_COMMANDS variable
• use python script to generate resource file from Makefile variables

[th0ma7]

I find this rather interesting but clearly I have to read about it as I haven't looked as DSM7 symlink nor followed-up on your SPK_LINKS variables... So I have a bit of catch-up before being able to comment.

[hgy59]

@publicarray as I wrote a comment and a hint I am not sure whether it is a good or a bad idea to switch from sc link creation to official usr-local-linker.
Did you switch to usr-local-linker for DSM7 due to missing permissions to create the links as we did before?

[publicarray]

Yes, writing files outside the package directory directly are not permitted (except when allowed through settings, for example shared folders) the only way to do that now is through resource "workers". I think giving a package "system" permissions may also work but I've not messed with that (I don't think it's the right approach but may be used as a band-aid with bubblegum).

jq to generate json from bash scripts is too limited

I think if more advanced features are required package developers can create the JSON resource file themselves, and we can add more variables to define the other resource types. lib, includes

If we can I would like to avoid including python. just my 2 cents

[hgy59]

If we can I would like to avoid including python. just my 2 cents

I have written a small python file to do this. We already have python 3 in the development environment, and this script runs on the development system only.

[publicarray]

Cool, My only concern would be is that we make spksrc more complicated for new contributors. Maybe makefiles/shellscripts where not the right choice and everything should be python or golang etc. But that's' for another topic. Or maybe is just me talking with little python experience. 🤷‍♂️

[publicarray]

Actually I'm interested to see the implementation. Maybe find and jq can do the same thing? jq can do some very advanced stuff: https://stedolan.github.io/jq/manual/
If Python is simpler I'm happy to accept / give in to Serpents 🐍

[hgy59]

@publicarray just added the script to the dsm7 branch.
I hope it is readable (it is my first python code ever written).

[publicarray]

@hgy59 Thanks! and congrats. only thing missing is the read write permissions for shared folders. Thanks for pushing this because while looking over the code again I've simplified the jq command for SPK_COMMANDS.

Sorry for asking but I don't think I understand why is jq not good enough?

[hgy59]

your current implementation does not regard different folders.
the python script handles all kind of folders like:

bin/tree => "bin": [ "bin/tree" ]
sbin/mount => "sbin": [ "sbin/mount" ]
lib/whatever => "lib": [ "lib/whatever" ]

[publicarray]

Thanks, I see the difference now. but AFAIK sbin linexec etc. are not supported by DSM7 so to maximise support linking all binaries should still work fine? I can add lib, etc support though.

If not should we add a filter?

echo '{}' | jq --arg binaries 'bin/foo bin/bar sbin/foooo' '."usr-local-linker" = {"bin": $binaries | split (" ") | map(scan("^bin.*")) }'
{
  "usr-local-linker": {
    "bin": [
      "bin/foo",
      "bin/bar"
    ]
  }
}

[hgy59]

AFAIU the dsm developer guide, the supported folders are bin, lib, etc, ... - so the list is not conclusive, and sbin may be supported as well (will have to test this).
EDIT: not all folders are supported, see my next comment.

for my dotnet-sdk evaluation I had to create a custom resource file anyway, as this is neither supported by jq nor by my python script.

{
  "usr-local-linker": {
    "bin": [ "share/dotnet/dotnet" ],
    "share": [ "share/dotnet" ]
  }
}

the second entry ("share") is supported by the python script.
for the first entry, an additional link must be created by the installer (bin/dotnet => share/dotnet/dotnet) to support both the jq solution and the python script.
SPK_COMMANDS = bin/dotnet share/dotnet

I would rather avoid a filter and make it open for any /usr/local subfolder.

[hgy59]

@publicarray I must revoke my comment above:

Just found that

{
  "usr-local-linker": {
    "share": [ "share/dotnet" ]
  }
}

does not create a link in /usr/local/share.
This is a confirmation that not all folders unter /usr/local are supported (as you assumed for sbin).

I didn't realize this, as dotnet-sdk is working as expected (without the link to /usr/local/share):

$ dotnet --info
.NET SDK (reflecting any global.json):
 Version:   5.0.102
 Commit:    71365b4d42

Runtime Environment:
 OS Name:     Linux
 OS Version:
 OS Platform: Linux
 RID:         linux-x64
 Base Path:   /volume1/@appstore/dotnet-sdk/share/dotnet/sdk/5.0.102/

Host (useful for support):
  Version: 5.0.2
  Commit:  cb5f173b96

.NET SDKs installed:
  5.0.102 [/volume1/@appstore/dotnet-sdk/share/dotnet/sdk]

.NET runtimes installed:
  Microsoft.AspNetCore.App 5.0.2 [/volume1/@appstore/dotnet-sdk/share/dotnet/shared/Microsoft.AspNetCore.App]
  Microsoft.NETCore.App 5.0.2 [/volume1/@appstore/dotnet-sdk/share/dotnet/shared/Microsoft.NETCore.App]

To install additional .NET runtimes or SDKs:
  https://aka.ms/dotnet-download

So my suggestions are:

1. keep your solution for SPK_COMMANDS implemented with jq
2. drop an unsupported error when SPK_LINKS is defined for DSM7+
3. (optionally) support a new Makefile variable like USR_LOCAL_LINKS (or SPK_USR_LOCAL_LINKS) for explicit definition of the target folder below /usr/local.
   For my dotnet example above, this would look like: USR_LOCAL_LINKS = bin:share/dotnet/dotnet
   This could be implemented by jq or python.

[publicarray]

It's possible to do with jq, however it's not necessarily readable though:

Edit: new version

echo '{}' | jq --arg links_str 'etc:var/foo lib:libs/bar etc:share/baz' '."usr-local-linker" += ($links_str | split (" ") | map(split(":")) | group_by(.[0]) | map({(.[0][0]) : map(.[1])}) | add)'
{
  "usr-local-linker": {
    "etc": [
      "var/foo",
      "share/baz"
    ],
    "lib": [
      "libs/bar"
    ]
  }
}

https://stedolan.github.io/jq/manual/

. - is kind of like the 'this' keyword in other languages
as $var - is an assignment 
map() - to iterate over an array or object
...

[hgy59]

solved by SPK_USR_LOCAL_LINKS in spksrc.service.mk