tools/geneos - "Pull" Package Repository

[itrs-dsobiepan]

For security reasons, the customer environment has locked down all “run-time” IDs (we call them functional IDs, or funcIDs) to prevent “interactive login”. This means that the “no_ssh” rule is applied to all funcIDs on all servers.

This will prevent the “remote management” functionality of GENEOS Netprobe installations.

==

As such, the request is instead of “pushing” commands/packages/actions from a central location, we are looking to “pull” the package on the Netprobe agent from a central location.

Within the customer, we have the ability to create a “geneos_sftp” ID... this is a special type of funcID which doesn’t ‘run’ any processes, and thus is permitted “SFTP login” (not a member of no_ssh group), but restricts “interactive login” by setting the SHELL=/bin/false.

The install command will, with no other options, download and unarchive the latest releases of each currently installed Geneos components from the official ITRS download servers

It would be beneficial if we could provide a URL of the form sftp://[user@]host[:port][/path] ... similar to the --host feature ... and we can provide a passwordless connection (register netprobe server “pub-key” with SFTP ) ... similar to the requirement for --host feature ... and reuse the existing cordial/geneos logic:
*) identify the components installed locally (or being requested to be “first-time” installed)
*) scan the remote repository (ITRS download server, or provided SFTP server) for “latest” package that matches component and OS-RHEL-revision (el7/el8/el9) ... or the “specific version” provided in the --version argument
*) download package/unarchive
*) “do the rest of things it do”

Similar to how the --local command exists, perhaps a --privateSFTP option?

Additional: A “-no_exec” or “-dry_run” option, with return-code/exit-code which indicates “THERE EXISTS A NEWER VERSION AVAILABLE” would help us to script a scheduled process to:
*) check if a version is available that meets the conditions that would trigger a new download (from the official site); if yes:
*) download/install new package version
*) upgrade running instance (once download/package install is complete)
*) clean-up older version

This will help us with “scheduling” our software currency... and each application/environment can schedule this currency script on their server(s) for when their application would experience the least impact, and monitoring window is “light”.

[itrs-dsobiepan]

I've had an edited version of the above sent to me by a customer using cordial - happy to connect you with them if this is feasible and if you have questions about their environment, @pgalbavy

[pgalbavy-itrs]

This should be do-able. Adding a new URL type. The only question is how easy it will be to list the remote directory and match the requested archives, but I think that's straight forward from the sftp package side.

[pgalbavy-itrs]

The basics now work in my local codebase, but there are lots of interactions between options and user messages need updating as well as docs.

Question however around paths; There is no official ssh or sftp URL standard that I can find, but looking at the ftp:// scheme there is still ambiguity over absolute and relative paths. e.g. sftp://host/path/to/dir or sftp://host//path/to/dir and sftp://host/./relative/path or even encoded sftp://host/%2Fpath/to/dir ? Do we need the complication of relative paths?

[pgalbavy-itrs]

Working on this further has opened a can of worms. I now find that while geneos package install is supposed to support plain URLs, it doesn't in fact do so. I am using this opportunity to consolidate some of the code so that URL handing simply includes ssh/sftp as schemes, but I also need to then add the ability to securely specify passwords / passphrases from the user to unlock private key files if an agent is not in use.

[pgalbavy-itrs]

After some experimentation and code review it looks like this will need more than a simple addition of new code. The existing code base has a number of different paths for "downloading" files, releases or otherwise.

There is a common openSource() function that can handle URLs and also "special" forms for STDIN and home directory shortcuts, however this is not used when downloading official or nexus housed released, which have their own functions for handling, mostly, the search features of the respective HTTP sources.

I propose to unify as much of this code as possible and make ssh:// and sftp:// style URLs work alongside the more normally understood ones but also add logic to look at the global download::url configuration value, which defaults to the official releases download URL. If this is changed then the rules governing the searches for component type, version and platform should not be used.

The simplest way to do this is to have a match table on source URL paths which then call different functions to do the search work and open the source, returning the io.ReadCloser, the filename, filesize etc. If not matched we just use the openSource() (or newly named function) instead.

This also has the side effect of allowing other file opening features (such as import) to use the same patterns.

Remaining is how to consolidate the secure handling of authentication methods, which as the moment is a mix of the user's .ssh directory and parameters passed in.