Issue with environments

[jimboid]

Some HPC machines seem to create a problem where the environment is not loaded under a non-login shell. This includes environment variables for the scheduler and even the Linux module system. A simple cure is that the user can add "source /etc/profile" into their ~/.bashrc file. Machines where this doesn't happen seem to have a .bashrc setup to reference the various environment files by default.

Simply asking for a login shell does not seem to work either, this appears to be specific to shells requested through a script. I'm not sure if there is some sort of safeguards preventing certain sessions from having a login shell (perhaps the tty issue).

There are several solutions to this:

1. Pass information to the user through documentation, informing them to how to add the relevant line to their .bashrc
2. Append the line "source /etc/profile" to every line that gets passed through to subprocess (not sure if this is bad)
3. Find out if there is a proper way to get a valid login shell when using python subprocess.

It is unclear at this stage which solution is the best one, more work and investigation is required so that we can support as many machines as possible.

[jimboid]

It has been reported that referring to the /etc/profile file does not work on at least one machine. This highlights the difference in how these environments are setup. There appears to be some range in how sys admins are linking various files and under what conditions.

This means that on the original comment under this thread only points 1 and 3 remain valid. These are copied below along with a new idea to investigate.

1. Pass information to the user through documentation, informing them to how to add the relevant line to their .bashrc. Have to be careful here, not all Linux distros, and not all sys admins do things the same. There is a bit of variance in how things are done so any solution put on the user needs proper instructions so they can deal with it.
2. Find out if there is a proper way to get a valid login shell when using python subprocess.
3. A better way could be as part of the tests on the machine, to include handling for when Linux modules can't be found. Then in this case, tests on the various files could reveal how the environment is loaded and then amend the ./bashrc

[jimboid]

Newer generation of Hartree machines have a slightly different environment to machines that Longbow is currently used to operate with. It appears from initial discussions that there is a module hierarchy that has to be loaded to get access to other more system specific modules.

We should look at supporting this, it could be that supporting this properly will actually resolve the initial situation in this ticket.

[jimboid]

Changing the connection tests to incorporate testing of the environment could be the best way. Here is a script that can identify this problem and "fix" it. In Longbow the fix could be done by appending a flag to toggle the sourcing command to on or off. When on this command is pre-pended to commands delivered over SSH.

#!/usr/bin/env python

import subprocess

machines = {
    "scarf": ["ssh", "-t", "scarf"],
    "archer": ["ssh", "-t", "archer-c01"],
    "iden": ["ssh", "-t", "hartree-iden"]
}

for machine in machines:

    print "Testing for environment on " + machine + "..."

    cmd = list(machines[machine])
    cmd.append("module avail")

    handle = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE)

    stdout, stderr = handle.communicate()

    if handle.returncode != 0:

        if "bash: module: command not found" in stdout or "bash: module: command not found" in stderr:

            print "Environment is borked."

            print "attempt fix"

            cmd = list(machines[machine])
            cmd.append("source /etc/profile; module avail")

            handle = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE)

            stdout, stderr = handle.communicate()

            if handle.returncode == 0:

                print "env is now ok"

            else:

                print "still busted."

    else:

        print "Env is ok"