Set environment for remote workers?

[ericaddison]

Hello Julia-ers!

When adding remote workers with addprocs(), I would really love to be able to send my current working environment along for the new Julia workers. Currently this does not happen for workers added using an SSHManager, though local workers do seem to inherit the same environment.

After browsing the repo, I think that this behavior could (gulp) easily be added by extending the code in managers.jl to set all, or some specified subset, of the current environment variables in the same way that JULIA_WORKER_TIMEOUT is set. Of course that might not be the default behavior, but maybe an option? Maybe something like (but not identical to):

    # the default worker timeout
    tval = haskey(ENV, "JULIA_WORKER_TIMEOUT") ?
        `export JULIA_WORKER_TIMEOUT=$(ENV["JULIA_WORKER_TIMEOUT"])\;` : ``

    envvals = map(key->`export $(key)=$(ENV[key])\;`, keys(ENV))

    # Julia process with passed in command line flag arguments
    cmd = `cd $dir '&&' $tval $envvals $exename $exeflags`

Is this a reasonable thing to propose? Please let me know if this is a terrible idea! Otherwise I will be happy to attempt to implement this. I just wanted to float the idea here first.

Thanks!

[amitmurthy]

I think this should be done via a SSHManager specific keyword arg to addprocs. addprocs(....; env=[]), i.e., the default would be no envvar set explicitly on the workers.

[ericaddison]

That makes sense. I have a functioning implementation at work ... just trying to jump through the right hoops to release it :)

[NeonNeon]

Any update on this issue?

[ericaddison]

Hi! Unfortunately I was not able to get approval to release the work before I left that job. There is some small chance that @samtkaplan may have some info...

[JeffBezanson]

Duplicate of #18074

[mgkuhn]

For the record: Julia 1.6 added a new env=["JULIA_DEPOT_PATH"=>"/depot", ...] keyword argument to Distributed.addprocs that solves this issue.