Notes about environment variables passed to the Scram environment or modified when running the CMSSW executable

The CMSSW executor in WMCore executes a series of scripts, either outside a Scram environment (e.g.: pre-scripts to modify the PSet), or inside this environment (e.g.: cmssw-wm-tools scripts).

The reason to execute scripts outside the Scram environment, is so we don't mix dependencies from WMCore and CMSSW itself. This allow us to run WMCore and CMSSW with different python versions if necessary. The PSet pre-scripts for example, will be executed using an independent COMP version of python and make external calls to scripts that depend on the CMSSW framework inside the scram environment, when needed (as in HERE, for example).

The scram environment used to call executables that depend on the CMSSW environment, will clean up the original WMCore environment and setup the CMSSW framework by default.

Usage example:

    scram = Scram(
        version = "CMSSW_X_Y_Z",             # CMSSW Version
        directory = "path to working area"   # defaults to cwd
        initialise = " . /uscmst1/prod/sw/cms/setup/shrc prod"  # boot strap env command
    )
    scram.project() # creates project area
    scram.runtime() # creates runtime environment

    # run something requiring scram env
    retval = scram("edmFileUtil")
    print(retval, scram.stdout, scram.stderr)

scram.project()(): Creates the CMSSW project area using the scram architecture selected.

scram.runtime() will execute the scramv1 command. E.g.: eval `scramv1 ru -sh` and will copy the environment in a dictionary: self.runtimeEnv

When a command is called inside Scram(), the original environment is cleaned up by default, and the variables defined at runtime() are exported in this new environment. In addition to that, some variables from the original environment are passed to this new environment, which are defined HERE. This includes environment variables needed by HTCondor or OSG tools for example, like _CONDOR_MACHINE_AD, OSG_APP, etc.

WMCore will use the method execute() from an executor object, to run a specific step.

The CMSSW executor runs different scripts:

• preScripts: If a pre-script (this is, a script that is run before the CMSSW executable itself) doesn't need the scram environment, it uses a subprocess call. The main example for this is the SetupCMSSWPset pre-script. Note though that this is called outside a scram environment, this script uses multiple other scripts from cmssw-wm-tools that do depend on the CMSSW environment, so we invoke them inside Scram()

Direct scram pre-scripts use the regular procedure of calling scram.project() , [scram.runtime()] (https://github.com/dmwm/WMCore/blob/a746fe30112dcb17fb336b4fe89cffc8742051ea/src/python/WMCore/WMSpec/Steps/Executors/CMSSW.py#L186) and the Scram _call() method.

• The CMSSW executable: The CMSSW executable (cmsRun) is not executed directly by the executor with Scram(). Instead, it runs a wrapper script that setups its own environment (with a name stepName-main.sh. E.g.: cmsRun1-main.sh). This bash script is written on the fly and defined HERE.

Since this wrapper script is setting up its own scram environment, the script itself is called by simply using a subprocess call. However, some environment variables need to be overridden in order to avoid problems between the CMSSW environment and the WM environment.

The environment override is defined HERE and basically make sure that:

• The WM PYTHONPATH is not passed in the subprocess call (This is to remove the WMCore.zip packages we introduce in submit.sh), we let scram runtime handle anything else related to the environment.
• It sets XRD_LOADBALANCERTTL to workaround a problem at CERN related to the GSI authentication plugin and EOS with XRootD
• It sets the HOME environment variable

After all these changes in the environment, the CMSSW executable is invoked through this wrapper script HERE However, since we clean up the WM PYTHONPATH in the os system, other steps (e.g.: in the stepChain workflow) would fail after this if they can't find the WM libraries, so the original WM PYTHONPATH is put back in the environment after calling the cmssw executable/cmsRun wrapper.