engineering_tools

A repository of scripts, configuration useful for the PCDS team

Push updates

git push -u origin master

Add Tag

git tag -a R{tag} -m '{comment}'

git push -u origin R{tag}

Creating a new release

# Clone the source code into a new folder
git clone https://github.com/pcdshub/engineering_tools.git R{tag}
# Enter repository
cd R{Tag}
# checkout tag number
git checkout tags/R{tag}

Updating latest

# Go to latest checkout
cd engineering_tools
# Pull latest from master branch
git pull origin master

The scripts

afs-remote-fix	usage: afs-remote-fix Will change your afs remotes from e.g. origin /afs/slac.stanford.edu/g/cd/swe/git/repos/package/epics/ioc/common/ims.git to origin git@github.com:your-username/ioc-common-ims.git upstream git@github.com:pcdshub/ioc-common-ims.git
ami_offline_psana	usage: ami_offline_psana options We will run ami_offline OPTIONS: -u user (needs to be able to log into the psananeh/feh, if not on psana already) -e EXPNUMBER -R rebinning (binned to 640x640) -n no timetool plugin
archive-status	usage: archive-status [-h] PV Return the status of the specified PV in the archiver. OPTIONS: -h, --help: Show the help message and exit.
camViewer	usage: /reg/g/pcds/engineering_tools/latest/scripts/camViewer options start the viewer for controls cameras OPTIONS: -c|--cam camera name as in camera list or gige # -m|--main bring up the edm screen -r|--reboot reboot the IOC -l|--list print list of cameras -w|--wait # (wait for # hours to ask to renew, default 2 and 12 for GIGEs) -u|--update # update rate limit (default 5) -H|--hutch: use a specific hutches camviewer config file -e|--enable disable camera ioc -d|--disable disable camera ioc -a|--acquire start acquiring images -s|--stop stop acquiring images -n|--cycle cycles acquisition (first stops then starts)
check_host	Usage: /reg/g/pcds/engineering_tools/latest/scripts/check_host HOSTNAME Display host info and run some checks.
configdb_readxtc	usage: configdb_readxtc options We will run configdb_readxtc OPTIONS: -u user (needs to be able to log into the psananeh/feh) -e expnumber
daq_control	daq_control COMMAND TARGET COMMAND : { start, stop, restart, status } TARGET : { daq, ami } COMMAND : ami TARGET : { [0], 1 }
daq_waitwin	Waits for the LCLS-I daq windows to load, then exits.
detector_totals.py	Generates a report for the detector group. Reports contains the number of events per detector type gathered from all experiments in a run period. For example, detector_totals.py --run_period 20 generates the detector total report for run 20
dev_conda	Source this to activate a pcds conda environment. By default, this activates the latest environment. Use export PCDS_CONDA_VER=VERSION before running to pick a different env. Pick up EPICS environment variable settings just in case user did not
eloggrabber	usage: eloggrabber options start the eloggrabber, by default look at current exp OPTIONS: -e pass in an experiment to look at -x instrument logbook -c controls logbook -u username
epicsArchChecker	usage: epicsArchChecker [-h] [-w] [-s] filepath Checks epicsArch files for mismatches of PVs and aliases, missing files, and unconnected PVs. positional arguments: filepath Full filepath of the file to check e.g /reg/g/pcds/dist/pds/xpp/misc/epicsArch.txt optional arguments: -h, --help show this help message and exit -w, --warnings Displays: -Pvs and Aliases duplicated. -Pvs with no alias and aliases no Pvs. -s, --status Displays Pvs not connected.
get_curr_exp	usage: get_curr_exp options OPTIONS: -l add live status -i/H information for hutch (override autodetection)
get_hutch_name	Returns the hutch name based on the host it is run on. See `get_info` for more information.
get_info	usage: get_info [-h] [--run] [--exp] [--live] [--ended] [--hutch HUTCH] [--station STATION] [--getHutch] [--gethutch] [--getstation] [--getbase] [--getinstrument] [--getcnf] [--files_for_run FILES_FOR_RUN] [--nfiles_for_run NFILES_FOR_RUN] [--setExp SETEXP] optional arguments: -h, --help show this help message and exit --run get last run --exp get experiment name --live ongoing? --ended ended --hutch HUTCH get experiment for hutch xxx --station STATION optional station for hutch with two daqs, e.g. cxi and mfx --getHutch get hutch (uppercase) --gethutch get hutch (lowercase) --getstation get hutch station (for multiple daqs) --getbase get base daq name (hutch_station if multiple daqs, otherwise hutch) --getinstrument get instrument (HUTCH_station if multiple daqs, otherwise hutch) --getcnf get cnf file name --files_for_run FILES_FOR_RUN get xtc files for run --nfiles_for_run NFILES_FOR_RUN get xtc files for run --setExp SETEXP set experiment name
get_lastRun	usage: get_lastRun options OPTIONS: -l add live status -i/H information for hutch (override autodetection)
getPVAliases	usage: gatherPVAliases [-h] [-d] patt hutch positional arguments: patt | Regex pattern to match IOCs with. -->Can match anything in the IOC procmanager object. e.g. "lm2k2" or "mcs2" or "ek9000" hutch | 3 letter hutch code. Use "all" to search through all hutches. -->Valid arguments: all, aux, cxi, det, hpl, icl, kfe, las, lfe, mec, mfx, rix, rrl, thz, tmo, tst, txi, ued, xcs, xpp, xrt optional arguments: -h, --help | show this help message and exit -d, --dry_run | Forces a dry run for the script. No files are saved.
grep_ioc	usage: grep_ioc KEYWORD [hutch] hutch can be any of: xpp, xcs, cxi, mfx, mec, xrt, aux, det, fee, hpl, icl, las, lfe, tst, thz, all If no hutch is specified, all hutches will be searched
grep_more_ioc	usage: grep_more_ioc [-h] [-d] patt hutch {print,search} positional arguments: patt Regex str to search through iocmanager.cfg e.g. 'mcs2', 'lm2k2-atm.*', 'ek9000', 'gige.*' hutch 3 letter hutch code to search through. Use 'all' to search through all hutches. Valid arguments: all, aux, cxi, det, hpl, icl, kfe, las, lfe, mec, mfx, rix, rrl, thz, tmo, tst, txi, ued, xcs, xpp, xrt -h, --help Show help message and exit -d, --ignore_disabled Exclude IOCs based on disabled state Necessary subcommands. Use: grep_more_ioc . all [subcommand] --help for more information {print, search} print | Prints all the matching IOCs in a dataframe usage: grep_more_ioc patt hutch print [-h] [-c] [-r] [-s] [-y] -h, --help | Show help message and exit -c, --skip_comments | Prints IOC.cfg file with comments skipped -r, --release | Includes the parent IOC release in the dataframe -s, --print_dirs | Dump child & parent directors to the terminal -y, --print_history | Dump child IOC's history to terminal, if it exists search | Regex-like search of child IOCs usage: grep_more_ioc patt hutch search [-h] [-q] [-o] PATT PATT | The regex str to use in the search -h, --help | Show help message and exit -q, --quiet | Surpresses file warning for paths that do not exist -o, --only_search | Skip printing dataframe, only print search results
grep_pv	GREP SEARCHES THROUGH ALL IOCs IN /reg/d/iocData/ FOR PVs THAT MATCH GIVEN KEYWORD/HANDLE. Ideally, find_pv should be used as it gives a lot more information (but can be slow sometimes)
hdf5_to_gif.py	Converts images in hdf5 files created with h5_img_collect to gifs. Specify the path with -p and and how long each frame should last (ms) with -t. Will save to cwd or a specified directory with -d as {original_filename}.gif
image_saver	Uses h5_img_collect to save images from a camera in an hdf5 format. Command line arguments -c, -n, -p, and -f to specify camera name, number of images, path to save hdf5 file to, and name to save hdf5 file as. Also can use -g switch to open a GUI with a button that when pressed takes images with specified parameters - can be pressed multiple times. The number of images and the label on each button can be changed within the gui. Images from the gui will be converted into gifs and saved into a (new) ~/gifs directory.
ioc-deploy	usage: ioc-deploy [-h] [--version] [--name NAME] [--release RELEASE]   [--ioc-dir IOC_DIR] [--path-override PATH_OVERRIDE]   [--auto-confirm] [--dry-run] [--verbose]   [--github_org GITHUB_ORG]   {update-perms} ...   ioc-deploy is a script for building and deploying ioc tags from github.   It will take one of two different actions: the normal deploy action, or a write permissions change on an existing deployed release.   The normal deploy action will create a shallow clone of your IOC in the standard release area at the correct path and "make" it. If the tag directory already exists, the script will exit.   In the deploy action, after making the IOC, we'll write-protect all files and all directories. We'll also write-protect the top-level directory to help indicate completion.   Note that this means you'll need to restore write permissions if you'd like to rebuild an existing release on a new architecture or remove it entirely.   Example command:   "ioc-deploy -n ioc-foo-bar -r R1.0.0"   This will clone the repository to the default ioc directory and run make using the currently set EPICS environment variables, then apply write protection.   With default settings, this will clone from https://github.com/pcdshub/ioc-foo-bar to /cds/group/pcds/epics/ioc/foo/bar/R1.0.0 then cd and make and chmod as appropriate.   If the repository exists but the tag does not, the script will ask if you'd like to make a new tag and prompt you as appropriate.   The second action will not do any git or make actions, it will only find the release directory and change the file and directory permissions. This can be done with similar commands as above, adding the subparser command, and it can be done by passing the specific path you'd like to modify if this is more convenient for you.   Example commands:   "ioc-deploy update-perms rw -n ioc-foo-bar -r R1.0.0" "ioc-deploy update-perms ro -n ioc-foo-bar -r R1.0.0" "ioc-deploy update-perms rw -p /cds/group/pcds/epics/ioc/foo/bar/R1.0.0" "ioc-deploy update-perms ro -p /cds/group/pcds/epics/ioc/foo/bar/R1.0.0"   positional arguments:   {update-perms} Subcommands (will not deploy):   update-perms Use 'ioc-deploy update-perms' to update the write   permissions of a deployment. See 'ioc-deploy update-   perms --help' for more information.   optional arguments:   -h, --help show this help message and exit   --version Show version number and exit.   --name NAME, -n NAME The name of the repository to deploy. This is a   required argument. If it does not exist on github,   we'll also try prepending with 'ioc-common-'.   --release RELEASE, -r RELEASE   The version of the IOC to deploy. This is a required   argument.   --ioc-dir IOC_DIR, -i IOC_DIR   The directory to deploy IOCs in. This defaults to   $EPICS_SITE_TOP/ioc, or /cds/group/pcds/epics/ioc if   the environment variable is not set. With your current   environment variables, this defaults to   /reg/g/pcds/epics/ioc.   --path-override PATH_OVERRIDE, -p PATH_OVERRIDE   If provided, ignore all normal path-selection rules in   favor of the specific provided path. This will let you   deploy IOCs or apply protection rules to arbitrary   specific paths.   --auto-confirm, --confirm, --yes, -y   Skip the confirmation promps, automatically saying yes   to each one.   --dry-run Do not deploy anything, just print what would have   been done.   --verbose, -v, --debug   Display additional debug information.   --github_org GITHUB_ORG, --org GITHUB_ORG   The github org to deploy IOCs from. This defaults to   $GITHUB_ORG, or pcdshub if the environment variable is   not set. With your current environment variables, this   defaults to pcdshub.   usage: ioc-deploy update-perms [-h] [--name NAME] [--release RELEASE]   [--ioc-dir IOC_DIR]   [--path-override PATH_OVERRIDE]   [--auto-confirm] [--dry-run] [--verbose]   {ro,rw}   Update the write permissions of a deployment. This will make all the files read-only (ro), or owner and group writable (rw).   positional arguments:   {ro,rw} Select whether to make the deployment permissions   read-only (ro) or read-write (rw).   optional arguments:   -h, --help show this help message and exit   --name NAME, -n NAME The name of the repository to deploy. This is a   required argument. If it does not exist on github,   we'll also try prepending with 'ioc-common-'.   --release RELEASE, -r RELEASE   The version of the IOC to deploy. This is a required   argument.   --ioc-dir IOC_DIR, -i IOC_DIR   The directory to deploy IOCs in. This defaults to   $EPICS_SITE_TOP/ioc, or /cds/group/pcds/epics/ioc if   the environment variable is not set. With your current   environment variables, this defaults to   /reg/g/pcds/epics/ioc.   --path-override PATH_OVERRIDE, -p PATH_OVERRIDE   If provided, ignore all normal path-selection rules in   favor of the specific provided path. This will let you   deploy IOCs or apply protection rules to arbitrary   specific paths.   --auto-confirm, --confirm, --yes, -y   Skip the confirmation promps, automatically saying yes   to each one.   --dry-run Do not deploy anything, just print what would have   been done.   --verbose, -v, --debug   Display additional debug information.
iocmanager	iocmanager [hutch] Control status of all IOCs running in a particular hutch in an interactive GUI. Current hutch is used if not provided.
ioctool	usage: ioctool <ioc>|<pv> [option] Script that returns information about an ioc given its name or a PV it hosts default option is 'name', list of options: status : print power status of machine, try to ping interfaces name : returns the name of the ioc dir : returns the path to the directory the ioc is running from cddir :opens the directory the ioc is running from (start with "source" before calling script with this option) cfg : returns the the file name of the ioc .cfg (or st.cmd) less: opens the ioc .cfg (or st.cmd) in less cat : opens the ioc .cfg (or st.cmd) in cat data : returns the path of the appropriate iocData directory if it exists autosave : opens the most recent autosave file in less archive : opens the most recent archive file in less log : opens the most recent log file in less telnet : starts a telnet session with the ioc pvs : opens the IOC.pvlist file in less
ipmConfigEpics	usage: ipmConfigEpics [-b boxname] [-H hutch] [-d] [-h] [-l] -b: specify boxname to view -H: specify a hutch to use, overriding the automated selection -d: fix issues with Bld Damage (likely camera IOC w/plugins on same machine) -h: display this help text -l: list available boxnames
kinit_helper	usage: kinit_helper Defines two functions - kauth and afsauth. kauth will create a new kerberos token, renew an existing one, or do nothing if a valid token exists. afsauth will check that the user and host are able to access afs; if so, and an afs token doesn't already exist, kauth will be called and a new afs token will be made. By default, calls afsauth.
makepeds	usage: makepeds options Make a pedestal file for offline use OPTIONS: -u user (needs to be able to log into the psananeh/feh) -r runnumber for pedestal -e EXPNAME in case you do not want pedestals for the ongoing experiment -J make pedestals for Jungfrau (default only cspad/EPIX detectors) -j make pedestals for Jungfrau - 3 run version(default only cspad/EPIX detectors) -O make pedestals for Opals (default only cspad/EPIX detectors) -Z make pedestals for Zyla (default only cspad/EPIX detectors) -p TEXT: add to elog post -c EVTCODE X use events with eventcode X set -n # : if you have created a noise file, then write pixel mask file for pixels with noise above #sigma -N # : use this number of events (default 1000) -D : dark run for XTCAV -L : lasing off run for XTCAV -v STR: validity range (if not run#-end, e.g. 123-567 or 123-end) -l: do NOT send to batch queue -F : use the FFB (default if no experiment is passed) -g : run on an FFB batch node -f : full epix10k charge injection run -C # : if noise filecreated, write pixel mask file for pixels with noise below xxx (currently integer only...) -m # : write pixel mask file for pixels with pedestal below xxx (currently integer only...) -x # : write pixel mask file for pixels with pedestal above xxx (currently integer only...) -i start calibman. -r 0: show all darks, -r n: show runs (n-25) - 25 -d give path for alternative calibdir
makepeds_psana	usage: makepeds_psana options Make a pedestal file OPTIONS: -r runnumber for pedestal -e EXPNAME in case you do not want pedestals for the ongoing experiment -H HUTCH in case you do not pass an experiment name -Z pedestal for zyla -O make pedestals for Opals -J pedestal for jungfrau (needs first of set of 3 runs!) -D : dark run for XTCAV -L : lasing off run for XTCAV (-b specifies the number of assumed bunches, def 1) -l : donot send to batch queue -F : use FFB -f : full epix10k charge injection run -v STR: validity range (if not run#-end, e.g. 123-567 or 123-end) -N # : use this number of events (default 1000) -n # : if noise filecreated, write pixel mask file for pixels with noise above xxx (currently integer only...) -C # : if noise filecreated, write pixel mask file for pixels with noise below xxx (currently integer only...) -m # : write pixel mask file for pixels with pedestal below xxx (currently integer only...) -x # : write pixel mask file for pixels with pedestal above xxx (currently integer only...) -c EVTCODE X use events with eventcode X set -i start calibman. -r 0: show all darks, -r n: show runs (n-25) - 25 -d give path for alternative calibdir -t : test, do not deploy. -y : when specify cuts for status mask, apply those for epix100.
motor-expert-screen	usage: motor-expert-screen options MOTOR_PV_BASENAME Start an EDM for the specified motor. Attempts to choose the correct type. OPTIONS: -h shows the usage information
motor-typhos	usage: motor-typhos options MOTOR_PV_BASENAME Start a typhos screen for the specified motor. Attempts to choose the correct type. OPTIONS: -h shows the usage information
motorInfo	usage: motorInfo MOTOR_PV (motor_pv_2/autosave/archive/pmgr_diff/pmgr_save) OPT If given two motors, compare their settings If given autosave as second argument, compare the current settings to the autosaved values: differences will be printed If given archive, the archive values will be printed for the last week. If only the base PV is given, extra arguments will be needed OPTIONS: -h shows the usage information -f fields to use as a comma separated list (default: use all autosave values) -s start time for archiver info (YYYY/MM/DD HH:MM:SS) -e end time for archiver info (YYYY/MM/DD HH:MM:SS)
pcds_conda	Source this to activate a pcds conda environment. By default, this activates the latest environment. Use export PCDS_CONDA_VER=VERSION before running to pick a different env. Pick up EPICS environment variable settings just in case user did not
pgpwave8_upgrade	usage $0 -i IOCNAME [-d DEVICE] [-r] [-p FIRMWARE_PATH]<br/> <br/> Read or upgrade the firmware version of a pgpwave8 given its ioc.<br/> <br/> -i The ioc whose pgpwave8's firmware should be upgraded.<br/> -d The path to the device, defaulting to /dev/datadev_0.<br/> -r Print the firmware version only. Cannot be used with -p.<br/> -p The path to the mcs file containing the new firmware.<br/> Cannot be used with -r. Some firmware images can be found<br/> in /cds/group/pcds/package/wave8/images. If -r and -p are not provided,<br/> user will be prompted if they want to use<br/> /cds/group/pcds/package/wave8/images/latest as the new firmware. EOF
pkg_release	Checks out a package from the pcdshub github at a particular tag. Does not update "latest" softlinks, these are inconsistent between packages. Make sure your tag exists before running.
pmgr	pmgr [hutch] [--debug] [--applyenable] --debug : Displays the debug button, which prints out any edits made --applyenable : Displays the apply all button, which applies settings to all motors
pydev_env	Source this file to activate a development environment based on the latest shared environment and on past calls to pydev_register
pydev_register	Use this script to register development packages so that they will be available when you source pydev_env
pyps-deploy	usage: pyps-deploy [-h] -r RELEASE -c CONDA [--repo REPO] [--app-bin APP_BIN] app Sets up a pyps/apps deployment for a particular github python package. This will create an executable under .../pyps/apps/APP-NAME/RELEASE/APP-NAME and repoint the symbolic link at .../pyps/apps/APP-NAME/latest to the new release folder. positional arguments: app Name of the app to deploy optional arguments: -h, --help show this help message and exit -r RELEASE, --release RELEASE App version -c CONDA, --conda CONDA Conda environment name --repo REPO Clone this repo and mask the environment package. Use this when you have only a small change that does not need a full environment release. --app-bin APP_BIN Use in conjunction with --repo arg when the launcher is not in the bin directory
questionnaire_tools	usage: questionnaire_tools [-h] [-f FROMEXP] [-t TOEXP] [-r READEXP] [-c] [-d ADD_DEVICE] [-l] [-p PRINT_DEVICE] [--dev] [--experimentList] [--propList] optional arguments: -h, --help show this help message and exit -f FROMEXP, --fromExp FROMEXP experiment to copy from -t TOEXP, --toExp TOEXP experiment to copy to -r READEXP, --readExp READEXP experiment to read CDS tag from -c, --copy_CDS copy data from CDS tab -d ADD_DEVICE, --add_device ADD_DEVICE name of device to be added -l, --list_devices list device to be added -p PRINT_DEVICE, --print_device PRINT_DEVICE print data for device --dev connect to dev database --experimentList list of experiments --propList list of proposals
restartdaq	usage: restartdaq options OPTIONS: -w sort windows after start -p select partition (same as used last) -s silent (do not email jana) -C [cnf] : (FOR LCLS2 Hutches) Specify a CNF to run. E.g. qrix.py or crix.py
serverStat	usage: serverStat servername options Script to check status of servers & reboot/power cycle them SIGNATURE: serverStat SERVERNAME [command] default command is 'status', list of commands: status : print power status of machine, try to ping interfaces on : power machine on off : power machine off cycle : power cycle machine, wait a few second in off state reset : reset machine (ideally try that before power cycling) console: open the ipmi console where possible expert : display info and run checks on server
ssh-agent-helper	usage: source ssh-agent-helper Helper script for starting the ssh agent if needed and doing an ssh-add -t 12h. This will let anyone smoothly run github/ssh related scripts without multiple password prompts. An ssh-agent process started by using this script will be automatically closed on logout. This script is intended to be sourced. Sourcing this script lets ssh-agent set the proper environment variables it needs to run properly.
startami	usage: startami options we are starting another ami session here This script restarts AMI1 in LCLS1 hutches and AMI2 in LCLS2 hutches OPTIONS: -s: stop the ami client current running on this machine -c: config file you'd like to use (i.e. cxi_test.cnf)
stopami	Kill an AMI process running in the current hutch. This script stops AMI1 in LCLS1 hutches and AMI2 in LCLS2 hutches
stopdaq	Stop the daq in the current hutch.
takepeds	usage: takepeds Takes a run with dark images for use in pedestals, and posts to the elog.
verify-hutch	usage: verify-hutch hutch Verifies that the passed argument is a known hutch, exit 0 for success and exit 1 for failure.
wheredaq	Discover what host is running the daq in the current hutch, if any.
wherepsana	Usage: where_psana [-h] [-c CONFIG] [-d DETAIL] Checks where we have shared memory servers for psana running and could run psana jobs. Optional arguments: -h Show usage -c Pick a specific DAQ config file rather than automatically selecting current hutch's file -d Also show information about dss node mapping
set_gem_timing	Usage: set_gem_timing [SC or NC]