Code

Add CI cron jobs#1476

Code

Merged

WalterKolczynski-NOAA merged 125 commits intoNOAA-EMC:developNOAA-EMC/global-workflow:developfrom TerrenceMcGuinness-NOAA:feature_1428_CI-cronjobsTerrenceMcGuinness-NOAA/global-workflow:feature_1428_CI-cronjobsCopy head branch name to clipboardApr 21, 2023

+387-179

Commits

Commits on Mar 28, 2023

• disambiguated env vars for HOMEgfs and GFS_CI_ROOT and removed re-assigned and redundent variables

  

  TerrenceMcGuinness-NOAA

  committed
  a2faebc

  Copy full SHA for a2faebc

• debugged updates to scripts (typos ect)

  

  Terry.McGuinness

  authored and

  Terry.McGuinness

  committed
  37b0c15

  Copy full SHA for 37b0c15

• Update create_experiment.py

  TerrenceMcGuinness-NOAA

  authored
  37cfedb

  Copy full SHA for 37cfedb

• Update driver.sh

  TerrenceMcGuinness-NOAA

  authored
  349a383

  Copy full SHA for 349a383

• Update clone-build_ci.sh

  TerrenceMcGuinness-NOAA

  authored
  f111c47

  Copy full SHA for f111c47

• added another layer in file structure for experments where each experment has a dir that contains COMROT and config

  

  Terry.McGuinness

  authored and

  Terry.McGuinness

  committed
  619bdce

  Copy full SHA for 619bdce

Commits on Mar 29, 2023

• Merge branch 'NOAA-EMC:develop' into feature_1428_CI-cronjobs

  TerrenceMcGuinness-NOAA

  authored
  28a509b

  Copy full SHA for 28a509b

• fixed bugs with tree restructure under ../PR/{PR#}/RUNTESTS

  

  Terry.McGuinness

  authored and

  Terry.McGuinness

  committed
  efd04aa

  Copy full SHA for efd04aa

• spelling errors in create experiment

  

  TerrenceMcGuinness-NOAA

  committed
  b7df0e7

  Copy full SHA for b7df0e7

• added run_ci.sh that is used in cron to run expirments

  

  Terry.McGuinness

  authored and

  Terry.McGuinness

  committed
  e2c211d

  Copy full SHA for e2c211d

Commits on Mar 30, 2023

• removed --login from shabang in bash scripts

  

  Terry.McGuinness

  authored and

  Terry.McGuinness

  committed
  07c81e7

  Copy full SHA for 07c81e7

• added environment for orion

  

  Terry Mcguinness

  authored and

  Terry Mcguinness

  committed
  7cca618

  Copy full SHA for 7cca618

• fixed bug with needing updated git from module load on orion

  

  Terry McGuinness

  authored and

  Terry McGuinness

  committed
  b0fcb3d

  Copy full SHA for b0fcb3d

Commits on Mar 31, 2023

• added correct ICDIR to orion and module load module_ci for uniformally getting rocoto for run_ci.sh

  

  Terry McGuinness

  committed
  801bbf3

  Copy full SHA for 801bbf3

• Merge branch 'NOAA-EMC:develop' into feature_1428_CI-cronjobs

  TerrenceMcGuinness-NOAA

  authored
  e76d66a

  Copy full SHA for e76d66a

Commits on Apr 4, 2023

• Merge branch 'NOAA-EMC:develop' into feature_1428_CI-cronjobs

  TerrenceMcGuinness-NOAA

  authored
  def8455

  Copy full SHA for def8455

• added old python compair GFS folders for development

  

  TerrenceMcGuinness-NOAA

  committed
  b067ed6

  Copy full SHA for b067ed6

Commits on Apr 5, 2023

• added Execute class for nccnp to comparitor

  

  TerrenceMcGuinness-NOAA

  committed
  7343b27

  Copy full SHA for 7343b27

• udpate logging level and fixed spelling errors

  

  TerrenceMcGuinness-NOAA

  committed
  393d570

  Copy full SHA for 393d570

• added not none over len 0 for support Null return on Execute on nccmp

  

  TerrenceMcGuinness-NOAA

  committed
  c2cb34d

  Copy full SHA for c2cb34d

• cleaned up and and made outout human readable for the comaritor code

  

  Terry.McGuinness

  authored and

  Terry.McGuinness

  committed
  b7e51db

  Copy full SHA for b7e51db

Commits on Apr 6, 2023

• update GitHub labels to all start with CI

  

  Terry McGuinness

  committed
  96d8498

  Copy full SHA for 96d8498

• format output dropped f and added nccmp to orion env modufile

  

  Terry McGuinness

  committed
  b905ef3

  Copy full SHA for b905ef3

Commits on Apr 7, 2023

• added grib file checker with corrlation checks in compare folders script

  

  Terry McGuinness

  committed
  4abb87f

  Copy full SHA for 4abb87f

• got wgrib2 checker to work

  

  Terry McGuinness

  committed
  1d75a06

  Copy full SHA for 1d75a06

• added wgrib2 to ci module file for Hera

  

  Terry.McGuinness

  authored and

  Terry.McGuinness

  committed
  2eee604

  Copy full SHA for 2eee604

• had to add hpc 1.1.0 to module file to get wgrib2

  

  Terry.McGuinness

  authored and

  Terry.McGuinness

  committed
  72a4bfd

  Copy full SHA for 72a4bfd

• fixed module load order for virtutal python environment

  

  Terry.McGuinness

  authored and

  Terry.McGuinness

  committed
  a3324a6

  Copy full SHA for a3324a6

Commits on Apr 10, 2023

• started code cleanup, removed job switches for storing a yaml file of all files

  

  TerrenceMcGuinness-NOAA

  committed
  88a9185

  Copy full SHA for 88a9185

• removed import collections

  

  Terry.McGuinness

  authored and

  Terry.McGuinness

  committed
  4fa9e3a

  Copy full SHA for 4fa9e3a

• test git edit

  TerrenceMcGuinness-NOAA

  authored
  888a400

  Copy full SHA for 888a400

• pcode edits

  TerrenceMcGuinness-NOAA

  authored
  3ecca64

  Copy full SHA for 3ecca64

• more pycode edits

  TerrenceMcGuinness-NOAA

  authored
  3882cac

  Copy full SHA for 3882cac

• more pycode edits

  TerrenceMcGuinness-NOAA

  authored
  227dad8

  Copy full SHA for 227dad8

• more pycode edits

  TerrenceMcGuinness-NOAA

  authored
  a26d4af

  Copy full SHA for a26d4af

• more pycode edits

  TerrenceMcGuinness-NOAA

  authored
  5f3535f

  Copy full SHA for 5f3535f

• more pycode edits

  TerrenceMcGuinness-NOAA

  authored
  bf3f5cc

  Copy full SHA for bf3f5cc

• few more pycode edits

  TerrenceMcGuinness-NOAA

  authored
  2f9b0a5

  Copy full SHA for 2f9b0a5

• pycode edits

  TerrenceMcGuinness-NOAA

  authored
  2f72f48

  Copy full SHA for 2f72f48

• two more pycode edits

  TerrenceMcGuinness-NOAA

  authored
  dd81b6e

  Copy full SHA for dd81b6e

• bash lint edit

  TerrenceMcGuinness-NOAA

  authored
  2773f09

  Copy full SHA for 2773f09

• fixed some formatting issues from pycode updates

  

  Terry.McGuinness

  authored and

  Terry.McGuinness

  committed
  44226f2

  Copy full SHA for 44226f2

• some more cleaning up format

  

  Terry.McGuinness

  authored and

  Terry.McGuinness

  committed
  e127ed2

  Copy full SHA for e127ed2

Commits on Apr 11, 2023

• moved module load in driver

  

  Terry McGuinness

  committed
  ce362ea

  Copy full SHA for ce362ea

• added ignore list for wild cards

  

  Terry McGuinness

  committed
  eb7471b

  Copy full SHA for eb7471b

• removed module load in driver and added rm pr_list at the end if used in cron

  

  Terry McGuinness

  committed
  ec71918

  Copy full SHA for ec71918

Commits on Apr 12, 2023

• fix syntax error with echo at end of clone-build

  

  Terry McGuinness

  committed
  1bc444f

  Copy full SHA for 1bc444f

• need to remove set -e from driver and build scripts to that CI system can report back to GitHub the fails and continue their work on other builds

  

  Terry.McGuinness

  authored and

  Terry.McGuinness

  committed
  2ee5051

  Copy full SHA for 2ee5051

• added checks and output results for checkout in clone-build script

  

  Terry McGuinness

  committed
  522aa27

  Copy full SHA for 522aa27

• moved message sent to GitHub on the status of each PR on pass or fail to end of loop to pick up all cases

  

  Terry.McGuinness

  authored and

  Terry.McGuinness

  committed
  59c3226

  Copy full SHA for 59c3226

• added two seperate module loads for workflow and CI, fixed some messaging

  

  Terry.McGuinness

  authored and

  Terry.McGuinness

  committed
  06f902e

  Copy full SHA for 06f902e

• added gwci and gwsetup module files for orion

  

  Terry.McGuinness

  authored and

  Terry.McGuinness

  committed
  aadb132

  Copy full SHA for aadb132

• update machine gwsetup module load in run_ci script

  

  Terry McGuinness

  committed
  909a31b

  Copy full SHA for 909a31b

Commits on Apr 14, 2023

• Merge branch 'NOAA-EMC:develop' into feature_1428_CI-cronjobs

  TerrenceMcGuinness-NOAA

  authored
  cb0ed72

  Copy full SHA for cb0ed72

Commits on Apr 17, 2023

• Merge branch 'NOAA-EMC:develop' into feature_1428_CI-cronjobs

  TerrenceMcGuinness-NOAA

  authored
  7333130

  Copy full SHA for 7333130

• udpated gwci on orion for threaded version of nccmp, updated reporting on grib files in compare, small bug in test/diff_grib with arg array

  

  Terry McGuinness

  committed
  adf6809

  Copy full SHA for adf6809

• Merge branch 'feature_1428_CI-cronjobs' of github.com:TerrenceMcGuinness-NOAA/global-workflow into feature_1428_CI-cronjobs

  

  Terry McGuinness

  committed
  1da08ab

  Copy full SHA for 1da08ab

• cleaned up path reporting in compare folders and fixed dependacies in gwci module file for Hera

  

  Terry.McGuinness

  authored and

  Terry.McGuinness

  committed
  8486d1d

  Copy full SHA for 8486d1d

• updated verbose output and added ignore list in file count conformity (had extra logs from multiple runs)

  

  Terry McGuinness

  committed
  0a83799

  Copy full SHA for 0a83799

• few pythoncode lint updates

  

  Terry McGuinness

  committed
  8698f71

  Copy full SHA for 8698f71

• more pycode linting (spaces and commas)

  

  Terry McGuinness

  committed
  ee50f86

  Copy full SHA for ee50f86

• Merge branch 'NOAA-EMC:develop' into feature_1428_CI-cronjobs

  TerrenceMcGuinness-NOAA

  authored
  b0ad92c

  Copy full SHA for b0ad92c

• added || exit on cd (change directory) in clone-build BASH script

  

  Terry McGuinness

  committed
  535721f

  Copy full SHA for 535721f

• Merge branch 'feature_1428_CI-cronjobs' of github.com:TerrenceMcGuinness-NOAA/global-workflow into feature_1428_CI-cronjobs

  

  Terry McGuinness

  committed
  aae6072

  Copy full SHA for aae6072

• indentation for visual effect PIP8

  

  Terry McGuinness

  committed
  2fc7b44

  Copy full SHA for 2fc7b44

• indentation for visual effect PIP8 E127, again

  

  Terry McGuinness

  committed
  126d435

  Copy full SHA for 126d435

• PIP8 E127 wanted alignment after continuation line to be a regular indent

  

  TerrenceMcGuinness-NOAA

  committed
  74415b3

  Copy full SHA for 74415b3

• PEP8 black line had spaces

  

  TerrenceMcGuinness-NOAA

  committed
  a71a916

  Copy full SHA for a71a916

• type spelling error of directories

  

  TerrenceMcGuinness-NOAA

  committed
  a3f05bb

  Copy full SHA for a3f05bb

Commits on Apr 18, 2023

• Merge branch 'NOAA-EMC:develop' into feature_1428_CI-cronjobs

  TerrenceMcGuinness-NOAA

  authored
  6d421e4

  Copy full SHA for 6d421e4

• Update ci/scripts/clone-build_ci.sh

  Show description for c0287ba
  

  TerrenceMcGuinness-NOAA

  and

  aerorahul

  authored
  c0287ba

  Copy full SHA for c0287ba

• Update ci/scripts/run_ci.sh

  Show description for 8009fb2
  

  TerrenceMcGuinness-NOAA

  and

  aerorahul

  authored
  8009fb2

  Copy full SHA for 8009fb2

• Update ci/experiments/C96_atm3DVar.yaml

  Show description for 94f8c8a
  

  TerrenceMcGuinness-NOAA

  and

  aerorahul

  authored
  94f8c8a

  Copy full SHA for 94f8c8a

• Update ci/scripts/clone-build_ci.sh

  Show description for e6ffad4
  

  TerrenceMcGuinness-NOAA

  and

  aerorahul

  authored
  e6ffad4

  Copy full SHA for e6ffad4

• Update ci/scripts/run_ci.sh

  Show description for 5080370
  

  TerrenceMcGuinness-NOAA

  and

  aerorahul

  authored
  5080370

  Copy full SHA for 5080370

• Apply suggestions from code review

  Show description for 0b8f38d
  

  TerrenceMcGuinness-NOAA

  and

  aerorahul

  authored
  0b8f38d

  Copy full SHA for 0b8f38d

• Update ci/scripts/clone-build_ci.sh

  Show description for 3e4e921
  

  TerrenceMcGuinness-NOAA

  and

  aerorahul

  authored
  3e4e921

  Copy full SHA for 3e4e921

• Update ci/scripts/driver.sh

  Show description for 22a9073
  

  TerrenceMcGuinness-NOAA

  and

  aerorahul

  authored
  22a9073

  Copy full SHA for 22a9073

• Update ci/scripts/run_ci.sh

  Show description for 8f04d79
  

  TerrenceMcGuinness-NOAA

  and

  aerorahul

  authored
  8f04d79

  Copy full SHA for 8f04d79

• Update ci/scripts/run_ci.sh

  Show description for 6889ee9
  

  TerrenceMcGuinness-NOAA

  and

  aerorahul

  authored
  6889ee9

  Copy full SHA for 6889ee9

• Update ci/scripts/run_ci.sh

  Show description for 856bb2b
  

  TerrenceMcGuinness-NOAA

  and

  aerorahul

  authored
  856bb2b

  Copy full SHA for 856bb2b

• added module-setup.sh in place of module purge

  

  Terry McGuinness

  committed
  bb34e9c

  Copy full SHA for bb34e9c

• couple more suggestions, fix build_status off of link and more build settings next to the build script

  

  Terry McGuinness

  committed
  f990c9c

  Copy full SHA for f990c9c

• just moved an export to where it is more approprate

  

  Terry McGuinness

  committed
  86d6104

  Copy full SHA for 86d6104

• improved symantics by replacing environments with platform and experments with cases

  

  Terry McGuinness

  committed
  530882a

  Copy full SHA for 530882a

• Update diff_grib_files.py

  TerrenceMcGuinness-NOAA

  authored
  969877b

  Copy full SHA for 969877b

• Delete C96_atm3DVar_also.yaml

  TerrenceMcGuinness-NOAA

  authored
  dffadd3

  Copy full SHA for dffadd3

• added second case C96C48-hyb-ATM-DA

  

  TerrenceMcGuinness-NOAA

  committed
  aa21ef7

  Copy full SHA for aa21ef7

• Update run_ci.sh

  Show description for e68fa99
  TerrenceMcGuinness-NOAA

  authored
  e68fa99

  Copy full SHA for e68fa99

Commits on Apr 19, 2023

• fixed fail logic in driver for setupexpt and bug with path in yaml for second case

  

  Terry McGuinness

  committed
  51a55d4

  Copy full SHA for 51a55d4

• Merge branch 'feature_1428_CI-cronjobs' of github.com:TerrenceMcGuinness-NOAA/global-workflow into feature_1428_CI-cronjobs

  

  Terry McGuinness

  committed
  5818dcb

  Copy full SHA for 5818dcb

• Merge branch 'NOAA-EMC:develop' into feature_1428_CI-cronjobs

  TerrenceMcGuinness-NOAA

  authored
  1f27cd9

  Copy full SHA for 1f27cd9

• fixed updated path to EXPDIR for staged test cases in run_ci script

  

  Terry McGuinness

  committed
  e9036cf

  Copy full SHA for e9036cf

• Added additional script to check to see if experments have completed succesfully

  

  Terry McGuinness

  committed
  c0ad448

  Copy full SHA for c0ad448

• added new label stage to include CI--Building before Running

  

  Terry McGuinness

  committed
  8afa313

  Copy full SHA for 8afa313

• moved check for open PR after it is cloned

  

  Terry.McGuinness

  authored and

  Terry.McGuinness

  committed
  5bec286

  Copy full SHA for 5bec286

• updated old repo_url varible name in check script

  

  Terry.McGuinness

  authored and

  Terry.McGuinness

  committed
  16a82f6

  Copy full SHA for 16a82f6

• added comment to GitHub when case completes

  

  Terry.McGuinness

  authored and

  Terry.McGuinness

  committed
  51700d4

  Copy full SHA for 51700d4

• added cool SHELLCHECK suggestiongs (amazing tool)

  

  Terry McGuinness

  committed
  482fc31

  Copy full SHA for 482fc31

• Merge branch 'feature_1428_CI-cronjobs' of github.com:TerrenceMcGuinness-NOAA/global-workflow into feature_1428_CI-cronjobs

  

  Terry McGuinness

  committed
  c1d677f

  Copy full SHA for c1d677f

• Update check_ci.sh

  TerrenceMcGuinness-NOAA

  authored
  e850d6d

  Copy full SHA for e850d6d

• Update check_ci.sh

  TerrenceMcGuinness-NOAA

  authored
  32396e7

  Copy full SHA for 32396e7

• added a could of shellcheck suggestions for || true

  

  Terry McGuinness

  committed
  54e9ebe

  Copy full SHA for 54e9ebe

• Merge branch 'feature_1428_CI-cronjobs' of github.com:TerrenceMcGuinness-NOAA/global-workflow into feature_1428_CI-cronjobs

  

  Terry McGuinness

  committed
  3d0c07b

  Copy full SHA for 3d0c07b

• Update module_gwci.orion.lua

  TerrenceMcGuinness-NOAA

  authored
  a51661c

  Copy full SHA for a51661c

• Update modulefiles/module_gwci.orion.lua

  Show description for 9865a74
  

  TerrenceMcGuinness-NOAA

  and

  WalterKolczynski-NOAA

  authored
  9865a74

  Copy full SHA for 9865a74

• Update modulefiles/module_gwci.orion.lua

  Show description for 3404bff
  

  TerrenceMcGuinness-NOAA

  and

  WalterKolczynski-NOAA

  authored
  3404bff

  Copy full SHA for 3404bff

• Update modulefiles/module_gwci.hera.lua

  Show description for 3e7a9af
  

  TerrenceMcGuinness-NOAA

  and

  WalterKolczynski-NOAA

  authored
  3e7a9af

  Copy full SHA for 3e7a9af

• Merge branch 'NOAA-EMC:develop' into feature_1428_CI-cronjobs

  TerrenceMcGuinness-NOAA

  authored
  2d36572

  Copy full SHA for 2d36572

Commits on Apr 20, 2023

• added set -e at top of BASH scripts and then only disabled it when we do not want to break out of checks

  

  Terry.McGuinness

  authored and

  Terry.McGuinness

  committed
  9851300

  Copy full SHA for 9851300

• fixed login of label update in check_ci script and added parallel version of NetCDF for threaded nccmp

  

  Terry McGuinness

  committed
  cdc57ce

  Copy full SHA for cdc57ce

• improved login for success of PR with check_ci script

  

  Terry McGuinness

  committed
  07ddb49

  Copy full SHA for 07ddb49

• forgot to add removing of PR from list on successfully completing all cases

  

  Terry McGuinness

  committed
  737483f

  Copy full SHA for 737483f

• forgot to add removing of PR from list on successfully completing all cases

  

  Terry McGuinness

  committed
  8c6d13d

  Copy full SHA for 8c6d13d

• fixed bug with counting remaining cases

  

  Terry McGuinness

  committed
  a64d1ab

  Copy full SHA for a64d1ab

• added || true to BASH command for shellnorms in check ci script

  

  Terry McGuinness

  committed
  0423bb9

  Copy full SHA for 0423bb9

• fixed race condition for check_ci script thinking success before experments are created (checks for RUNTESTS dir)

  

  Terry McGuinness

  committed
  e6f4550

  Copy full SHA for e6f4550

• Upates to final success check with multiple cases in check_ci ie num_cases=0

  

  Terry McGuinness

  committed
  daebe97

  Copy full SHA for daebe97

• Update check_ci.sh

  TerrenceMcGuinness-NOAA

  authored
  4b7a73a

  Copy full SHA for 4b7a73a

Commits on Apr 21, 2023

• Merge branch 'NOAA-EMC:develop' into feature_1428_CI-cronjobs

  TerrenceMcGuinness-NOAA

  authored
  458c5ea

  Copy full SHA for 458c5ea

• Update clone-build_ci.sh

  TerrenceMcGuinness-NOAA

  authored
  dc87c92

  Copy full SHA for dc87c92

• couple of bug updates for check script when deleting successfull PRs, fixed issue with nccmp linking on Orion

  

  Terry McGuinness

  committed
  0779ff6

  Copy full SHA for 0779ff6

• Merge branch 'feature_1428_CI-cronjobs' of github.com:TerrenceMcGuinness-NOAA/global-workflow into feature_1428_CI-cronjobs

  

  Terry McGuinness

  committed
  0140a17

  Copy full SHA for 0140a17

• Update check_ci.sh

  Show description for ff73987
  TerrenceMcGuinness-NOAA

  authored
  ff73987

  Copy full SHA for ff73987

• updated nccmp 1.8.7.0 and fixed a shellnorm typo to close quote

  

  Terry McGuinness

  committed
  c183221

  Copy full SHA for c183221