Code

Feature #2283 include times#2345

Code

Merged

georgemccabe merged 94 commits intodevelopdtcenter/METplus:developfrom feature_2283_include_timesdtcenter/METplus:feature_2283_include_timesCopy head branch name to clipboardSep 14, 2023

+1,114-142

Commits

Commits on Aug 8, 2023

• added helper function to clean up logic to add new dictionaries that contain time and file information to a list

  georgemccabe

  committed
  3c74be4

• fix incorrect variable name

  georgemccabe

  committed
  60662e4

• log error if wrapper was not loaded properly to help debug issue

  georgemccabe

  committed
  927d631

• improve logging to print input dir/template once at beginning to avoid cluttering log output, make wrapper a LoopTimes wrapper to remove run_at_time function and reduce redundancy, log if example f…

  Show description for ffe1f23
  georgemccabe

  committed
  ffe1f23

• make wrappers LoopTimes (RuntimeFreq using run all times) to remove redundant run_at_time functions in each wrapper, ci-run-all-diff

  georgemccabe

  committed
  fee61a2

Commits on Aug 9, 2023

• fix function name in test, ci-run-all-diff

  georgemccabe

  committed
  89a34d8

• set self.app_name if unset for all wrappers to prevent errors when creating unit tests

  georgemccabe

  committed
  a3f6c70

• set self.app_name if unset for all wrappers to prevent errors when creating unit tests

  georgemccabe

  committed
  5673a61

• change all CompareGridded (GridStat, PointStat, EnsembleStat, MODE, MTD) and ReformatGridded (PCPCombine, RegridDataPlane) wrappers to be LoopTimes wrappers so they use the same run_at_time method

  georgemccabe

  committed
  d82c5e8

• add test for usage wrapper, ci-run-all-diff

  georgemccabe

  committed
  39e8582

• fix call to RegridDataPlane from ExtractTiles wrapper

  georgemccabe

  committed
  6dbc6aa

• changed more wrappers to use consistent time looping, ci-run-all-cases

  georgemccabe

  committed
  82f04e4

• formatting

  georgemccabe

  committed
  469bbe5

• more formatting

  georgemccabe

  committed
  83e0286

• remove unnessary check for skip times because it is called upstream, remove command check that is done inside build function

  georgemccabe

  committed
  9cb8b7e

• add SKIP_LEAD_SEQ check to RuntimeFreq run_at_time and change TCPairs, TCStat, and TCRMW to be RuntimeFreq wrappers -- testing if any issues arise because there may be an assumption that init==vali…

  Show description for da19788
  georgemccabe

  committed
  da19788

Commits on Aug 10, 2023

• run each time input through ti_calculate before passing it to run_at_time_once to compute missing time values, ci-run-all-diff

  georgemccabe

  committed
  f19aaf5

Commits on Aug 11, 2023

• move runtime log banner

  georgemccabe

  committed
  e0a68d2

• rename function to avoid confusion with other find_input_files functions

  georgemccabe

  committed
  21f8f27

Commits on Aug 14, 2023

• refactored logic in ti_calculate and added unit tests for common inputs to function

  georgemccabe

  committed
  ba81987

• formatting cleanup

  georgemccabe

  committed
  3bc665a

• moved logic to compute formatted init or valid so it is still run even if init/valid are not computed

  georgemccabe

  committed
  92d5d26

• pass data type to find_data and change template c_dict key if data type is FCST or OBS

  georgemccabe

  committed
  2c60cd4

• refactor MTD wrapper to use RuntimeFreq methods to find files to process, update unit tests to match changes to calls to wrapper and naming of file list files

  georgemccabe

  committed
  77987c2

• use f-strings for command, ci-run-all-diff

  georgemccabe

  committed
  a1ba59b

• added unit test for case that happens in gfdl tracker wrapper

  georgemccabe

  committed
  c9610e0

• formatting

  georgemccabe

  committed
  95a245f

• fixed failures when trying to format times that are wildcards, fix checks for lead not wildcard (relativedelta comparison operator to string '*' always reports False)

  georgemccabe

  committed
  55dbcd8

• set default runtime freq for TCPairs wrapper

  georgemccabe

  committed
  c60b3eb

• fix use case to use init instead of valid, ci-run-all-diff

  georgemccabe

  committed
  9bac908

Commits on Aug 15, 2023

• Merge branch 'develop' into feature_2283_include_times

  georgemccabe

  committed
  460851e

• preserve old behavior by using init and first lead to compute valid time used to name output directories and file_list file names, change tests back to original values

  georgemccabe

  committed
  9b7ee25

• if [INIT/VALID]_BEG == [INIT/VALID]_END and runtime freq is RUN_ONCE, set init/valid to the time listed to allow TCPairs use cases to run with time info needed to preserve behavior

  georgemccabe

  committed
  c133bb4

• fix bug introduced by using RuntimeFreq that causes duplicate regex group

  georgemccabe

  committed
  70fa3cb

• fix config to use same init beg/end for RUN_ONCE mode

  georgemccabe

  committed
  9a433e2

• use RuntimeFreq time looping, preserve old behavior by setting runtime freq variables based on deprecated variables and warn that they are deprecated

  georgemccabe

  committed
  4c903c6

• turn on use case groups that were failing to test , ci-run-diff

  georgemccabe

  committed
  95eac30

• add missing import, ci-run-diff

  georgemccabe

  committed
  334f921

• name file_list file with 'single' if MTD_SINGLE_RUN to prevent diffs in use case output

  georgemccabe

  committed
  9e1eeab

• remove unused variable

  georgemccabe

  committed
  0d53a0c

• call self.clear before running in RUN_ONCE mode to clear out command line arguments for multiple runs via CUSTOM_LOOP_LIST

  georgemccabe

  committed
  101f5d0

• change TCGen wrapper to RuntimeFreq

  georgemccabe

  committed
  1e4a953

• set lead_hours/minutes/seconds even if lead is months or years so relativedelta lead is still accurate if init/valid change but lead values can still be set from total seconds between init and vali…

  Show description for df2d18a
  georgemccabe

  committed
  df2d18a

• update use case config to use single run time (RUN_ONCE) and set init_beg and init_end explicitly in TCPairs and TCStat MET config

  georgemccabe

  committed
  9a1a610

• turn off use cases that now pass and test if other cases will pass, ci-run-diff

  georgemccabe

  committed
  569c6cf

• turn off all use cases

  georgemccabe

  committed
  50fbe5a

• Create class variables for RuntimeFreq wrappers to define default value to use for runtime frequency and list of supported frequencies (or 'ALL' if any are supported). Added function to validate RU…

  Show description for 4383869
  georgemccabe

  committed
  4383869

• error and return if start and end times cannot be read, e.g. if end time comes before start time

  georgemccabe

  committed
  2bcda33

• update config files to use current config options, ci-run-all-diff

  georgemccabe

  committed
  8f308ff

• fixed bug when beg/end times are not set and aren't needed, ci-run-all-diff

  georgemccabe

  committed
  74e52bd

Commits on Aug 16, 2023

• fix use case reference to init time

  georgemccabe

  committed
  b9eef8f

• update contrib guide

  georgemccabe

  committed
  75da7ab

• clean up contrib guide additions

  georgemccabe

  committed
  407aba4

• change supported runtime frequency values for wrappers that need enhancements to actually support running different frequencies

  georgemccabe

  committed
  cef425d

• fix job args to remove hours from -init_beg and -init_end

  georgemccabe

  committed
  4ed89d6

• per #2283, add support for INCLUDE_TIMES (opposite of SKIP_TIMES) and support for filtering times using day of the week %a and %A

  georgemccabe

  committed
  fea1c43

• Merge branch 'feature_2283_time_looping' into feature_2283_include_times

  georgemccabe

  committed
  9739279

• move timing log output of group so it can be easily viewed without opening each run and scrolling to the bottom of all of the log output

  georgemccabe

  committed
  c5a538e

• per #2283, add documentation about INCLUDE_TIMES and using days of the week for [SKIP/INCLUDE]_TIMES

  georgemccabe

  committed
  6ef7f6d

Commits on Aug 29, 2023

• create URL to easily navigate to commit that triggered workflow run

  georgemccabe

  committed
  58fcc2f

Commits on Sep 5, 2023

• turn on a use case that is failing due to this issue

  georgemccabe

  committed
  a90e31b

• updated version of Docker base image used for all conda env images to use Debian 12 and an updated version of Miniconda

  georgemccabe

  committed
  5dc6222

• added instructions to rebuild conda.vX.Y images as needed

  georgemccabe

  committed
  ab32aca

Commits on Sep 6, 2023

• change version extension of new docker images for conda environments to v6.0

  georgemccabe

  committed
  21c31c1

• try building geovista via conda to see if required functionality is available in latest release

  georgemccabe

  committed
  6a1ec46

• per #2283, implement skip and include times for init and valid

  georgemccabe

  committed
  a62feab

• merged develop and resolved conflicts

  georgemccabe

  committed
  66f69c5

• per #2283, add unit tests to ensure skip/include init/valid times works as expected

  georgemccabe

  committed
  cb33318

• add other packages that were missing

  georgemccabe

  committed
  9b18228

Commits on Sep 8, 2023

• add files to create conda envs via GHA workflow

  georgemccabe

  committed
  6491808

• use job ID to set env var used to name conda environment -- note cannot use github.job in if statement or contains command for some reason

  georgemccabe

  committed
  9e6637b

• whitespace

  georgemccabe

  committed
  acbebce

• undo whitespace change to, ci-run-all-diff

  georgemccabe

  committed
  07d7278

• renamed function that was changed in geovista between v0.1.4 and v0.2.0

  georgemccabe

  committed
  0a10dff

Commits on Sep 11, 2023

• remove specific versions from packages

  georgemccabe

  committed
  25dd1f1

• remove versions from packages

  georgemccabe

  committed
  95489f8

• removed unused environment

  georgemccabe

  committed
  205fb88

• update use case to replace use of deprecated pandas DataFrame.append function with pandas.concat

  georgemccabe

  committed
  6c85d5f

Commits on Sep 12, 2023

• update geovista env to install libGL and libEGL to prevent dynamic library errors

  georgemccabe

  committed
  273287f

• update steps to install Java OpenJDK for GempakToCF.jar

  georgemccabe

  committed
  7af0704

• copy files from new java JDK location

  georgemccabe

  committed
  7d1cdf6

• run use case that is failing due to gempak install issues

  georgemccabe

  committed
  f90c3e4

• copy additional java files needed to run

  georgemccabe

  committed
  061317d

• turn off use case now that it passes

  georgemccabe

  committed
  2803c92

• added version numbers for python packages to ensure the env is reproducible

  georgemccabe

  committed
  2a64e07

• Update internal/scripts/docker_env/Dockerfile.conda

  Show description for 56262e2
  

  georgemccabe

  and

  JohnHalleyGotway

  authored
  56262e2

• fixed incorrect info in input description

  georgemccabe

  committed
  5a09634

• Merge branch 'feature_2338_debian12_conda_envs' of github.com:dtcenter/METplus into feature_2338_debian12_conda_envs

  georgemccabe

  committed
  2fead00

Commits on Sep 13, 2023

• merged feature branch that is in an open PR

  georgemccabe

  committed
  787e037

• clarify version numbers

  georgemccabe

  committed
  63a3399

• add new variables to glossary/wrappers for each wrapper-specific variable for skip/include init/valid times

  georgemccabe

  committed
  0da2c3f

• fix doc errors

  georgemccabe

  committed
  4cba461

• fix function documentation

  georgemccabe

  committed
  b3569ed

• Merge branch 'develop' into feature_2283_include_times

  georgemccabe

  committed
  8ce5375