Have the driver script process both python scripts and YAMLs + bufr2ioda.x #730
Conversation
|
Coding norms are failing, I will fix this after my meeting(s) |
There was a problem hiding this comment.
Test proposed changes in JEDI gdas-validation. 2021080100 gdasprepatmiodaobs aborted with the following traceback
2023-11-15 18:38:52,983 - INFO - run_bufr2ioda.py: Executing /work2/noaa/da/rtreadon/gdas-validation/global-workflow/sorc/gdas.cd/build/bin/bufr2ioda.x /work2/noaa/da/rtreadon/gdas-validation/comrot/gdas_eval_satwind_JEDI/gdas.20210801/00/obs/atms_2021080100.yaml
PRE-MAIN-INFO BufrParser: Parsing file /work/noaa/rstprod/dump/gdas.20210801/00/atmos/gdas.t00z.atms.tm00.bufr_d
PRE-MAIN-INFO Executing Queries
PRE-MAIN-INFO Building Bufr Data
PRE-MAIN-INFO Exporting Data
ATMS_Spatial_Average: chechking bt_inout (original) ...
minval/maxval bt_inout (original) = 133.0100 310.3900
Unable to open atms_beamwidth.txt
PRE-MAIN-ERROR Exception: Bad parameter: Missing required parameters: sensor. Check your configuration.
terminate called after throwing an instance of 'eckit::BadParameter'
what(): Bad parameter: Missing required parameters: sensor. Check your configuration.
Traceback (most recent call last):
File "/work2/noaa/da/rtreadon/gdas-validation/global-workflow/ush/run_bufr2ioda.py", line 94, in <module>
bufr2ioda(args.current_cycle, args.RUN, args.DMPDIR, args.config_template_dir, args.COM_OBS)
File "/work2/noaa/da/rtreadon/gdas-validation/global-workflow/ush/python/wxflow/logger.py", line 266, in wrapper
retval = func(*args, **kwargs)
File "/work2/noaa/da/rtreadon/gdas-validation/global-workflow/ush/run_bufr2ioda.py", line 79, in bufr2ioda
cmd()
File "/work2/noaa/da/rtreadon/gdas-validation/global-workflow/ush/python/wxflow/executable.py", line 230, in __call__
raise ProcessError(f"Command exited with status {proc.returncode}:", long_msg)
wxflow.executable.ProcessError: Command exited with status -6:
'/work2/noaa/da/rtreadon/gdas-validation/global-workflow/sorc/gdas.cd/build/bin/bufr2ioda.x' '/work2/noaa/da/rtreadon/gdas-validation/comrot/gdas_eval_satwind_JEDI/gdas.20210801/00/obs/atms_2021080100.yaml'
Not sure if this is due to my set up or an issue with the proposed changes. Investigating.
|
@RussTreadon-NOAA I get the same error now in testing, let me investigate |
|
We have |
|
@RussTreadon-NOAA yeah I think so, now I just need to figure out how best to copy it to the runtime directory. |
|
Add the following to before executing |
|
Copying Required parameter Who developed the atms converter? How did they test it? FYI, we need to increase the wall time for prepatmiodaobs. 10 minutes is no longer sufficient. Later we can refactor prepatmiodaobs to concurrently run multiple converters via CFP |
|
@emilyhcliu is going to look into it |
|
I think I know the issue, I'm testing now, (it's a one liner) |
Yes, @RussTreadon-NOAA I will take over and test ATMS with the changes in this PR. |
|
I think the atms yaml for bufr2ioda is missing the sensor in the following section: |
|
@emilyhcliu agreed, now I'm getting a splits issue, which is related to my script, so I will still need to do a bit more debugging. Standby. |
|
The latest commit works: But @RussTreadon-NOAA is right, 10 mins is no longer sufficient. We need to parallelize this. I will look into that next. |
Here is the updated ATMS YAML for bufr2ioda: I tested it using a stand-alone IODA-bundle. It worked. You can grab this one and add it to the GDASApp. Please feel free to rename the yaml. |
There was a problem hiding this comment.
Fold feature/gdas-validation-driver at 5a8a535 into working copy of gdas-validation. Run 2021080100 gdasprepatmiodaobs. Job ran to completion with rc=0.
Approve.
This PR allows for all obtypes to be processed in two loops.
We still need to figure out a way to process these in parallel (somewhat).