Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

build quadrature + validation for 3 storms #64

Open
ghost opened this issue Dec 2, 2021 · 23 comments
Open

build quadrature + validation for 3 storms #64

ghost opened this issue Dec 2, 2021 · 23 comments
Assignees
@WPringle

Comments

@ghost
Copy link

ghost commented Dec 2, 2021 •
edited by ghost
Loading

Storm mesh subsetted ensemble built ensemble completed results combined surrogate fit
Florence 2018
Sandy 2012
Irma 2017
@ghost ghost assigned ghost and WPringle Dec 2, 2021
@ghost
Copy link
Author

ghost commented Dec 2, 2021 •
edited by ghost
Loading

list of COASTAL Act storms:

  • Michael 2018
  • Florence 2018
  • Irma 2017
  • Maria 2017
  • Harvey 2017
  • Sandy 2012
  • Irene 2011
  • Ike 2008
  • Isabel 2003

@WPringle
Copy link
Contributor

WPringle commented Dec 2, 2021 •
edited
Loading

@zacharyburnettNOAA Thanks, I have made the subsetted meshes and got the storm data, but seems like cannot access Orion at the moment. Will let you know once I can login to Orion and transfer them over.

@WPringle
Copy link
Contributor

WPringle commented Dec 3, 2021

@zacharyburnettNOAA Please find the subsetted meshes for each storm on Orion at: /work/noaa/nosofs/share/wpringle/HSOFS_Subset_Storms

This directory is actually something I made for the paper, where you can select any storm you want and run the besttrack_subset_merge.py python script to subset and merge the mesh in a few minutes. But need MATLAB and OceanMesh2D.

@ghost
Copy link
Author

ghost commented Dec 3, 2021

excellent, thanks!

@WPringle
Copy link
Contributor

@zacharyburnettNOAA Just wondering if have any update on Sandy and Irma simulations?

@ghost
Copy link
Author

ghost commented Dec 14, 2021

@zacharyburnettNOAA Just wondering if have any update on Sandy and Irma simulations?

I had a few issues with running aswip but I have now figured out the problems, I should have the model results soon

@WPringle
Copy link
Contributor

@zacharyburnettNOAA Just wondering if have any update on Sandy and Irma simulations?

I had a few issues with running aswip but I have now figured out the problems, I should have the model results soon

Great, thanks.
But just a note if we do use nws=8 standard holland model then aswip is not needed.

@ghost
Copy link
Author

ghost commented Dec 22, 2021

I've tracked down the issue; when I was reading in the Sandy and Irma ATCF, the parser was parsing the storm name as a single newline \n for some reason, which was messing up the output. I'm fixing it now so we should have Sandy and Irma results running soon

@ghost
Copy link
Author

ghost commented Dec 23, 2021 •
edited by ghost
Loading

Spinup for Sandy 2012 fails at 0.11% on the subsetted mesh with the following error:

(ensembleperturbation)  Zachary.Burnett@hfe12 � /scratch2/COASTAL/coastal/scrub/Zachhary.Burnett/working/zach/nems_adcirc/run_20211202_sandy_besttrack_250m_quadrature�                                                                                
 > check_completion spinup --verbose

{
    "spinup": {
        "status": "error",
        "progress": "0.11%",
        "error": {
            "ADCIRC_COLDSTART_SPINUP.err.log": [
                " TIME STEP =     639     ITERATIONS =   16     TIME =  0.12780000E+04\n",
                "  ELMAX =  2.0421E+001 AT NODE   557516  SPEEDMAX =  1.5602E+000 AT NODE   944594  ON MYPROC =    9   ** WARNING: Elevation.gt.WarnElev **\n",
                " TIME STEP =     640     ITERATIONS =   16     TIME =  0.12800000E+04\n",
                "  ELMAX =  2.2397E+001 AT NODE   557516  SPEEDMAX =  1.6384E+000 AT NODE   944594  ON MYPROC =    9   ** WARNING: Elevation.gt.WarnElev **\n",
                " TIME STEP =     641     ITERATIONS =   16     TIME =  0.12820000E+04\n",
                "  ELMAX =  2.4613E+001 AT NODE   557516  SPEEDMAX =  1.5678E+000 AT NODE   944594  ON MYPROC =    9   ** WARNING: Elevation.gt.WarnElev **\n",
                " TIME STEP =     642     ITERATIONS =   17     TIME =  0.12840000E+04\n",
                "  ELMAX =  2.7103E+001 AT NODE   557516  SPEEDMAX =  1.5741E+000 AT NODE   944594  ON MYPROC =    9   ** WARNING: Elevation.gt.WarnElev **\n",
                " TIME STEP =     643     ITERATIONS =   16     TIME =  0.12860000E+04\n",
                "  ELMAX =  2.9921E+001 AT NODE   557516  SPEEDMAX =  1.5810E+000 AT NODE   944594  ON MYPROC =    9   ** WARNING: Elevation.gt.WarnElev **\n",
                " TIME STEP =     644     ITERATIONS =   16     TIME =  0.12880000E+04\n",
                "  ELMAX =  3.3123E+001 AT NODE   557516  SPEEDMAX =  1.5891E+000 AT NODE   944594  ON MYPROC =    9   ** WARNING: Elevation.gt.WarnElev **\n",
                " TIME STEP =     645     ITERATIONS =   16     TIME =  0.12900000E+04\n",
                "  ELMAX =  3.6771E+001 AT NODE   557516  SPEEDMAX =  1.5983E+000 AT NODE   944594  ON MYPROC =    9   ** WARNING: Elevation.gt.WarnElev **\n",
                " TIME STEP =     646     ITERATIONS =   17     TIME =  0.12920000E+04\n",
                "  ELMAX =  4.1013E+001 AT NODE   557516  SPEEDMAX =  1.6086E+000 AT NODE   944594  ON MYPROC =    9   ** WARNING: Elevation.gt.WarnElev **\n",
                " TIME STEP =     647     ITERATIONS =   17     TIME =  0.12940000E+04\n",
                "  ELMAX =  4.6434E+001 AT NODE   557516  SPEEDMAX =  1.6199E+000 AT NODE   944594  ON MYPROC =    9   ** WARNING: Elevation.gt.WarnElev **\n",
                " TIME STEP =     648     ITERATIONS =   17     TIME =  0.12960000E+04\n",
                "  ELMAX =  5.3241E+001 AT NODE   557516  SPEEDMAX =  1.6321E+000 AT NODE   944594  ON MYPROC =    9   ** WARNING: Elevation.gt.WarnElev **\n",
                " TIME STEP =     649     ITERATIONS =   18     TIME =  0.12980000E+04\n",
                "  ELMAX =  6.4476E+001 AT NODE   557516  SPEEDMAX =  1.8708E+000 AT NODE   557516  ON MYPROC =    9   ** WARNING: Elevation.gt.WarnElev **\n",
                " TIME STEP =     650     ITERATIONS =   23     TIME =  0.13000000E+04\n",
                "  ELMAX =  8.2971E+001 AT NODE   557516  SPEEDMAX =  2.7947E+000 AT NODE   557518  ON MYPROC =    9   ** WARNING: Elevation.gt.WarnElev **\n",
                " TIME STEP =     651     ITERATIONS =   22     TIME =  0.13020000E+04\n",
                "  ELMAX =  1.1504E+002 AT NODE   557516  SPEEDMAX =  4.5430E+000 AT NODE   557518  ON MYPROC =    9   ** WARNING: Elevation.gt.WarnElev **\n",
                " TIME STEP =     652     ITERATIONS =   21     TIME =  0.13040000E+04\n",
                "  ELMAX =  1.7060E+002 AT NODE   557516  SPEEDMAX =  7.0660E+000 AT NODE   557518  ON MYPROC =    9   ** WARNING: Elevation.gt.WarnElev **\n",
                " TIME STEP =     653     ITERATIONS =   19     TIME =  0.13060000E+04\n",
                "  ELMAX =  2.6385E+002 AT NODE   557516  SPEEDMAX =  1.0913E+001 ATNODE   557518  ON MYPROC =    9   ** WARNING: Elevation.gt.WarnElev **\n",
                " TIME STEP =     654     ITERATIONS =   18     TIME =  0.13080000E+04\n",
                "  ELMAX =  4.1041E+002 AT NODE   557516  SPEEDMAX =  1.6903E+001 AT NODE   557518  ON MYPROC =    9   ** WARNING: Elevation.gt.WarnElev **\n",
                " TIME STEP =     655     ITERATIONS =   18     TIME =  0.13100000E+04\n",
                "  ELMAX =  6.1703E+002 AT NODE   557518  SPEEDMAX =  2.2961E+001 AT NODE   557518  ON MYPROC =    9   ** WARNING: Elevation.gt.WarnElev **\n",
                " TIME STEP =     656     ITERATIONS =   16     TIME =  0.13120000E+04\n",
                "  ELMAX =  8.5691E+002 AT NODE   557518  SPEEDMAX =  3.4116E+001 AT NODE   557516  ON MYPROC =    9   ** WARNING: Elevation.gt.WarnElev **\n",
                " TIME STEP =     657     ITERATIONS =   19     TIME =  0.13140000E+04\n",
                "  ELMAX =  1.0244E+003 AT NODE   557518  SPEEDMAX =  3.8499E+001 AT NODE   557516  ON MYPROC =    9   ** WARNING: Elevation.gt.WarnElev **\n",
                " TIME STEP =     657     ITERATIONS =   19     TIME =  0.13140000E+04\n",
                "  ELMAX =  1.0244E+003 AT NODE   557518  SPEEDMAX =  3.8499E+001 AT NODE   557516  ON MYPROC =    9\n",
                "  ** ERROR: Elevation.gt.ErrorElev, ADCIRC stopping. **\n",
                "srun: error: h15c16: tasks 0-10: Exited with exit code 1\n",
                "srun: launch/slurm: _step_signal: Terminating StepId=26799133.0\n"
            ]
        }
    }
}

I will attempt to use the full 250m HSOFS mesh to rule out problems with the mesh subsetting.

@WPringle
Copy link
Contributor

@zacharyburnettNOAA Thanks that's a good idea to try the non-subsetted version. Also you may try to reduce time step (e.g., if using 6 sec reduce by half to 3 sec).

@ghost
Copy link
Author

ghost commented Dec 27, 2021

Also you may try to reduce time step (e.g., if using 6 sec reduce by half to 3 sec).

good idea, I will try that as well.

I also found out that the parsed best track data for Sandy 2012 had incorrect Rmax (it was set to 0 across the entire track), which made ADCIRC break upon hotstart. I manually updated the file and perturbations and am running the new ensemble for Sandy 2012 now.

@ghost
Copy link
Author

ghost commented Dec 29, 2021 •
edited by ghost
Loading

quadrature + validation runs for Sandy 2012 and Irma 2017 have completed at a 3 second time interval and proper best track (manually fixed the Rmax) over the full HSOFS 250m mesh (i will retry the subsets later).

I am now running combine_results to build the necessary NetCDF files from which we can fit a surrogate

@WPringle
Copy link
Contributor

WPringle commented Jan 5, 2022

@zacharyburnettNOAA Can you please share with me (wpringle@anl.gov) the google slides we were making? My NOAA account was temporally disabled. Thanks.

@ghost
Copy link
Author

ghost commented Jan 5, 2022

wpringle@anl.gov

sure thing, sent

@WPringle
Copy link
Contributor

WPringle commented Jan 5, 2022

@zacharyburnettNOAA Thanks I got it. I'll try out your edits to the code. The joint KLPC seems to be working anyway. Actually gives very similar result to without using KL. I guess that's expected, just less computationally expensive.

@WPringle
Copy link
Contributor

WPringle commented Jan 7, 2022 •
edited
Loading

@zacharyburnettNOAA Hi Zach, can you do the Florence best track again for another 50 random samples? I want to try build training without the quadrature (maybe also try UQTk again and then convert to the chaospy ndpoly type). So then I can use 50 of samples for training and 50 for validation. Or even do another 100 to make sure.

@WPringle
Copy link
Contributor

WPringle commented Jan 7, 2022

@zacharyburnettNOAA Are the Sandy combined results available on Orion?

@WPringle
Copy link
Contributor

@zacharyburnettNOAA I noticed that the along_track perturbation doesn't seem to be doing anything in the Florence best track example (the model results and the fort.22 for location is same as other perturbations for different along_track and same cross_track). Can please make sure that along_track is being called correctly.

@ghost
Copy link
Author

ghost commented Jan 11, 2022 •
edited by ghost
Loading

@zacharyburnettNOAA Are the Sandy combined results available on Orion?

@WPringle sorry I forgot to reply to this. I put the Sandy results on Orion at /work/noaa/nosofs/share/working/hera/nems_adcirc/run_20211227_sandy_besttrack_250m_quadrature/. The Irma results still have some issues that I have been working through.

I will build another 100-member validation run for Florence once Hera comes out of its current maintenance period, which should be tomorrow

@WPringle
Copy link
Contributor

@zacharyburnettNOAA Zach I had some other thoughts now.
Let's try to simplify things and start with a 2 variable ensemble. That way quadrature is just 16 members. We can also try other sampling methods like latin_hypercube.

It seems like cross_track and max_wind_speed are the most impactful variables, so can we work on producing an ensemble of just perturbing those, with various sampling methods, and then can do a thorough investigation of best methodology.

@ghost
Copy link
Author

ghost commented Jan 11, 2022 •
edited by ghost
Loading

that sounds like a simpler way to start, good idea

I will focus on compiling an ensemble for those two variables

@WPringle
Copy link
Contributor

WPringle commented Jan 11, 2022 •
edited
Loading

@zacharyburnettNOAA I implemented sample rule options in 4aa8ad3. So now we can try the different options to try and determine the best one.

@WPringle
Copy link
Contributor

@zacharyburnettNOAA FYI,I'm trying out the different sampling methods and running the two variable ensembles on Argonne machine right now

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@WPringle WPringle

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@WPringle