Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Minimize the seasonal TEC bias by revising the eddy mixing parameterization #69

Open
ZhuxiaoLi66 opened this issue Oct 7, 2022 · 39 comments
Open

Minimize the seasonal TEC bias by revising the eddy mixing parameterization #69

ZhuxiaoLi66 opened this issue Oct 7, 2022 · 39 comments
Assignees
@astridmaute @ZhuxiaoLi66 @twfang @akubaryk @timfullerrowell

Comments

@ZhuxiaoLi66
Copy link
Contributor

This project is mainly aiming to minimize the operational seasonal TEC bias through the revision of the eddy mixing parameterization in WAM.
At the same time, we will also keep a close eye on the influence of parameterization revision on the change of the neutral density with some validation.
The bias improvement also includes estimating the TEC change due to the day-number bug fix in WAM.
The output of the operational WAM-IPE and that of several designed free WAM-IPE coupling annual runs for the period about 20210715 - 20220705, also the output of the operational GloTEC during the same year-long period will be applied to do the diagnostics and mathematic analysis for parameterization change.
@ZhuxiaoLi66
Copy link
Contributor Author

Here are some project history & experience here for future applications & notes.

  1. Since there are no drivers on Hera for this recent year-long period, it took me quite some time to download the right data from HPSS to Hera and learn how to make the model driver files from the original files with the help of Adam. Now the run script for WAM-IPE free coupling and the drivers folder is built up very well.
  2. Also got the lesson that we can't use some of the operational WAM output as the simulation IC files directly, Since, after the weighting of some extreme abnormal observational values, the output becomes numerically & physically unbalanced, and will cause the runs crash at a certain time after quite a period, sometimes takes 2 or 3 weeks of the simulation time length to crash. it is better to just use the output before data assimilation.

Current runs situation & history:

  1. Downloaded the related 2D output of year-long oprational WAM-IPE from HPSS.
  2. There are two WAM-IPE coupling free runs that have been completed for 9 months since 20210715.
    a) the run with the same setting of the current operational version. includes the dayno bug & keddy0=140.
    b) the run with the same setting except getting the dayno bug fixed. Keddy0=140.
  3. The two final effective annual runs started on Sep. 21, which means it takes 17 days to complete 9 months of simulation time length for the annual coupling simulations.

@ZhuxiaoLi66
Copy link
Contributor Author

We tested different eddy mixing parameterization sets as following,

  1. original: keddy0=140, keddy_semiann= 60
  2. para1: keddy0=80, keddy_semiann= 20
  3. para2: keddy0=40, keddy_semiann= 0
  4. para3: keddy0=60, keddy_semiann= 0
  5. para4: keddy0=70, keddy_semiann= 0

Based on many comparisons of the TEC bias ratio against GloTEC of the annual runs with the different parameterization sets,
we finally choose para3 whose bias ratio time serious is the closest to the ideal value of 1.0.
The following are two plots for the latest update of the TEC bias ratio.
bias_ratio_dayno_para3_para4_20211229
bias_ratio_dayno_para3_20220322

@ZhuxiaoLi66
Copy link
Contributor Author

In the meantime, we verified the performance of the new parameter set on neutral density for 2015 St. Patrick storm.
We do validation of the WAM neutral density along the GRACE satellite orbits during the storm time.
The results are very encouraging, the storm run with the new parameter set & observational driver input have a better match with GRACE data at storm peak time against former parameterization.
Den_20150316_storm_para3_newBz_ori_msis2_orbit_mean300_5lines

@ZhuxiaoLi66
Copy link
Contributor Author

Here the plot also includes the performance of the WAM run with new parameters in the newBz SW driver-derived mode.
Den_20150316_storm_para3_newBz_ori_msis2_GRACE_orbit_mean300_6lines

@ZhuxiaoLi66
Copy link
Contributor Author

bias_ratio_dayno_para3_20220322

@ZhuxiaoLi66
Copy link
Contributor Author

bias_ratio_dayno_para3_20220528

@twfang
Copy link
Contributor

twfang commented Jan 24, 2023

@ZhuxiaoLi66 Thanks for the plots. Moving forward, could we modify the legend so that it says "operational run with dayno_fix" instead of "dayno_fix", so that we remember what are these runs? Glad to see that the testing is going well. Thanks!

@ZhuxiaoLi66
Copy link
Contributor Author

@twfang this run is not an operational run, we could do 'keddy0=140 & keddy_semiann=60' to remind us for later review. since the dayno_fix applies for both of the runs, maybe no need to mention it here.

@ZhuxiaoLi66
Copy link
Contributor Author

The following is the update of the para3 run which extends to 20220710 now. so far so good.

bias_ratio_dayno_para3_20220710

@ZhuxiaoLi66
Copy link
Contributor Author

The para3 (keddy0=60) annual run is extended to 20221226, almost done.
The following is the update of its bias ratio plot. Looks like we have gotten kind of high values at the end of August and the beginning of September.

@ZhuxiaoLi66
Copy link
Contributor Author

bias_ratio_dayno_para3_20221226

@ZhuxiaoLi66
Copy link
Contributor Author

In general, it is a little bit high, while looks fine.

@timfullerrowell
Copy link

timfullerrowell commented Feb 10, 2023 via email

@ZhuxiaoLi66 ZhuxiaoLi66 reopened this Feb 10, 2023
@ZhuxiaoLi66
Copy link
Contributor Author

Thanks, Tim.
It seems the semi-annual peaks are already at the equinox, are you sure what we want to add more? maybe in the opposite way?
Please correct me if I am wrong.

@ZhuxiaoLi66
Copy link
Contributor Author

maybe we can use a minus sign for the peak.

@akubaryk
Copy link
Member

dev run on WCOSS2 getting keddy0=70 and keddy_semiann=0 from cycle 2023021618

outputs on HPSS: /NCEPDEV/swpc-para/1year/swpc.wam/WCOSS2/upgrade

@ZhuxiaoLi66
Copy link
Contributor Author

Got it, thanks, Adam.

@ZhuxiaoLi66
Copy link
Contributor Author

A new eddy mixing ratio parameter set (keddy0=70 and keddy_semiann=-10) has been designed for a new annual test WAM-IPE run which aims to get a better TEC output bias ratio against GloTEC data.

@ZhuxiaoLi66
Copy link
Contributor Author

The following is the performance comparison among para5 (keddy0=70 and keddy_semiann=-10), para4 (keddy0=70 and keddy_semiann=0)) and para3 (keddy0=60 and keddy_semiann=0). Seem para5 works well and decrease the semiannual amplitude as expected so far.

bias_ratio_dayno_para345_20220410_orange

@ZhuxiaoLi66
Copy link
Contributor Author

the following is the update of the bias ratios for para5, and para4 annual runs.
bias_ratio_dayno_para345_20220726_orange

@ZhuxiaoLi66
Copy link
Contributor Author

This plot shows the comparison of para4 (keddy0=70) and para3 (keddy0=60) only with further extended time.
bias_ratio_dayno_para34_20220915

@ZhuxiaoLi66
Copy link
Contributor Author

The following bias ratio plot is a comparison between the latest eddy mixing coefficient test (keddy0=70, keddy_semiann=-10) annual run and the run with the original eddy mixing parameters (keddy0=140, keddy_semiann=60).
bias_ratio_dayno_para5_20221214_CONOPS12_present

@ZhuxiaoLi66
Copy link
Contributor Author

With mean, stddev, and RMSE from 1.0 base.

bias_ratio_dayno_para5_20221214_CONOPS12_std

@ZhuxiaoLi66
Copy link
Contributor Author

A new parameterization run with setting keddy_semiann to -15 against the value of -10 is running, aiming to further mitigate the slight seasonal amplitude in the above bias ratio plot for -10 run.
so far we have gotten output for the period of 20211101-20220408. The results are not as expected, it seems the new parameter makes the spikes bigger, both at around equinox and some other points as shown in the plot below. this might due to the big variation amplitude, or the complicated response of the physics to the parameterization.

bias_ratio_dayno_para56_20220408

@ZhuxiaoLi66
Copy link
Contributor Author

the run also get a bigger rmse and std, about 0.11 against 0.10 for -10 run.

@ZhuxiaoLi66
Copy link
Contributor Author

The annual run with the new parameterization of skeddy0=70, skeddy_semiann=-10, has been extended to the end of Apr. 2023. The extended run shows a lower bias ratio against GloTEC on average from Mar to Apr of 2023 than the same period of 2022. This explains the recent (from Mar) persistent low bias ratio for operational WAM-IPE output in plots of Daily_TEC_bias_ratio_WAM-IPE/GloTEC.

bias_ratio_dayno_para5_20230429
daily_TEC_bias_ratio_WAM-IPE_GloTEC_20230428
daily_TEC_bias_ratio_WAM-IPE_GloTEC_20230329

@ZhuxiaoLi66
Copy link
Contributor Author

bias_ratio_20211101_20230728

@ZhuxiaoLi66
Copy link
Contributor Author

daily_TEC_bias_ratio_WAM-IPE_GloTEC (4)
daily_TEC_bias_ratio_WAM-IPE_GloTEC (3)

@ZhuxiaoLi66
Copy link
Contributor Author

daily_TEC_bias_ratio_WAM-IPE_GloTEC (2)

@ZhuxiaoLi66
Copy link
Contributor Author

The plot below shows the comparison of the TEC bias ratio against GloTEC between recent operational WAM-IPE output and WAM-IPE freerun with the new Keddy parameters. The period of Apr.14-Oct.1 2023 is time after applying the new parameters in the operational runs.
bias_ratio_opr_para5_20230414_1001

daily_TEC_bias_ratio_WAM-IPE_GloTEC_0930

@ZhuxiaoLi66
Copy link
Contributor Author

bias_ratio_opr_wfs_wrs_para5_20230414_1001

@timfullerrowell
Copy link

timfullerrowell commented Nov 8, 2023 via email

@akubaryk
Copy link
Member

akubaryk commented Nov 8, 2023

No, this is the relevant namelist for the 2023/11/08 18Z WFS forecast that is currently running.

     wam_climate=.false., wam_swpc_3day=.true., wam_cires_rdata=.false.,
     wam_swin = .true.,
     wam_smin=.false., wam_smax=.false., wam_saver=.true., wam_geostorm=.false.,
     wam_gwphys=.false., wam_solar_in=.false., wam_ion_in=.false.,
     wam_das_in=.false., wam_smet_in=.false., wam_netcdf_inout=.true.,
     wam_tides_diag=.false., wam_pws_diag=.false., wam_gws_diag=.false.,
     jh0=1.75, jh_tanh=0.5, jh_semiann=0.5, jh_ann=0.0, jh_st0=25000.0, jh_st1=5000.0,
     skeddy0=70.0, skeddy_semiann=-10.0, skeddy_ann=0.0,
     tkeddy0=280.0, tkeddy_semiann=0.0, tkeddy_ann=0.0
/

@ZhuxiaoLi66
Copy link
Contributor Author

I checked the ratio values, they are very close, but not exactly the same, also given they are global mean values. We could say that the wrs runs are different from the corresponding wfs runs by different IC files, but output very close to the latter, especially at the global mean value level.

@ZhuxiaoLi66
Copy link
Contributor Author

Tim, if you mean picking up the new parameters for the operational runs, I think Adam gave the answer.

@timfullerrowell
Copy link

timfullerrowell commented Nov 9, 2023 via email

@ZhuxiaoLi66
Copy link
Contributor Author

Below is the plot same with above but add the latest October output, it seems we still running low in TEC against GloTEC with the new parameters and recent drivers...

bias_ratio_TEC_20230414_1031_wfs_wrs_freerun

@ZhuxiaoLi66
Copy link
Contributor Author

The PPT below is a summary of the recent investigation on the low bias-ratios of WAM-IPE against GloTEC in 2023 than 2022 with the new eddy-mixing parameterizations.

Uploading Bias_ratio_low_in 2023.pptx…

@ZhuxiaoLi66
Copy link
Contributor Author

I am sorry, everyone. It seems that the uploading of the PPT in my home machine didn't succeed. I have to retry this after I go back home this afternoon. Sorry about this!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@astridmaute astridmaute

@ZhuxiaoLi66 ZhuxiaoLi66

@twfang twfang

@akubaryk akubaryk

@timfullerrowell timfullerrowell

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@astridmaute @ZhuxiaoLi66 @twfang @akubaryk @timfullerrowell