2D Decomposition (#339)

* Changes to support 2D Decomposition shape changes. Full I dimension remains in all arrays bu t SURFCE is altered to use isx:iex bounds which are equivalent to IM which is equivalent to 1:IM * test of commit only added one comment to PROCESS.f * comment test 2 * Added more routines to 2D decomposition * Changed isx and iex to ISTA and IEND in listed routines. ISTA had prior use in MDL2P.f and that needed changing. IEND had prior use in PARA_CONFIG, changed to IENDJ in PARA_CONFIG2 where needed PARA_CONFIG did not need changing * test commit of new base into clone * repair of a bunch of changed routines somehow corrupted by a merge 5/3/2021 * fixed 5/25 problem * added ctlblk change to support ista and iend partial I dimensons * added para_range2 to PARA_RANGE.f to support 2D decomposition * Modified MPI_FIRST.f to support a 2D decomposition but the actual numbers used remain the 1D special case and the changes just add two more indices for start and end I domains and logic to convert scatter counts and displacements to the product of the I and J subdomain sizes rather than I full domain x J subdomain. * Added support for halo settings of 2D boundaries i.e ista_2l to match the 1D analogs long present in the source 6/1/2021 * Added definitions for ista_2u, and ista_2u in MPI_FIRST.f * Reshaped arrays in MDL2FLD.f to support 2D decomposition. Modified TWO (only) loops to try thie * Remove TIMEF.f. * 20210702 JesseMeng modify MPI_FIRST.f MDLFLD.f for 2D decomposition * 20210706 BoCui modify INITPOST_GFS_NETCDF_PARA.f * 20210707 JesseMeng modified grib2_module.f for 2d decomposition * test version INITPOST_GFS_NETCDF_PARA.f * test version INITPOST_GFS_NETCDF_PARA.f and MPI_FIRST.f * 20210713 BoCui test INITPOST_GFS_NETCDF_PARA.f, MPI_FIRST.f and ALLOCATE_ALL.f * 20210713 BoCui test INITPOST_GFS_NETCDF_PARA.f, MPI_FIRST.f and ALLOCATE_ALL.f * 20210719 BoCui Modified CLDRAD.f for 2d decomposition * 20210816 Jesse Meng commit George's EXCH update for 1 layer 2D halos transforming * 20210816 Jesse Meng remove ifcore in EXCH * 20210903 Bo Cui update ALLOCATE_ALL.f after new merge from 'upstream/develop' * 20210903 Bo Cui Added new routines to 2D decomposition * 20210903 Jesse Meng fixed the ieql allocation bug in ALLOCATE_ALL.f * 20210904 Bo Cui fixed CLDRAD.f * 20210913 Jesse Meng commit progress of 2D decomposition * 20210917 Bo Cui add new routines to 2D decomposition * 20210917 Jesse Meng remove 4 legacy files * 20210917 Jesse Meng remove legacy code SLP_NMM EXCH2 * 20210917 Jesse Meng remove legacy code INITPOST_GFS_SIGIO.f INITPOST_GFS_NEMS.f * 20210917 Jesse Meng add INITPOST_GFS_NEMS.f back to avoid compiling error. * 20210930 Jesse Meng add George's itag/numx entry and progress on 2D decompose * 20211015 Jesse Meng progress on 2D DECOMPOSITION * 20211017 Bo Cui add new subroutines to UPP 2D decomposition * 20211026 Jesse Meng commit progress in 2D DECOMPOSITION * 20211026 Jesse Meng fix INITPOST_GFS_NETCDF_PARA.f for 2D DECOMPOSITION * 20211026 Jesse Meng update SURFCE.f for 2D DECOMPOSITION * 20211103 Jesse Meng commit progress of 2D DECOMPOSITION * 20211106 Bo Cui commit progress of 2D decomposition * 20211109 Jesse Meng updates for 2D DECOMPOSITION * 20211110 Jesse Meng updates in 2D DECOMPOSITION * 20211112 Jesse Meng updates for 2D DECOMPOSITION * 20211115 Jesse Meng updates for 2D DECOMPOSITION * 20211119 Jesse Meng updates for 2D DECOMPOSITION * 20211201 Jesse Meng move CALVOR to UPP_PHYSICS module; implement fullpole in MPI_FIRST * 20211202 Jesse Meng minor update of MPI_FIRST using mpi_allgatherv * 20211203 Jesse Meng implement fullpole in MDL2THANDPV and CALDIV * 20211207 Jesse Meng update MISCLN.f SURFCE.f * 20211208 Jesse Meng add parm/hafs_nosat files * 20211213 Bo Cui updates for 2D decomposition * 20211213 Jesse Meng update the merged 'upstream/develop' and 'post_2d_decomp' branch to 2d_decomp style * 20211215 Jesse Meng use i=0:im+1 for GFS in MPI_FIRST and EXCH * 20211216 Jesse Meng minor update UPP_PHYSICS for i=0:im+1 expansion * 20220104 Jesse Meng add George's minor fix in EXHC.f * 20220118 Jesse Meng commit George's cleaned up code. * 20220203 Jesse Meng - update MAPSSLP.f to be consistent with develop branch. * 20220222 Jesse Meng add checkcoords flag in EXCH.f to move around George's debug code. * 20220302 Jesse Meng Restrict computation from undefined grids * 20220304 Bo Cui Add a reset of numx=1 if remainder of num_procs/numx is not 0 * 20220310 Jesse Meng Add a reset of numx=1 if(numx>num_procs/2) * 20220322 Jesse Meng mpi_allgatherv change communicator to MPI_COMM_COMP * 20220328 Jesse Meng bug fix for passing 2d subarrays between subroutines * 20220328 Jesse Meng minor fix for CLDRAD call BOUND variable array size * 20220330 Jesse Meng fix cloud cover variabes full field collection in 2d decomp * 20220401 Jesse Meng minor fix for CALMCVG.f and update CALUPDHEL.f with develop * 20220414 BoCui sync and merge UPP/develop into post_2d_decomp (#7) * Added lambert conformal projection for FV3SAR and handled composite reflectivity correctly (#14) (#2) * exec is selectable for executable directory (#448) * Doxygen for CALGUST (#451) * Doxygen for CALDRG.f * This is part of Issue #392 Fixes the doxygen warnings in CALGUST.f Please review this subroutine. Thank you. * The fix in CALUPDHEL.f (#458) * Modify CALUPDHEL.f for restricting undefined grids in computation. * Update VERSION to 10.0.12. * Bug fix for SLLEVEL bound issue when not RUC LSM (#463) Co-authored-by: Tracy <tracy.hertneky@noaa.gov> * Doxygen caldrg caldwcp calgust refinement (#464) * Further refinement to the tables and logs. * Further refinement to the tables and logs. (#455) * This is part of Issue #392 (#460) Fixes the doxygen warnings in CALHEL.f CALHEL2.f CALHEL3.f Please review this subroutine. Thank you. * Unify global and regional FV3 read interfaces (#453) * Unify the interfaces for reading FV3 outputs in netcdf. * Remove interface INITPOST_GFS_NETCDF_PARA. * Remove INITPOST_GFS_NETCDF.f. * Remove the capability of serial netcdf reading FV3 outputs. * Correct reading rswinc. * Remove duplication in CLDRAD.f. * Remove duplicated avgalbedo reading * Add changes for reading pwat from model. * Clean up commented out code * Clean up duplicated lines * Add ability to compile script to use non-intel compilers; add Cheyenne modulefiles for gnu and intel (#468) * Add cheyenne modulefile * Intel 19 --> 2021 * Add ability to specify compiler; move all existing modulefiles to ${name}_intel since they are all for intel compilers; add "cheyenne_gnu" modulefile * Revert move of intel modulefiles; now the implicit default for a modulefile is intel, only gnu and other compilers will have the compiler name appended to the modulefile * Allow for lua modulefiles with ".lua" appended * This is part of Issue #392 (#465) Fixes the doxygen warnings in CALLCL.f CALMCVG.f CALMICT.f * This is part of Issue #392 (#466) Fixes the doxygen warnings in CALPBL.f CALPBLREGIME.f CALPOT.f CALPW.f. * Doxygen in CALRAD_WCLOUD_newcrtm.f CALRCH.f CALSTRM.f CALTAU.f CALTHTE.f CALUPDHEL.f. (#467) * This is part of Issue #392. Fixes the doxygen warnings in CALRAD_WCLOUD_newcrtm.f CALRCH.f CALSTRM.f CALTAU.f CALTHTE.f CALUPDHEL.f. * Fixed typo. * Update to PR #458. * Doxygen in CALVOR.f, CALWXT_BOURG.f, CLDRAD.f, COLLECT.f, COLLECT_LOC.f, DEALLOCATE.f, and DEALLOCATE.f (#469) * This is part of Issue #392. Fixes the doxygen warnings in CALVOR.f, CALWXT_BOURG.f, CLDRAD.f, COLLECT.f, COLLECT_LOC.f, DEALLOCATE.f, and DEALLOCATE.f. * Fixed minor bug in CALVOR.f. * Update to PR#453. * More updates. * 20220411 Bo Cui 2D decompositio CALVOR.f and INITPOST_NETCDF.f * 20220411 Bo Cui update doxgen and global and regional FV3 read interface * 20220414 Bo Cui: remove read_netcdf_2d_scatter and read_netcdf_3d_scatter from INITPOST_NETCDF.f * 20220415 Bo Cui delete CALVOR.f, add exch of gdlon in INITPOST_NETCDF.f * 20220421 Bo Cui Doxygen in UPP_PHYSICS.f,add restriction run 2D decomp only for GFS/FV3R in WRFPOST.f Co-authored-by: WenMeng-NOAA <48260754+WenMeng-NOAA@users.noreply.github.com> Co-authored-by: Wen Meng <wen.meng@noaa.gov> Co-authored-by: Chan-Hoo.Jeon-NOAA <60152248+chan-hoo@users.noreply.github.com> Co-authored-by: kayeekayee <kayee.wong@noaa.gov> Co-authored-by: Tracy Hertneky <39317287+hertneky@users.noreply.github.com> Co-authored-by: Tracy <tracy.hertneky@noaa.gov> Co-authored-by: Michael Kavulich <kavulich@ucar.edu> * 20220502 Bo Cui code cleanup * 20220512 Jesse Meng minor fix for INITPOST_GFS_NEMS_MPIIO.f calling EXCH * 20220525 Jesse Meng minor fix to 2d_decomp syntax * Update VERSION to 11.0.0 * 20220607 Jesse Meng add variable declaration block in PARA_RANGE.f * 20220608 Jesse Meng remove blank spaces in sorc/ncep_post.fd/AllGETHERV_GSD.f * Add 2D decomp overview documentation (#6) * Add 2D decomp overview documentation * Update 2D overview Co-authored-by: George Vandenberghe <George.Vandenberghe@noaa.gov> Co-authored-by: wx15gv <George.Vandenberghe@v71a1.ncep.noaa.gov> Co-authored-by: wx22mj <Jesse.Meng@m71a1.ncep.noaa.gov> Co-authored-by: Bo Cui <Bo.Cui@noaa.gov> Co-authored-by: wx22mj <Jesse.Meng@m71a2.ncep.noaa.gov> Co-authored-by: wx22mj <Jesse.Meng@m72a1.ncep.noaa.gov> Co-authored-by: wx22mj <Jesse.Meng@m71a3.ncep.noaa.gov> Co-authored-by: wx22mj <Jesse.Meng@v71a1.ncep.noaa.gov> Co-authored-by: wx22mj <Jesse.Meng@v72a1.ncep.noaa.gov> Co-authored-by: wx22mj <Jesse.Meng@v71a3.ncep.noaa.gov> Co-authored-by: wx22mj <Jesse.Meng@v71a2.ncep.noaa.gov> Co-authored-by: Jesse.Meng <Jesse.Meng@noaa.gov> Co-authored-by: BoCui-NOAA <53531984+BoCui-NOAA@users.noreply.github.com> Co-authored-by: Chan-Hoo.Jeon-NOAA <60152248+chan-hoo@users.noreply.github.com> Co-authored-by: kayeekayee <kayee.wong@noaa.gov> Co-authored-by: Tracy Hertneky <39317287+hertneky@users.noreply.github.com> Co-authored-by: Tracy <tracy.hertneky@noaa.gov> Co-authored-by: Michael Kavulich <kavulich@ucar.edu> Co-authored-by: Jesse Meng <jmeng@Orion-login-4.HPC.MsState.Edu> Co-authored-by: Kate Fossell <fossell@ucar.edu>
NOAA-EMC · Jun 8, 2022 · 6b4a79c · 6b4a79c
1 parent 9ede326
commit 6b4a79c
Show file tree

Hide file tree

Showing 87 changed files with 7,385 additions and 6,242 deletions.
diff --git a/VERSION b/VERSION
@@ -1 +1 @@
-10.2.0
+11.0.0
diff --git a/docs/2D-decomp.md b/docs/2D-decomp.md
@@ -0,0 +1,21 @@
+# 2-D Decomposition Overview
+
+**Author:** George Vandenberghe
+
+**Date:** June 2022
+
+## Comparison of 1D vs. 2D Decomposition
+The 1D decomposition can read state from a model forecast file, either by reading on rank 0 and scattering, or by doing MPI_IO on the model history file using either nemsio, sigio, or netcdf serial or parallel I/O. Very old post tags also implement the more primitive full state broadcast or (a performance bug rectified 10/17) read the entire state on all tasks. This is mentioned in case a very old tag is encountered.  
+
+The 2D decomposition only supports MPI_IO, namely NetCDF Parallel I/O. But the code is backwards compatible and all I/O methods remain supported for the 1D decomposition cases and works for all cases currently supported by older 1D tags and branches.
+
+## 2D Decomposition Design 
+
+The 2D decomposition operates on subdomains with some latitudes and some longitudes.  The subdomains are lon-lat rectangles rather than strips. This means state must be chopped into pieces in any scatter operation and the pieces reassembled in any gather operation that requires a continuous in memory state. I/O and halo exchanges both require significantly more bookkeeping.
+
+The structural changes needed for the 2D decomposition are implemented in MPI_FIRST.f and CTLBLK.f. The CTLBLK.f routine contains numerous additional variables describing left and right domain boundaries. Many additional changes are also implemented in EXCH.f to support 2D halos.  Many additional routines required addition of the longitude subdomain limits but changes to the layouts are handled in CTLBLK.f and the "many additional routines" do not require additional changes when subdomain shapes are changed and have not been a trouble point.
+
+Both MPI_FIRST.f and EXCH.f contain significant additional test code to exchange arrays containing grid coordinates and ensure EXACT matches for all exchanges before the domain exchanges are performed. This is intended to trap errors in the larger variety of 2D decomposition layouts that are possible and most of it can eventually be removed or made conditional at build and run time.
+
+Indices and variables to facilitate the 2D decomposition are found in CTLBLK.f and shared in the rest of UPP through use of CTLBLK.mod.
+
diff --git a/docs/Doxyfile.in b/docs/Doxyfile.in
@@ -855,6 +855,7 @@ WARN_LOGFILE           =
 # Note: If this tag is empty the current directory is searched.
 
 INPUT                  = @abs_top_srcdir@/docs/user_guide.md \
+                       = @abs_top_srcdir@/docs/2D-decomp.md \
                          @abs_top_srcdir@/sorc/ncep_post.fd \
                          @config_srcdir@
 

diff --git a/sorc/ncep_post.fd/ALLOCATE_ALL.f b/sorc/ncep_post.fd/ALLOCATE_ALL.f
diff --git a/sorc/ncep_post.fd/AVIATION.f b/sorc/ncep_post.fd/AVIATION.f
@@ -67,7 +67,7 @@ SUBROUTINE CALLLWS(U,V,H,LLWS)
 !
       USE vrbls2d, only: fis, u10, v10
       use params_mod, only: gi
-      use ctlblk_mod, only: jsta, jend, im, jm, lsm, spval
+      use ctlblk_mod, only: jsta, jend, im, jm, lsm, spval, ista, iend
 !- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
       implicit none
 !
@@ -84,7 +84,7 @@ SUBROUTINE CALLLWS(U,V,H,LLWS)
 !
 
       DO 100 J=JSTA,JEND
-        DO I=1,IM
+        DO I=ISTA,IEND
 
           Z1 = 10.0 + FIS(I,J)*GI                              !Height of 10m levels geographic height (from sea level)
 
@@ -158,20 +158,65 @@ SUBROUTINE CALLLWS(U,V,H,LLWS)
 !>     
 !> @author Binbin Zhou NCEP/EMC  @date 2005-08-16       
       SUBROUTINE CALICING (T1,RH,OMGA, ICING)
-      use ctlblk_mod, only: jsta, jend, im, spval
+!$$$  SUBPROGRAM DOCUMENTATION BLOCK
+!                .      .    .     
+! SUBPROGRAM:    CALICING       COMPUTES In-Flight Icing
+!   PRGRMMR: Binbin Zhou      /NCEP/EMC  DATE: 2005-08-16       
+!     
+! ABSTRACT:  
+!    This program computes the in-flight icing condition
+!    with the T-RH-OMGA algorithm provided by S. Silberberg of
+!    NCEP/AWC (improved new version)
+! 
+!    According to S. Silberberg, Icing happens in following 
+!    situation:
+!       (1) -22C < T < 0C to      
+!       (2)  RH > 70 %
+!       (3) Ascent air, OMGA < 0 
+!       (4) Equivalent Potential Vorticity (EPV) < 0
+!       (5) Cloud water if SLD (supercooled large droplet)
+!
+!    Current version dosn't consider SLD, so cloud water           
+!    is not used. EPV computation is not available for current
+!    NCEP/EMC models(NAM, WRF, RSM), so EPV is also not
+!    used
+!
+! USAGE:    CALL CALICING(T1,RH,OMGA,ICING)
+!   INPUT ARGUMENT LIST:
+!     T1     - TEMPERATURE (K)
+!     RH     - RELATIVE HUMIDITY  (DECIMAL FORM)
+!     OMGA   - Vertical velocity (Pa/sec)
+!
+!   OUTPUT ARGUMENT LIST: 
+!     ICING     - ICING CONDITION (1 or 0)
+!     
+!   OUTPUT FILES:
+!     NONE
+!     
+!   SUBPROGRAMS CALLED:
+!     UTILITIES:
+!     LIBRARY:
+!       NONE
+!     
+!   ATTRIBUTES:
+!     LANGUAGE: FORTRAN 90/77
+!     MACHINE : BLUE AT NCEP
+!$$$  
+!
+      use ctlblk_mod, only: jsta, jend, im, spval, ista, iend
 !- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
       implicit none
 !     
 !     DECLARE VARIABLES.
 !     
-      REAL, DIMENSION(IM,jsta:jend), INTENT(IN)    :: T1,RH,OMGA
-      REAL, DIMENSION(IM,jsta:jend), INTENT(INOUT) :: ICING 
+      REAL, DIMENSION(ista:iend,jsta:jend), INTENT(IN)    :: T1,RH,OMGA
+      REAL, DIMENSION(ista:iend,jsta:jend), INTENT(INOUT) :: ICING 
       integer I,J
 !***************************************************************
 !
 !
       DO J=JSTA,JEND
-        DO I=1,IM
+        DO I=ISTA,IEND
         IF(OMGA(I,J)<SPVAL.AND.T1(I,J)<SPVAL.AND.RH(I,J)<SPVAL) THEN
          IF(OMGA(I,J) < 0.0 .AND.                       &
             (T1(I,J) <= 273.0 .AND. T1(I,J) >= 251.0)   &
@@ -219,7 +264,7 @@ SUBROUTINE CALICING (T1,RH,OMGA, ICING)
       SUBROUTINE CALCAT(U,V,H,U_OLD,V_OLD,H_OLD,CAT)
       use masks, only: dx, dy
       use ctlblk_mod, only: spval, jsta_2l, jend_2u, jsta_m, jend_m, &
-              im, jm
+              im, jm, ista_2l, iend_2u, ista_m, iend_m, ista, iend
       use gridspec_mod, only: gridtype
 !
 !- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
@@ -228,10 +273,10 @@ SUBROUTINE CALCAT(U,V,H,U_OLD,V_OLD,H_OLD,CAT)
 !     
 !     DECLARE VARIABLES.
 !     
-      REAL,DIMENSION(IM,jsta_2l:jend_2u),INTENT(IN)    :: U,V,H, &
+      REAL,DIMENSION(ista_2l:iend_2u,jsta_2l:jend_2u),INTENT(IN)    :: U,V,H, &
                                                 U_OLD,V_OLD,H_OLD
 !      INTEGER,INTENT(IN)                      :: L
-      REAL,DIMENSION(IM,jsta_2l:jend_2u),INTENT(INOUT) :: CAT
+      REAL,DIMENSION(ista_2l:iend_2u,jsta_2l:jend_2u),INTENT(INOUT) :: CAT
 
       REAL  DSH, DST, DEF, CVG, VWS, TRBINDX
       INTEGER  IHE(JM),IHW(JM)
@@ -247,22 +292,22 @@ SUBROUTINE CALCAT(U,V,H,U_OLD,V_OLD,H_OLD,CAT)
        IF(GRIDTYPE == 'A')THEN
         IHW(J)=-1
         IHE(J)=1 
-	ISTART=2
-        ISTOP=IM-1
+	ISTART=ISTA_M
+        ISTOP=IEND_M
         JSTART=JSTA_M
         JSTOP=JEND_M
        ELSE IF(GRIDTYPE=='E')THEN
         IHW(J)=-MOD(J,2)
         IHE(J)=IHW(J)+1
-	ISTART=2
-        ISTOP=IM-1
+	ISTART=ISTA_M
+        ISTOP=IEND_M
         JSTART=JSTA_M
         JSTOP=JEND_M
        ELSE IF(GRIDTYPE=='B')THEN
         IHW(J)=-1
         IHE(J)=0 
-	ISTART=2
-        ISTOP=IM-1
+	ISTART=ISTA_M
+        ISTOP=IEND_M
         JSTART=JSTA_M
         JSTOP=JEND_M
        ELSE	
@@ -271,12 +316,12 @@ SUBROUTINE CALCAT(U,V,H,U_OLD,V_OLD,H_OLD,CAT)
        END IF	
       ENDDO
 
-      call exch_f(U)
-      call exch_f(V)
-      call exch_f(U_OLD)
-      call exch_f(V_OLD)
-      call exch_f(H)
-      call exch_f(H_OLD)
+      call exch(U)
+      call exch(V)
+      call exch(U_OLD)
+      call exch(V_OLD)
+      call exch(H)
+      call exch(H_OLD)
 
       DO 100 J=JSTART,JSTOP
         DO I=ISTART,ISTOP
@@ -451,20 +496,20 @@ SUBROUTINE CALCAT(U,V,H,U_OLD,V_OLD,H_OLD,CAT)
       SUBROUTINE CALCEILING (CLDZ,TCLD,CEILING)
       USE vrbls2d, only: fis
       use params_mod, only: small, gi
-      use ctlblk_mod, only: jsta, jend, spval, im, modelname
+      use ctlblk_mod, only: jsta, jend, spval, im, modelname, ista, iend
 !- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
       implicit none
 !     
 !     DECLARE VARIABLES.
 !     
-      REAL, DIMENSION(IM,jsta:jend), INTENT(IN)    :: CLDZ, TCLD
-      REAL, DIMENSION(IM,jsta:jend), INTENT(INOUT) :: CEILING
+      REAL, DIMENSION(ista:iend,jsta:jend), INTENT(IN)    :: CLDZ, TCLD
+      REAL, DIMENSION(ista:iend,jsta:jend), INTENT(INOUT) :: CEILING
       integer I,J
 !***************************************************************
 !
 !
       DO J=JSTA,JEND
-        DO I=1,IM
+        DO I=ISTA,IEND
           IF(ABS(TCLD(I,J)-SPVAL) <= SMALL) THEN
             CEILING(I,J)=SPVAL
           ELSE IF(TCLD(I,J) >= 50.) THEN
@@ -504,22 +549,22 @@ SUBROUTINE CALCEILING (CLDZ,TCLD,CEILING)
 !> @author Binbin Zhou NCEP/EMC @date 2005-08-18       
       SUBROUTINE CALFLTCND (CEILING,FLTCND)
       use vrbls2d, only: vis
-      use ctlblk_mod, only: jsta, jend, im, spval
+      use ctlblk_mod, only: jsta, jend, im, spval, ista, iend
 !- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
       implicit none
 !     
 !     DECLARE VARIABLES.
 !     
-      REAL, DIMENSION(IM,jsta:jend), INTENT(IN)    :: CEILING
-      REAL, DIMENSION(IM,jsta:jend), INTENT(INOUT) :: FLTCND
+      REAL, DIMENSION(ista:iend,jsta:jend), INTENT(IN)    :: CEILING
+      REAL, DIMENSION(ista:iend,jsta:jend), INTENT(INOUT) :: FLTCND
       REAL  CEIL,VISI
       integer I,J
 !
 !***************************************************************
 !
 !
       DO J=JSTA,JEND
-        DO I=1,IM
+        DO I=ISTA,IEND
 
         IF(CEILING(I,J)<spval.and.VIS(I,J)<spval)THEN
           CEIL = CEILING(I,J) * 3.2808               !from m -> feet

diff --git a/sorc/ncep_post.fd/AllGETHERV_GSD.f b/sorc/ncep_post.fd/AllGETHERV_GSD.f
@@ -9,8 +9,9 @@ SUBROUTINE AllGETHERV(GRID1)
 !
 ! PROGRAM HISTORY LOG:
 !
+!    21-09-02  Bo Cui - Decompose UPP in X direction               
 
-     use ctlblk_mod, only : im,jm,num_procs,me,jsta,jend,mpi_comm_comp
+     use ctlblk_mod, only : im,jm,num_procs,me,jsta,jend,ista,iend,mpi_comm_comp
 
      implicit none
 
@@ -22,11 +23,11 @@ SUBROUTINE AllGETHERV(GRID1)
 
      REAL GRID1(IM,JM)
      REAL ibufrecv(IM*JM)
-     REAL ibufsend(im*(jend-jsta+1))
+     REAL ibufsend((iend-ista+1)*(jend-jsta+1))
      integer SENDCOUNT,RECVCOUNTS(num_procs),DISPLS(num_procs)
 !
 !     write(*,*) 'check mpi', im,jm,num_procs,me,jsta,jend
-     SENDCOUNT=im*(jend-jsta+1)
+     SENDCOUNT=(iend-ista+1)*(jend-jsta+1)
      call MPI_ALLGATHER(SENDCOUNT, 1, MPI_INTEGER, RECVCOUNTS,1 , &
                 MPI_INTEGER, mpi_comm_comp, ierr)
      DISPLS(1)=0
@@ -40,7 +41,7 @@ SUBROUTINE AllGETHERV(GRID1)
      ij=0
      ibufsend=0.0
      do j=jsta,jend
-        do i=1,IM
+        do i=ista,iend
            ij=ij+1
            ibufsend(ij)=GRID1(i,j)
         enddo