Debug mode crash in w3src4md.F90

[JessicaMeixner-NOAA]

Describe the bug
When running one of our coupled model test cases, with PR #1343 which is basically adding commit 3952826 to the dev/ufs-weather-model branch.

I get the following error:

forrtl: severe (408): fort: (3): Subscript #1 of the array A has value 0 which is less than the lower bound of 1
Line of code this is in (for the develop branch) is https://github.com/NOAA-EMC/WW3/blob/develop/model/src/w3src4md.F90#L2277 in the W3SDS4 subroutine :

BTH(IS)=DOT_PRODUCT(SATWEIGHTS(:,ITH), A(IS0+SATINDICES(:,ITH)) )**FACSAT

When I write out values before this crash

            write(*,*) 'IS0', IS0
            write(*,*) 'IS', IS
            write(*,*) 'ITH', ITH
            write(*,*) 'IK', IK
            write(*,*) 'SATINDICES(:,ITH)', SATINDICES(:,ITH)

I get the following output:

277:  IS0           0
277:  IS           1
277:  ITH           1
277:  IK           1
277:  SATINDICES(:,ITH)           0           0           0           0           0
277:            0           0           0           0           0           0
277:            0           0           0           0           0           0
277:            0           0           0           0           0           0
277:            0           0           0           0           0           0
277:            0           0           0           0           0           0

I'm thinking we should add a ISO+1 here like is done in these lines of code: https://github.com/NOAA-EMC/WW3/blob/develop/model/src/w3src4md.F90#L2269C9-L2269C37:

        BTH(IS0+1)=0.
        ASUM = SUM(A(IS0+1:IS0+NTH))

Or something should be different in the SATINDICES(:,ITH) variable?

@mickaelaccensi @ardhuin - Any thoughts or comments?

To Reproduce
Unfortunately, I've only run into this error in the coupled model, so it's hard for others to reproduce but it's the ufs-waether-model test cpld_debug_pdlib_p8 with intel see PR #1343

Expected behavior
Should not crash in debug mode.

[JessicaMeixner-NOAA]

Update: I tried:
BTH(IS)=DOT_PRODUCT(SATWEIGHTS(:,ITH), A(IS0+1+SATINDICES(:,ITH)) )**FACSAT

But then I fail later in the run because now I'm going 1 over the upper bound:

201: forrtl: severe (408): fort: (2): Subscript #1 of the array A has value 1189 which is greater than the upper bound of 1188

279:  IS0         216
279:  IS         232
279:  ITH          16
279:  IK           7
279:  SATINDICES(:,ITH)           8           9          10          11          12
279:           13          14          15          16          17          18
279:           19          20          21          22          23          24
279:            1           1           1           1           1           1
279:            1           1           1           1           1           1
279:            1           1           1           1           1           1

[mickaelaccensi]

I my opinion the issue is more when SATINDICES is computed. It should never be set at 0.
SATINDICES(I_INT-(ITH-SDSNTH)+1,ITH)=J_INT

@ardhuin what do you think about this issue ?

[JessicaMeixner-NOAA]

@mickaelaccensi - I appear to be getting around this issue by creating new mod_defs which have non-zero SATINDICES

That being said, based on looking at the code and some comments here, I'm wondering if we should change this line: https://github.com/NOAA-EMC/WW3/blob/develop/model/src/w3gdatmd.F90#L2098
from:
MPARS(IMOD)%SRCPS%SATINDICES(:,:)=0.
to

MPARS(IMOD)%SRCPS%SATINDICES(:,:)=1.
as another safe-guard.

Thoughts?