The routing model: IndexError: only integers, slices (:), ellipsis (...), numpy.newaxis (None) and integer or boolean arrays are valid indices

[Tarikbouram]

Hello
I am trying to use the routing model of the tutorial, and I encounter this indexerror, I tried several solution but it didn't work.

python rout_vic.py ../data/input/Nyando_UHFile.csv ../data/input/gis/Nyando_gauge_fraction.tif ../data/output/runoff_2005.nc ../data/output/base_2005.nc ../data/output/rout_out.csv 20050101 20131231 False
Traceback (most recent call last):
  File "rout_vic.py", line 290, in <module>
    main()
  File "rout_vic.py", line 284, in main
    sys.argv[6],sys.argv[7],sys.argv[8])
  File "rout_vic.py", line 218, in rout_vic
    uh = make_uh(xmask_val,uh_box,float(diffusion),float(velocity))
  File "rout_vic.py", line 109, in make_uh
    uh_daily[tt-1] = uh_daily[tt-1]+fr[t,1]
IndexError: only integers, slices (`:`), ellipsis (`...`), numpy.newaxis (`None`) and integer or boolean arrays are valid indices

Is there any known way to fix it ?

Thanks

[Tarikbouram]

one of the solutions I tried is to wrap [tt-1],[tt-1] and[t,1] in int(.....) :
uh_daily[int(tt-1)] = uh_daily[int(tt-1)]+fr[int(t),1]
but this has generated a new Indexerror:

Traceback (most recent call last):
  File "rout_vic.py", line 290, in <module>
    main()
  File "rout_vic.py", line 284, in main
    sys.argv[6],sys.argv[7],sys.argv[8])
  File "rout_vic.py", line 221, in rout_vic
    runoff = (rovar[t1off:t2off,idx[0][i],idx[1][i]]*factor)
  File "netCDF4/_netCDF4.pyx", line 3961, in netCDF4._netCDF4.Variable.__getitem__
  File "netCDF4/_netCDF4.pyx", line 4796, in netCDF4._netCDF4.Variable._get
IndexError

[KMarkert]

I had a similar problem to your original post. Basically, the script was indexing using a float datatype, which is not allowed anymore. I fixed the issue in script where the index is forced to be an integer datatype. Please get the new script and try to rerun the process. Please let me know if you get it to work.

[Tarikbouram]

Thank you for your answer and update , this error message no longer appears, but now there is a new issues:

python rout_vic.py ../data/input/Nyando_UHFile.csv ../data/input/gis/Nyando_gauge_fraction.tif ../data/output/runoff_2005.nc ../data/output/base_2005.nc ../data/output/rout_out.csv 20050101 20131231 False
Traceback (most recent call last):
  File "rout_vic.py", line 290, in <module>
    main()
  File "rout_vic.py", line 284, in main
    sys.argv[6],sys.argv[7],sys.argv[8])
  File "rout_vic.py", line 221, in rout_vic
    runoff = (rovar[t1off:t2off,idx[0][i],idx[1][i]]*factor)
  File "netCDF4/_netCDF4.pyx", line 3961, in netCDF4._netCDF4.Variable.__getitem__
  File "netCDF4/_netCDF4.pyx", line 4796, in netCDF4._netCDF4.Variable._get
IndexError

Do you have any idea what caused this problem and how to solve it. thank you again.

[KMarkert]

It seems that the model is trying to grab runoff data from a time period that is not available in the NetCDF data. Can you please check that the NetCDF data has the time period 2005-01-01 to 2013-12-31 as you are specific to run the routing model for?

[Tarikbouram]

I just rebuilt the netCDFS runoff and baseflow data from 2005 to 2013 just to be sure, and it's still the same problem. would you like me to send you the file that contains the netCDFS data?

[KMarkert]

Yes, please send both the runoff and baseflow NetCDF data.

[Tarikbouram]

Here I sent you the netCDFS runoff and baseflow data. Thanks

netCDFS.zip

[KMarkert]

This is interesting...I am not even to open the NetCDF data on my system which makes me believe it might be an issue with your NetCDF library installation to where it is not writing the data properly. Can you please check your configuration for the netCDF software on your system? You can do this by running nc-config --all in your terminal and post the output.

[Tarikbouram]

indeed it is very strange, on my machine I manage to open this netcdf data, and analyze them on panoply.
Here is the output of the command nc-config --all:

This netCDF 4.4.1.1 has been built with the following features: 

  --cc        -> /opt/rh/devtoolset-2/root/usr/bin/cc
  --cflags    -> -I/home/lgee/miniconda3/include
  --libs      -> -L/home/lgee/miniconda3//home/lgee/miniconda3/lib -lnetcdf -lmfhdf -ldf -lhdf5_hl -lhdf5 -lrt -lpthread -lz -ldl -lm -lcurl

  --has-c++   -> no
  --cxx       -> 

  --has-c++4  -> no
  --cxx4      -> 

  --has-fortran-> no
  --has-dap   -> yes
  --has-nc2   -> yes
  --has-nc4   -> yes
  --has-hdf5  -> yes
  --has-hdf4  -> yes
  --has-logging-> yes
  --has-pnetcdf-> no
  --has-szlib -> no

  --prefix    -> /home/lgee/miniconda3
  --includedir-> /home/lgee/miniconda3/include
  --libdir    -> /home/lgee/miniconda3//home/lgee/miniconda3/lib
  --version   -> netCDF 4.4.1.1

[KMarkert]

It seems there has been some bugs in the version of the netCDF library that you have installed were the information may not be written correctly. Can you please try updating you netCDF package in anaconda using the following command: conda update -c conda-forge libnetcdf and try to make the netCDF files again.

[Tarikbouram]

I updated my netCDF package, as you can see below:

This netCDF 4.6.1 has been built with the following features: 

  --cc        -> /opt/rh/devtoolset-2/root/usr/bin/cc
  --cflags    -> -I/home/lgee/miniconda3/include
  --libs      -> -L/home/lgee/miniconda3//home/lgee/miniconda3/lib -lnetcdf -lmfhdf -ldf -lhdf5_hl -lhdf5 -lrt -lpthread -lz -ldl -lm -lcurl

  --has-c++   -> no
  --cxx       -> 

  --has-c++4  -> no
  --cxx4      -> 

  --has-fortran-> no
  --has-dap   -> yes
  --has-dap2  -> yes
  --has-dap4  -> yes
  --has-nc2   -> yes
  --has-nc4   -> yes
  --has-hdf5  -> yes
  --has-hdf4  -> yes
  --has-logging-> yes
  --has-pnetcdf-> no
  --has-szlib -> no
  --has-cdf5 -> yes
  --has-parallel-> no

  --prefix    -> /home/lgee/miniconda3
  --includedir-> /home/lgee/miniconda3/include
  --libdir    -> /home/lgee/miniconda3//home/lgee/miniconda3/lib
  --version   -> netCDF 4.6.1

but unfortunately I still have the same issue. would you like me to send you the feeds file? maybe the problem has a connection with the fluxes. In fact when opening the netCDF runoff file the values range from 00 to 1.0e-25. which I know is not correct but even by changing the maximum value to for example 2.2, the runoff hardly varies.

[optyrox12]

Hi everyone, I just got to this stage of processing after making some changes to the flux2nc.py script, like changing "string" to "str", adding some "int' before "input', here attached the script:

 try:
        for f in file_list:
            lat_t.append(float(str.split(f, "_")[1]))
            lon_t.append(float(str.split(f, "_")[2]))
    except ValueError:
        raise ValueError('Input path contains files that are not flux files')

`````` varini = int(input('Choose output (1 a 6)>'))

`
if the date information is not set get it from user
if start_year == None:
# for what date?
start_year = int(input("Enter start year:"))
if end_year == None:
end_year = int(input("End year:"))

`` # get lat & lon and it's index
        latitude = float(str.split(f, sep="_")[1])
        longitude = float(str.split(f, sep="_")[2])
        lat_id = lat.index(latitude)
        lon_id = lon.index(longitude)
    
        c = c -1
        
        infile = open(dirin+'/'+f, "r")
        lixo = infile.readlines()
        infile.close()
        dado = []
    
        for l in lixo:
            if int(str.split(l, sep="\t")[0]) in range(inidate.year, enddate.year+1):
                dado.append(float(str.split(l, sep="\t")[var]))
```
after the feed generation, I have this first syntax error:
```
```
 File "rout_vic.py", line 26, in <module>
    from dd_conversions import *
  File "/home/lgee/Documents/servir-vic-training-master/scripts/dd_conversions.py", line 106
    print 'Output meters: {0}\nOutput decimal degrees: {1}'.format(outMeters,outDD)
                                                          ^
SyntaxError: invalid syntax

```
```and it is only after his correction in the form:
`print ('Output meters: {0}\nOutput decimal degrees: {1}').format(outMeters,outDD)`

that I get the error that found by @Tarikbouram:
```
Traceback (most recent call last):
  File "rout_vic.py", line 290, in <module>
    main()
  File "rout_vic.py", line 284, in main
    sys.argv[6],sys.argv[7],sys.argv[8])
  File "rout_vic.py", line 221, in rout_vic
    runoff = (rovar[t1off:t2off,idx[0][i],idx[1][i]]*factor)
  File "netCDF4/_netCDF4.pyx", line 3961, in netCDF4._netCDF4.Variable.__getitem__
  File "netCDF4/_netCDF4.pyx", line 4796, in netCDF4._netCDF4.Variable._get
IndexError

```

[KMarkert]

So, there was an error with the initial model setup where the underlying grid had an extra row and column but the way the model outputs the data was omitting that extra row and column. Basically, this led to mismatched dimensions between the model output and expected grid. Any indexing on the edges of the grid while routing was throwing Index Errors.

I have updated the input files for the grids on the repo. Users will have to download the new Nyando_basin shapefile, setup the model input data from the beginning, and rerun the model. I apologize for any inconvenience.

[KMarkert]

@optyrox12 thanks for your suggestions on the code. It seems like this issue was a Python 3 compatibility issue which should have been fixed with b453fe4.

Considering this issue should be fixed and the time since latest comment, I will go ahead and close. Please reopen if this is still an issue.