problems using nco with netcdf files created by disdrodb

[saveriogzz]

Hi guys, I'm opening the issue here so we can have a public discussion!

Recently @mschleiss has experienced an issue while trying to use ncdump on the converted netcdf files using the [latest version of] parser_RASPBERRY.py.
The error that Marc was receiving was

NetCDF: HDF error
Location: file ; line 1705

which was pretty uninformative.
The headers file got (marc_headers.txt) were resulting incomplete, with the first line failing being string weather_code_METAR_4678(time) ;

Also my colleague Rob faced a similar issue using Matlab:

I was able to open the "corrupted" files using any method (nco, xarray, ecc.) on my machine.
When I finally ran wither the operator nccopy (without any additional option) or ncks -4 -L 5 notworking.nc good.nc to create new files out of the corrupted ones, both Marc and Rob were finally able to use them.

Please find here PAR001.tar.gz a few example files of both not-working files and working ones.

I'm curious to know what you guys think of this!

Thanks :)

[ghiggi]

Hey @saveriogzz !
The problem could be related to the writing of string arrays.
Currently the encoding dtype is set to str

I would suggest you to proceed as follow:

1. Try to save a netcdf without the variables encoded as str (weather_code_METAR_4678, weather_code_NWS).
2. Check if the problem still occurs
3. If not, it means we need to parse/encode such variable differently.

I am unsure what encoding should be specified ... try to have a look at this

[saveriogzz]

Hey @saveriogzz ! The problem could be related to the writing of string arrays. Currently the encoding dtype is set to str

Hey @ghiggi , I am not sure that I can set an other value than str for these two variables!

[ghiggi]

@saveriogzz did you try 1 and 2 first? The problem does still appear? Or is the string array the problem?
Then we will figure out a workaround ;)

[saveriogzz]

Hey @ghiggi , thanks for following up!
You spotted the error from a distance, good catch!! ;)

I ran the command ncks -x -v weather_code_METAR_4678,weather_code_NWS filethatgaveerror.nc filethatwillbeok.nc in order to remove the variables, and Rob can now open the file with MATLAB, while it was not able to do so before.

[ghiggi]

Ok so we just need to figure out how to encode differently the character array.
I don't have MATLAB on which to test, but I guess that if we make it work with ncdump then the problem will be resolved.
Could you start by looking at the encoding difference before applying nco and after nco (as this solved the issue right?)

[saveriogzz]

Hey,
I ran a unix diff command against the two files (original after disdrodb processing and itself after nccopy)
diff <(ncdump CABAUW_sPAR001_20211101.nc) <(ncdump CABAUW_sPAR001_20211101.workingcopy.nc)
but it doesn't highlight any difference in the encoding of the variables. They both seem to be strings! What do you think?

filestotest.zip

[ghiggi]

This bug has been solved by the PR #68