nccopy variable subsetting via OPeNDAP and global variables

[dblodgett-usgs]

OSX 10.12
netCDF library version 4.4.1.1 of Feb 23 2017 13:31:18 (Homebrewed)

When I do: $ nccopy https://cida.usgs.gov/thredds/dodsC/new_gmo?longitude[0:1:461],latitude[0:1:221],time[0:1:22644] out.nc

I get:

$ ncdump -h out.nc 
netcdf out {
dimensions:
	latitude = 222 ;
	longitude = 462 ;
	time = 22645 ;
variables:
	float longitude(longitude) ;
		longitude:long_name = "Longitude" ;
		longitude:units = "degrees_east" ;
		longitude:standard_name = "longitude" ;
		longitude:axis = "X" ;
		longitude:bounds = "longitude_bnds" ;
	float latitude(latitude) ;
		latitude:long_name = "Latitude" ;
		latitude:units = "degrees_north" ;
		latitude:standard_name = "latitude" ;
		latitude:axis = "Y" ;
		latitude:bounds = "latitude_bnds" ;
	double time(time) ;
		time:units = "days since 1940-01-01 00:00:00" ;
		time:calendar = "standard" ;

// global attributes: 
// (expected global attributes removed to be concise)
		:pr.units = "mm/d" ;
		:pr._FillValue = 1.e+20f ;
		:pr.long_name = "Precipitation" ;
		:tas.units = "C" ;
		:tas._FillValue = 1.e+20f ;
		:tas.long_name = "Average Temperature" ;
		:tasmax.units = "C" ;
		:tasmax._FillValue = 1.e+20f ;
		:tasmax.long_name = "Maximum Temperature" ;
		:tasmin.units = "C" ;
		:tasmin._FillValue = 1.e+20f ;
		:tasmin.long_name = "Minimum Temperature" ;
		:wind.units = "m/s" ;
		:wind.axis = "TYX" ;
		:wind.missing_value = 1.00000002004088e+20 ;
		:wind.associate = "time latitude longitude" ;
		:wind.long_name = "Wind Speed" ;
}

Should the variables and attributes that were not requested be showing up as global attributes? Adding the -v or -V options with those variables listed doesn't affect the output.

[WardF]

Hi, sorry for the delay responding. The output you're seeing is correct. From the ncdump man page:

-h   Show only the header information in the output, that is, output only the  declarations
     for the dimensions, variables, attributes, groups, and user-defined types of the input
     file, but no data values for any variables.  The output is identical to using  the  -c
     option  except that the values of coordinate variables are not included.  (At most one
     of -c or -h options may be present.)

I hope this answers your question!

[dblodgett-usgs]

Sorry, I wasn't clear the problem I'm pointing out. The nccopy command should not have included the global attributes I included above.

I should have said "Should the variables and attributes that were not requested in the nccopy command be showing up as global attributes? Adding the -v or -V options with those variables listed doesn't affect the output."

Have a look at the OPeNDAP endpoint: https://cida.usgs.gov/thredds/dodsC/new_gmo.info

The global attributes I get back from nccopy are actual variables and the variables' attributes that I did not ask for.

[dblodgett-usgs]

@WardF - Not sure if you can re-open this... should I create a new ticket?

[WardF]

My mistake thanks for clarifying! I'll reopen and revisit in the morning.

[WardF]

@dblodgett-usgs No need, I reopened this one. :).

[WardF]

@DennisHeimbigner Can you weigh in on this? I missed the subsetting that I now see in the nccopy request. I don't know if this is expected behavior without doing a bit of research and I feel like you might know if this is how nccopy/subsetting is supposed to work off the top of your head :).

[dblodgett-usgs]

In the past, the request would just subset the actual global variables and the variables requested using OPeNDAP subsetting syntax. It is now adding the other variable (that I don't want) names and attributes as global attributes.

[DennisHeimbigner]

ok, Let me look at it. =Dennis

…

On 3/9/2017 9:45 AM, Ward Fisher wrote: @DennisHeimbigner <https://github.com/DennisHeimbigner> Can you weigh in on this? I missed the subsetting that I now see in the nccopy request. I don't know if this is expected behavior without doing a bit of research and I feel like you might know if this is how nccopy/subsetting is supposed to work off the top of your head :). — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#373 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AA3P28k7BcX3J3LoykxLGBLhlDNbOvjUks5rkCylgaJpZM4MRjcO>.

[DennisHeimbigner]

If you stick this into your browser
https://cida.usgs.gov/thredds/dodsC/new_gmo.das?longitude[0:1:461],latitude[0:1:221],time[0:1:22644]
you will see that the server is sending back attributes for non-requested variables like 'pr'.
The netcdf-c opendap code converts such "unattached" attributes to global ones, which is why you
are seeing those extra global attributes.

So, the real issue is: why is the server sending back those extra attributes. Since this is apparently
a thredds server issue I need to transfer this issue over there. I will do so and add a comment here
pointing to the new issue.

[DennisHeimbigner]

I have created issue Unidata/thredds#774
on the thredds repo to continue this problem.

[dblodgett-usgs]

Interesting. This is definitely different behavior than we used to get. This is a technique we've been using for a long time. Not sure if THREDDS or nccopy changed, but I guess you guys see what's up and can advise if I should change how I use nccopy or if it's an issue that should be fixed in THREDDS. Thanks!

[DennisHeimbigner]

I am trying to fix it, although the legacy opendap code is making this hard.
Looking at the code, I would have guessed that it has always been this way.

[WardF]

@dblodgett-usgs Can you say with any certainty what old version you used that had different behavior? If not no worries, I can step back through versions, but it would take the guess work out of figuring out what changed in netCDF and/or is this netCDF or Thredds.

[dblodgett-usgs]

We have nccopy on one of our cent OS machines... netCDF library version 4.3.3.1 of Apr 15 2015 15:18:07 $ I do NOT get the extra global variables with that version.

[DennisHeimbigner]

I suppose it is possible that at some point, the netcdf-c code suppressed those
extra attributes. I will check.

[dblodgett-usgs]

Exactly the same request, yeah. If that old OPeNDAP code hasn't changed, maybe the C code was smart and worked around this before? Variable subsetting was added to nccopy recently right?

…

On March 9, 2017 9:50:21 PM CST, DennisHeimbigner ***@***.***> wrote: As far as I can tell, the problem should be on the server. Are you accessing the same URL? If so, I am baffled. -- You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub: #373 (comment)

-- Sent from my Android device with K-9 Mail. Please excuse my brevity.

[DennisHeimbigner]

Turns out this "feature" was added in response to a user issue:
#152
and was implemented by pull request:
#164

Looks like I need to modify the added code to be a little smarter.

[DennisHeimbigner]

Fixed by pull request #376