DAP4: url with .dap.csv?... does not display tabular data correctly on the browser

[Mikejmnez]

Description of problem

I tried to visualize data on the browser in tabular form. With DAP2 everything seems to work as expected, but with DAP4 things don't go as planned.

What I did

DAP2 works

Pasting this url: http://test.opendap.org/opendap/data/nc/sst.mnmean.nc.gz.ascii?sst[3][4][5] on a browser shows:

Dataset: sst.mnmean.nc
sst.lon, 10
sst.sst[sst.time=19813][sst.lat=80], -51

DAP4 does not

If I paste the dap4 url: http://test.opendap.org/opendap/data/nc/sst.mnmean.nc.gz.dap.csv?dap4.ce=/sst[3][4][5] onto a browser it downloads the sst.mnmean.nc.gz.csv file. Looking at its contents with a simple text editor

Dataset: sst.mnmean.nc
/sst[0][0], 

With pandas:

That is pretty confusing.

If I set the url instead as http://test.opendap.org/opendap/data/nc/sst.mnmean.nc.gz.dap.csv?dap4.ce=/sst[3:4][4:5][5:6], it downloads a csv with the sst.mnmean.nc.gz.csv file with values now:

Dataset: sst.mnmean.nc
/sst[0][0], -51, -123
/sst[0][1], 175, 143
/sst[1][0], 50, -18
/sst[1][1], 242, 206

Expected behavior?

With DAP4 I was expected to see the browser data in tabular form as in DAP2 case. Instead:
a) Downloads data rather than visualize on browser.
b) When inspecting the contents of the csv file, the data seems incorrect.

[ndp-opendap]

This is a browser thing and not a bug.

For the DAP4 csv request:

curl -i http://test.opendap.org/opendap/data/nc/fnoc1.nc.dap.csv

The server returns the following response headers:

Content-Description: text/csv
Content-Disposition:  attachment; filename="fnoc1.nc.csv"
Content-Type: text/csv

And the browser determines that the content should be downloaded and not displayed.

The DAP2 ascii response:

curl -i http://test.opendap.org/opendap/data/nc/fnoc1.nc.ascii

Returns the response headers:

Content-Description: dods_ascii
Content-Type: text/plain

Which browsers are happy to display.

The DAP2 ascii response and it headers are "historic/legacy" and do not really conform to the semantics of the modern web.

I believe that the DAP4 response headers are correct.

Additionally it's important to remember that the DAP4 service supports HTTP 1.1 client/server content negotiation which can make things confusing when using a browser to interact with the service. See section 4 of DAP4 Volume 2.

For example when browser is used to deference the URL: http://test.opendap.org/opendap/data/nc/fnoc1.nc.dmr the server will return the Data Request Form page because the browser tells the service "I'll take anything, but I prefer HTML" Since the Data Request Form page is the HTML representation of the DMR that is what its returned. If cURL is used to deference the same URL the service will return the XML encoded DMR document. This is because cURL, by default, does not include an Accept request header. The client can specify a different response encoding either by including an Accept header in the request, or by adding an additional suffix to the request URL. The URL http://test.opendap.org/opendap/data/nc/fnoc1.nc.dmr.xml specifies the XML encoded version of the DMR and dereferencing this URL in a browser will retrieve the XML encoded DMR from the service.

If there is an issue with the content (which there surely is) we should discuss, as "CSV" might nominally be expected to be something more "tabular" rather then the "discrete" form which is returned currently.

References:

• https://github.com/OPENDAP/dap4-specification/blob/main/03_spec-volume-2.md
• https://github.com/OPENDAP/dap4-specification/blob/main/04_extension-csv.md

I'll be back next week and we can discuss, or see @jgallagher59701 for more context.

[Mikejmnez]

Thanks @ndp-opendap !