CDAWeb irregularities?

[sandyfreelance]

Doing a query on CDAWeb yielded good CSV data (though hapiplot could not plot it; Time lacked the trailing Z). However, the same query in 'binary' format was unparseable. In ways that confuse us.

Making it a bit messier, CDAWeb apparently defaults to 'binary' if you don't tell it 'format=csv' in **opts. So this odd data is the default it delivered.

So this is a CDAWeb issue, a HAPI 2.0 version issue, or a hapiclient issue?

Example:

server='https://cdaweb.gsfc.nasa.gov/hapi'
dataset='ELA_L1_EPDEF'
parameters='Time'
start='2022-08-10T03:23:03Z'
stop='2022-08-10T03:38:03Z'
from hapiclient import hapi

CSV:

opts = {'format':'cvs','logging':True,'usecache':False}
data, meta = hapi(server,dataset,parameters,start,stop)
data[0:4]
array([(b'2022-08-10T03:36:00.5542',), (b'2022-08-10T03:36:00.7315',),
(b'2022-08-10T03:36:00.9089',), (b'2022-08-10T03:36:01.0862',)],
dtype=[('Time', 'S24')])

Binary:

opts = {'format':'binary','logging':True,'usecache':False}
data, meta = hapi(server,dataset,parameters,start,stop,**opts)
data[0:4]
array([(b'2022-08-10T03:36:00.5542',), (b'33000Z2022-08-10T03:36:0',),
(b'0.731577000Z2022-08-10T0',), (b'3:36:00.908921000Z2022-0',)],
dtype=[('Time', 'S24')])

[rweigel]

The time length looks correct in the csv b/c the csv parser seems to be removing the trailing zeros. The raw csv shows a time length that is 30 and not 24 as claimed in the metadata: https://hapi-server.org/servers/#server=CDAWeb&dataset=ELA_L1_EPDEF&parameters=Time&start=2018-12-14T20:31:00Z&stop=2018-12-16T20:31:00.000Z&return=data&format=csv&style=noheader This is the same issue as hapi-server/server-issues#2 and client handling has been discussed at hapi-server/client-python#57

…

On Mon, Aug 15, 2022 at 5:00 PM Sandy Antunes ***@***.***> wrote: Doing a query on CDAWeb yielded good CSV data (though hapiplot could not plot it; Time lacked the trailing Z). However, the same query in 'binary' format was unparseable. In ways that confuse us. So this is a CDAWeb issue, a HAPI 2.0 version issue, or a hapiclient issue? *Example:* server='https://cdaweb.gsfc.nasa.gov/hapi' dataset='ELA_L1_EPDEF' parameters='Time' start='2022-08-10T03:23:03Z' stop='2022-08-10T03:38:03Z' from hapiclient import hapi *CSV:* opts = {'format':'cvs','logging':True,'usecache':False} data, meta = hapi(server,dataset,parameters,start,stop) data[0:4] array([(b'2022-08-10T03:36:00.5542',), (b'2022-08-10T03:36:00.7315',), (b'2022-08-10T03:36:00.9089',), (b'2022-08-10T03:36:01.0862',)], dtype=[('Time', 'S24')]) *Binary:* opts = {'format':'binary','logging':True,'usecache':False} data, meta = hapi(server,dataset,parameters,start,stop,**opts) data[0:4] array([(b'2022-08-10T03:36:00.5542',), (b'33000Z2022-08-10T03:36:0',), (b'0.731577000Z2022-08-10T0',), (b'3:36:00.908921000Z2022-0',)], dtype=[('Time', 'S24')]) — Reply to this email directly, view it on GitHub <#147>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAUQ57U3BPHLYZ34YPT466LVZKVXRANCNFSM56TPESUQ> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[sandyfreelance]

So who can fix? CDAWeb shifting to (a) csv as the default instead of binary and (b) accurate length in metadata would seem to fix.

[VoyagerPWS]

It's worth double-checking that nothing is cavalierly disposing of the micro- and nano-second portions. We don't want any "epoch time" assumptions bleeding into "tt2000" handling.

[rweigel]

I spoke with the CDAWeb group yesterday and they are looking into a fix; further discussion of this issue should be at hapi-server/server-issues#2

The Python client does not do extensive validation on the data returned by a HAPI server. Servers should put out valid streams and there is an extensive validation service that can be used to verify a server is working properly. (Putting extensive validation into all of the clients seems like an unnecessary duplication of effort.)

[rweigel]

For future reference, I suggest that server and client issues are posted in the server and client repositories as opposed to this general discussions forum, where they may be more difficult to find. There is also a server-issues repository where we post issues related to a HAPI server.

[sandyfreelance]

Worth discussing where. I favor using Discussions for non-software-related questions, since we made it as a general entry/discussion point for new users (so they don't have to figure out which software package to reference). I can see the 'server' as maybe useful for discussing server issues which are not dependent on the choice of client program, but 'server' seems to be focused on presenting the list of servers. So if we want that to be the general server uptime & quality category, we should indicate it in that section's README.md, and update the overall README.md to point people there.