Appending timestamps?

[tjr1]

Hi NWB community,

I'm trying to write timestamped data iteratively by modifying the example found here:
(https://neurodatawithoutborders.github.io/matnwb/tutorials/html/dataPipe.html#13)

I can add the timestamps when writing the first chunk of data (the portion before export() ), but how do I iteratively write timestamps for chunks 2 through 4?

I imagined it would be something like
nwb.acquisition.get('time_series').timestamps.append(time_vec)

But nwb.acquisition.get('time_series').timestamps doesn't have an append() function of course.

Thanks for pointing me the right direction,
Tom

[oruebel]

@bendichter @lawrence-mbf

[bendichter]

@tjr1

nwb.acquisition.get('time_series').timestamps should have the .append method, but you first need to write and then read the NWB file. Have you done that? Could you provide some code showing what you are trying to do?

[tjr1]

Thank you Ben,

Below is how I've tried to modify your example.

Gratefully,
Tom

`
dataPart1 = randi(250, 10000, 1); % "load" 1/4 of the entire dataset
fullDataSize = [40000 1]; % this is the size of the TOTAL dataset

% create an nwb structure with required fields
nwb=NwbFile( ...
'session_start_time', '2020-01-01 00:00:00', ...
'identifier', 'ident1', ...
'session_description', 'DataPipeTutorial');

% sample rate and timestamp vector for first portion of data
fs = 250; % Hz
timestamp_vec = (1:10000)./fs; % seconds

% compress the data
fData_use = types.untyped.DataPipe( ...
'data', dataPart1, ...
'maxSize', fullDataSize, ...
'axis', 1);

%Set the compressed data as a time series

fdataNWB = types.core.TimeSeries( ...
'timestamps',timestamp_vec, ...
'timestamps_unit','seconds', ...
'data', fData_use, ...
'data_unit', 'mV');

nwb.acquisition.set('time_series', fdataNWB);

nwbExport(nwb, 'DataPipeTutorial_iterate.nwb');

nwb = nwbRead('DataPipeTutorial_iterate.nwb'); %load the nwb file with partial data

% check that the timestamps and data load and can be plotted
plot(nwb.acquisition.get('time_series').timestamps.load, nwb.acquisition.get('time_series').data.load) %

% load each of the remaining 1/4ths of the large dataset
for i = 2:4 % iterating through parts of data
dataPart_i=randi(250, 10000, 1); % faked data chunk as if it was loaded
nwb.acquisition.get('time_series').data.append(dataPart_i); % append the loaded data

% creating timestamp vector
fs = 250; % Hz
timestamp_vec = ((i-1)*10000+(1:10000))./fs;

% the below line doesn't work
nwb.acquisition.get('time_series').timestamps.append(timestamp_vec); 
% Errors on the above line with error message:
%     
% Unrecognized method, property, or field 'append' for class 'types.untyped.DataStub'.
% 
% Error in indexing (line 369)
%                 B = builtin('subsref', obj, S);
% 
% Error in iterative_timestamp_test (line 52)
%     nwb.acquisition.get('time_series').timestamps.append(timestamp_vec);

end
`

[lawrence-mbf]

Hi @tjr1 you will also need to configure timestamp_vec as a DataPipe as well following @bendichter 's instructions

[bendichter]

Yes, something like this:

replace

timestamp_vec = (1:10000)./fs; % seconds

with:

timestamp_vec = types.untyped.DataPipe( ...
'data', (1:10000)./fs; % seconds, ...
'maxSize', fullDataSize, ...
'axis', 1);

However, I would add that if you timestamps are truly regular, you are better off defining your TimeSeries with a sampling rate and starting time like this:

fdataNWB = types.core.TimeSeries( ...
  'data', fData_use, ...
  'data_unit', 'mV',...
  'starting_time', 0.0, ...
  'starting_time_rate', fs, ...
);

which would avoid this problem altogether.

[tjr1]

Thank you Ben and Lawrence,

That fixed it! Thank you.

I was thinking I'd use timestamps because I'm converting week long stereo EEG recordings that drift in time and have occasional discontinuities. I'm not sure what the best practice is really. I could fill in short discontinuities with NaNs or start a new TimeSeries after long discontinuities, but that seems cumbersome. And dealing with drift could be handled by starting new files, new TimeSeries, or timestamping. Is there are best practice?

[bendichter]

If your data is more-or-less continuous, I think using timestamps is the right approach