error: unpack requires a buffer of 4 bytes

[FarnoodF]

I get the following error when I try to read one of my .xdf files:

~/lib/python3.7/site-packages/pyxdf/pyxdf.py in load_xdf(filename, select_streams, on_chunk, synchronize_clocks, handle_clock_resets, dejitter_timestamps, jitter_break_threshold_seconds, jitter_break_threshold_samples, clock_reset_threshold_seconds, clock_reset_threshold_stds, clock_reset_threshold_offset_seconds, clock_reset_threshold_offset_stds, winsor_threshold, verbose)
    258             log_str = ' Read tag: {} at {} bytes, length={}'.format(tag, f.tell(), chunklen)
    259             if tag in [2, 3, 4, 6]:
--> 260                 StreamId = struct.unpack('<I', f.read(4))[0]
    261                 log_str += ', StreamId={}'.format(StreamId)
    262             else:

error: unpack requires a buffer of 4 bytes

Although, I can easily read the same file using EEGLab in Matlab. Do you have any recommendations on how I can solve this?
Thanks.

[cboulay]

That suggests that you've reached the end of the file buffer, but if True then it should have followed a different code path to break out of the loop.
What logs are printed before this point?
Can you share the datafile?

[FarnoodF]

@cboulay , Thank you for the quick answer. There are no other lines prior to this error.
Here's a link to the file.

[cbrnr]

I tried loading the file with libxdf using the simple loader executable from xdf-modules/libxdf#26, but I also get an error:

Invalid variable-length integer length (0) encountered.
Invalid variable-length integer length (0) encountered.
it took 76260 clicks (0.07626 seconds) reading XDF data
Stream: EEG 18 channels 45725 samples

This indicates that the file is not valid XDF. I wonder why it works in MATLAB though (can't test because I don't have MATLAB).

[FarnoodF]

I just checked it again with MATLAB and it raises this warning:

Warning: Specified key type does not match the type expected for this container. 
> In load_xdf (line 332)
  In eeg_load_xdf (line 35)
  In pop_loadxdf (line 112) 

But it still outputs the data inside the file without any errors. It would be nice if we could get rid of this error in pyxdf.
Note: Matlab uses eegplugin_xdfimport to import xdf.

[cboulay]

Do you know if your eegplugin_xdfimport is the latest version? Could you try comparing it to https://github.com/xdf-modules/xdf-EEGLAB? (Note that uses xdf-Matlab as a submodule)

I'm not about to install and try to remember how to use EEGLAB, but if your EEGLAB importer is using the same version as general xdf-Matlab then I can test the file on my local Matlab install.

[FarnoodF]

@cboulay I checked it and it is the latest version. It is using exactly the same code as in https://github.com/xdf-modules/xdf-EEGLAB

[agricolab]

According to XDFBrowser, there is a stream packet after the footer. Weirdly, enough, for a different stream which apparently does not have a Stream Header. This would explain why at least for Stream 1, it can be loaded.

[cboulay]

@FarnoodF How did you record this datafile? Did you use LabRecorder?

Can you think of anything that was different about this file that might lead to a ghost stream?

While we COULD modify pyxdf to not fail when it encounters a 0-byte stream with no header, really the better solution is to figure out how this happened in the first place and prevent it from happening again.

[agricolab]

So there is only one valid stream inside, to be more specific, EEG with 18 channels, and 500 Hz sampling rate. Is this as expected, or did you also plan to record more than this stream?

[FarnoodF]

Thank you guys for your time.
I recorded this using a cEEGrid and SMARTING. I just intended to record an 18-channel stream at 500Hz, that's all.
I have recorded over 1000 .xdf files and I have 5 files with the same problem. To my best of knowledge, there was no change in the way I did those experiments.
If the data is present in one of the streams, It would be more convenient to get a warning, instead of an error.

[agricolab]

I understand that you recorded the stream with SMARTING streamer, and not with LabRecorder? If that's the case, you might want to inform their developers about the issue.