Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

unpack requires a buffer 4 bytes #272

Closed
mdevore300 opened this issue May 31, 2022 · 13 comments · Fixed by #273
Closed

unpack requires a buffer 4 bytes #272

mdevore300 opened this issue May 31, 2022 · 13 comments · Fixed by #273
Labels
Bug

Comments

@mdevore300
Copy link

My code using TDMS usually works fine. This file got interrupted and restarted with a different index number. But when you TDMSFile.open i get the unpack error

[nptdms.reader WARNING] Last segment of file has unknown size, will attempt to read to the end of the file
Traceback (most recent call last):

File "P:\Data by Fiscal Year\2022\ER\untitled0.py", line 31, in
tdms_file = TdmsFile.open("%s_0000.tdms" % name, raw_timestamps = True)

File "C:\Users\284210\Anaconda3\lib\site-packages\nptdms-1.3.1-py3.8.egg\nptdms\tdms.py", line 87, in open
return TdmsFile(

File "C:\Users\284210\Anaconda3\lib\site-packages\nptdms-1.3.1-py3.8.egg\nptdms\tdms.py", line 130, in init
self._read_file(self._reader, read_metadata_only, keep_open)

File "C:\Users\284210\Anaconda3\lib\site-packages\nptdms-1.3.1-py3.8.egg\nptdms\tdms.py", line 226, in _read_file
tdms_reader.read_metadata(require_segment_indexes=keep_open)

File "C:\Users\284210\Anaconda3\lib\site-packages\nptdms-1.3.1-py3.8.egg\nptdms\reader.py", line 87, in read_metadata
segment, properties = self._read_segment_metadata(

File "C:\Users\284210\Anaconda3\lib\site-packages\nptdms-1.3.1-py3.8.egg\nptdms\reader.py", line 241, in _read_segment_metadata
properties = segment.read_segment_objects(

File "C:\Users\284210\Anaconda3\lib\site-packages\nptdms-1.3.1-py3.8.egg\nptdms\tdms_segment.py", line 103, in read_segment_objects
raw_data_index_header = _struct_unpack(endianness + 'L', raw_data_index_header_bytes)[0]

error: unpack requires a buffer of 4 bytes
@adamreeve
Copy link
Owner

Hi, thanks for the bug report. It looks like the current logic for trying to read the last segment when writing was interrupted is assuming that the metadata was written okay and the data segment might be truncated, but doesn't account for cases where the segment metadata was not fully written.

@adamreeve adamreeve added the Bug label May 31, 2022
@adamreeve adamreeve mentioned this issue Jun 12, 2022
Merged
@adamreeve
Copy link
Owner

Hi @mdevore300, I've just pushed a fix for this in #273, are you able to test with the current master branch of the code to see if your file can now be read?

@mdevore300
Copy link
Author

mdevore300 commented Jun 15, 2022 via email

@adamreeve
Copy link
Owner

Hmm okay I'm not sure what's going on there, are you able to provide a copy of the TDMS file that reproduces the error?

@adamreeve adamreeve reopened this Jun 15, 2022
@mdevore300
Copy link
Author

mdevore300 commented Jun 15, 2022 via email

@mdevore300
Copy link
Author

mdevore300 commented Jun 15, 2022 via email

@adamreeve
Copy link
Owner

Sorry I'm guessing you've sent the files as email attachments but it doesn't look like GitHub adds them to the issue, you could send them directly to me at adreeve@gmail.com.

@adamreeve
Copy link
Owner

We've confirmed this is fixed but just updating the source files wasn't enough to pick up the fix. The fix has been released in version 1.5.0.

@strontium5
Copy link

I'm using release 1.5.0, but still getting this same error:
line 103, in read_segment_objects raw_data_index_header = _struct_unpack(endianness + 'L', raw_data_index_header_bytes)[0] struct.error: unpack requires a buffer of 4 bytes

@adamreeve
Copy link
Owner

Hi @strontium5, are you able to provide an example file that reproduces the problem?

@mdevore300
Copy link
Author

mdevore300 commented Oct 11, 2022 via email

@ba05
Copy link

ba05 commented May 16, 2024

I'm still having this issue with version 1.9.0.

@adamreeve
Copy link
Owner

Hi @ba05, if you can email me an example file that reproduces the error I can take a look.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Bug
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Fix reading files with truncated metadata in the final segment adamreeve/npTDMS
4 participants
@adamreeve @strontium5 @ba05 @mdevore300