struct.error: unpack requires a buffer of 8 bytes

[Dragony]

Hello there! Thanks for investing so much time into such a usefull tool :-)

Today I tried to extract data via the CLI:

python3 .\PyPoE\cli\exporter\core.py dat json content.json --files Mods.dat

That seems to work for a while until:

C:\Users\drago\AppData\Local\Temp\tmpe4h6a1cv\chunk38.in:    23496 =>   262144 (0.00 seconds, 311.48 MB/s)
C:\Users\drago\AppData\Local\Temp\tmpe4h6a1cv\chunk39.in:    23672 =>   262144 (0.00 seconds, 303.55 MB/s)
C:\Users\drago\AppData\Local\Temp\tmpe4h6a1cv\chunk40.in:    14561 =>   147600 (0.00 seconds, 259.27 MB/s)
  0%|                                                                                                                                                                                                                   | 0/1 [00:09<?, ?it/s]
23:43:09 Traceback (most recent call last):
  File "C:\Users\drago\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\local-packages\Python39\site-packages\pypoe-1.0.0a0-py3.9.egg\PyPoE\cli\core.py", line 145, in run
    code = args.func(args)
  File "C:\Users\drago\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\local-packages\Python39\site-packages\pypoe-1.0.0a0-py3.9.egg\PyPoE\cli\exporter\dat\parsers\json.py", line 109, in handle
    dat_files = self._read_dat_files(args)
  File "C:\Users\drago\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\local-packages\Python39\site-packages\pypoe-1.0.0a0-py3.9.egg\PyPoE\cli\exporter\dat\handler.py", line 135, in _read_dat_files
    df.read(file_path_or_raw=data, use_dat_value=False)
  File "C:\Users\drago\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\local-packages\Python39\site-packages\pypoe-1.0.0a0-py3.9.egg\PyPoE\poe\file\shared\__init__.py", line 214, in read
    return self.get_read_buffer(file_path_or_raw, self._read, *args, **kwargs)
  File "C:\Users\drago\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\local-packages\Python39\site-packages\pypoe-1.0.0a0-py3.9.egg\PyPoE\poe\file\shared\__init__.py", line 172, in get_read_buffer
    return function(*args, buffer=BytesIO(file_path_or_raw), **kwargs)
  File "C:\Users\drago\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\local-packages\Python39\site-packages\pypoe-1.0.0a0-py3.9.egg\PyPoE\poe\file\dat.py", line 946, in _read
    self.reader.read(buffer.read())
  File "C:\Users\drago\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\local-packages\Python39\site-packages\pypoe-1.0.0a0-py3.9.egg\PyPoE\poe\file\dat.py", line 849, in read
    self.table_data.append(self._process_row(i))
  File "C:\Users\drago\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\local-packages\Python39\site-packages\pypoe-1.0.0a0-py3.9.egg\PyPoE\poe\file\dat.py", line 792, in _process_row
    row_data.append(self._cast_from_spec(spec, casts, data=cell_data, offset=offset))
  File "C:\Users\drago\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\local-packages\Python39\site-packages\pypoe-1.0.0a0-py3.9.egg\PyPoE\poe\file\dat.py", line 763, in _cast_from_spec
    value.append(self._cast_from_spec(specification, casts[1:], value, data_offset+i*casts[1:][0][1]))
  File "C:\Users\drago\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\local-packages\Python39\site-packages\pypoe-1.0.0a0-py3.9.egg\PyPoE\poe\file\dat.py", line 713, in _cast_from_spec
    ivalue = data[0] if data else struct.unpack('<' + casts[0][2], self._file_raw[offset:offset+casts[0][1]])[0]
struct.error: unpack requires a buffer of 8 bytes

Any idea why this is happening?

[John-Paul-R]

This error can sometimes occur when the spec does not match what is present in the actual dat file you are trying to read.

[Dragony]

OK thanks. I've seen the Specification row size ... error a lot and wasn't sure if this one was from the same category.

[John-Paul-R]

To be a bit more specific: I'm not sure if that is the only thing that can cause this error, but it's the only thing that I've personally had cause it.

[Dragony]

Do you know if it is reasonable to expect an update from the maintainers? Or do you think I will have more luck fixing this myself?

[John-Paul-R]

checks project's commit history

It seems that some people occasionally submit specification updates, but I don't see a standard release cycle anywhere.

Generally, if I have a spec error I fix it myself. (There are instructions for how to name different types of fields in fields.py). If you do so, consider submitting your spec update as a PR, so that everyone can benefit.

If you wait, there is a chance someone might come along and update it, but there is no official update strategy/deadline afaik.

E: I imagine many people simply update the spec for just the files that they need to reference, and leave the rest.