meshconvert: invalid binary vtk files

[neurolabusc]

Describe the bug

meshconvert 3.03 with MacOS x86_64 executable creates binary vtk files that use little-endian order. This conflicts with the VTK format which requires thatVTK data files described here are written in big endian form.

To Reproduce

Consider a mesh named plane.obj with the contents:

v 0 0 0
v 1 0 0
v 1 1 0
v 0 1 0
f 1 2 3
f 1 3 4

Run the commands

meshconvert plane.obj plane.vtk
meshconvert -binary plane.obj bplane.vtk

Note that the ASCII plane.vtk is a valid VTK file. However, the binary bplane.vtk is unable to be opened with VTK viewers like Surfice or Slicer3D.

Using a hex editor reveals the error: both the vertices and faces are being stored in the native little-endian order, rather than the required big-endian format.

As an aside, saving vertex position as double instead of float seems excessive.

Platform/Environment/Version

Please provide the following information:

• OS: macOS: `11.7

• MRtrix3 version == mrinfo 3.0.3 ==

[Lestropie]

I have some recollection of having looked at this in some detail at the time.

For clarity, here's the full quote from the VTK format document:

Most of the time VTK manages the byte ordering of binary files for you. When you write a binary file on one computer and read it in from another computer, the bytes representing the data will be automatically swapped as necessary. For example, binary files written on a Sun are stored in big endian order, while those on a PC are stored in little endian order. As a result, files written on a Sun workstation require byte swapping when read on a PC. (See the class vtkByteSwap for implementation details.) The VTK data files described here are written in big endian form.

To me, this means that it is permissible for binary data to be written to the file system in either endianness, and it is the responsibility of the software reading such to identify the endianness and convert as necessary for the host system. When it says "VTK data files described here are written in big endian form", I presume that I had interpreted that at the time as "any files described henceforth in this document have been arbitrarily chosen to be big endian rather than little endian". However I fully recognise that my own code currently fails to autodetect endianness at read time, likely due to the absence of a magic number that would simplify identifying such.

There are also relics in the code where at one point I had been saving as big-endian but then reverted.

But if common softwares are unable to read little-endian data, then agreed that it needs to be modified. And given the software has been writing little-endian data for some time, I probably need to try to come up with a way of reading data in either form... Or maybe just have an environment variable or the like that can be temporarily set to modify the load when need be.

As an aside, saving vertex position as double instead of float seems excessive.

Agreed. Looking at the code, my suspicion is that the relevant code was being written around the same time as we were debating the transition to using double-precision as the default across the software base.

[neurolabusc]

Pragmatically, ParaView, Slicer3D, Surfice, etc. always create and expect legacy VTK files to be big-endian. The header has no flags to specify endianess. In my example, it is in theory possible for POLYGONS n to be either 3 or 50331648. I interpret the VTK document differently than you: it is saying is that developers using the VTK libraries do not need to worry if they are developing on big or little endian computers, as the library will always byte swap data to native endian when reading and handle storing the correct ordering when writing. However, it notes that the resulting files of this format are always big-endian.

The MRtrix code mentions the issue but does not seem to detect endianess or write big endian.

// FIXME Binary VTK output _still_ not working (crashes ParaView)
// Can however export as binary then -reconvert to ASCII and al is well...?
// Changed to big-endian output, doesn't seem to have fixed...

The current MRtrix code has the BigEndian code commented out

//  MR::putBE ((*i)[id], &temp[id]);

But the currently commented out code is clearly wrong, as is does not discriminate the size of the datatype (e.g. you need to swap 32-bit floats differently from 64-bit doubles).

temp[0] = ByteOrder::BE<float> temp[0];

[Lestropie]

The comments & commented code definitely suggest that I somehow gave up trying to get compliant data and was content with having MRtrix3 write & read data to / from itself. Unfortunately get spread thin far too often...

I can definitely see your interpretation of the text. Partly it's confusing as it seems to fail to distinguish between the VTK format and software library; my interpretation is based on the former whereas yours is based on the latter. But for example, with a little bit of elision I can produce:

For example, binary files written ... on a PC are stored in little endian order.

So I don't think my interpretation is unreasonable. But if common software won't read such, it is indeed pragmatically a moot point.