Adds support for reading Matlab Objects from matfile V5

[hernot]

The official version is not able to read matlab files using matfile V5
format, which contain matlab varaibles eposing a variable type of
MAT_C_OBJECT. Fields, arrays and cells of this type are reported to be
emtpy.

From documentation of matfile format V5 variables,fields, arrays and
cells of type MAT_C_OBJECT only fiffer from their MAT_C_STRUCT silblings
only by an additional field in the variable entry header stating the
name of the corresponding MATLAB class. This allows to implement the
support for reading MAT_C_OBJECT type variables by using following
strategy:

1) Wherever possible extend type checks such that operation to be

exectued on MAT_C_OBJECT variables is handled equal to MAT_C_STRUCT type
2) Keep MAT_C_OBJECT specific code as small as possible and resort
to MAT_C_STRUCT path as soon as there is no difference between handling
either.

The changes were originally implemented by this author for an initial
release of matio 1.5.3, used by data conversion tool of
g.tec (www.gtec.at) to allow reading data generated by the commercial
MATLAB and Simulink based tools.

Implementation is based upon pull from git://git.code.sf.net/p/matio/matio on
Sat May 27 13:21:32 2017

[tbeu]

Thanks for this PR. I merged your branch on top of recent master (and hopefully got the merge conflicts right).

• There is classname added to matvar_t, which changes the public API. Instead I prefer to have it in struct matvar_internal, where we are free of backward compatibility breaking changes.
• There are some TODOs added. Are you going to do them?
• Why did you add Mat_GetHeader to the public API? It is of course nice to have it, but I wonder, since it is not needed as part of your PR.

[hernot]

Regarding class_name field of matvar_t moving to matvar_internal. Is ok for me as long as you advise me how an appropriate function should look like to access and print the name of the matlab class definition should look like when it is stored on matvar_internal.

Regarding the TODO's I have put them as i would need your review whether the if conditions would be sufficient or whether in the following code there would be any extra code necessary. For now it seems that the code as is would be ok. But i currentl do not have access to the data and facility to throughly test them, and verify that that these two reminder todos can be solved by deleting them, which would be the best case.

[tbeu]

OK, I will move class_name member and check the TODOs in more detail.

[tbeu]

This is what I tried:

1. MATLAB commands

inline_object = inline('t^2');
memmap_object = memmapfile('c:\temp\mem.map');
save object_uncompressed -v6 inline_object memmap_object

2. MAT-File
   mat.zip

3. Matdump command

matdump -d -v object_uncompressed.mat

4. matdump output
   https://gist.github.com/tbeu/f0b077df1c278a4687b8aa4b9640d967

Thus, it seems to work on the inline object, but not on the memmapfile object. What kind of objects did you build?

[hernot]

Hi Am Freitag, den 16.06.2017, 07:41 -0700 schrieb tbeu:

This is was I tried: 1. MATLAB commands ```m inline_object = inline('t^2'); memmap_object = memmapfile('c:\temp\mem.map'); save object_uncompressed -v6 inline_object memmap_object ``` 2. MAT-File [mat.zip](https://github.com/tbeu/matio/files/1081060/mat.zip) 3. Matdump command ``` matdump -d -v object_uncompressed.mat ``` 4. matdump output https://gist.github.com/tbeu/f0b077df1c278a4687b8aa4b9640d967 Thus, it seems to work on the inline object, but not on the memmapfile object. What kind of objects did you build?

I created an m file describing the class object (old style @dir or new with classdef should not matter)

…

------ test_class.m classdef test_class      properties (access=public)        firstprop;        secondprop:     end     methods         function obj=test_class()             obj.firstprop=12;             obj.secondprop=10;         end     end end

-------------------- and than  obj = test_class(); save('c:\temp\mem.map','obj') With memmap file i havent yet worked in matlab, so i do not have any idea how memmap would have an influence on the file content.  Best Xristoph

[tbeu]

I now took your test_class example (mat.zip), however I cannot see that matdump dumps the class object. Instead, when debugging it, I see that class_type is MAT_C_UINT8 for class_object or memmap_object (and MAT_C_OBJECT for inline_object as before). I believe this issue is related to #47 (MCOS). Can you please debug and check the expected behavior. Thanks.

[hernot]

Hi Am Samstag, den 26.08.2017, 03:53 -0700 schrieb tbeu:

I now took your test_class example ([mat.zip](https://github.com/tbeu /matio/files/1253582/mat.zip)), however I cannot see that matdump dumps the class object. Instead, when debugging it, I see that class_type is MAT_C_UINT8 for class_object or memmap_object (and MAT_C_OBJECT for inline_object as before). I believe this issue is related to #47 (MCOS). Can you please debug and check the expected behavior. Thanks.

Afaik it should be MAT_C_OBJECT, and the MAT_c_UINT8 i can check, eventhoug it may take some time until i get to. Do you have some testdata on github already. Than i would update my fork before checking, so that I'm in sync with your corresponding branch. Best Xristoph

[tbeu]

You can take the MAT file from mat.zip or mat.zip as test data. I checked it on your plain commit , thus you could debug it with your local branch (no need to sync right now).

[hernot]

Ok checking the matlab documentation of memmapedfile i fear you can not memmap objects

From the documentation of the memmaped file Format parameter you can only specify numerical types no cells no char no struct and no object.

https://de.mathworks.com/help/matlab/ref/memmapfile.html

Thus could you please check (i do not have a matlab here,*) matdump reports the proper type for the following examples

char_object = 'Hello world'

memmap_object = memmapfile('c:\temp\mem_struct.map');
save object_uncompressed -v6 char_object memmap_object

struct_object=struct('Hello',{42},'World',{8*6});

memmap_object = memmapfile('c:\temp\mem_struct.map');
save object_uncompressed -v6 struct_object memmap_object

------ TestClass.m ------

classdef TestClass

properties (access = public)
a=1;
b="Hello World"
c
end
methods (access = public)
function obj = TestClass(surprise)
c = surprise;
end
end
end

object_object = TestClass('So long. Goodby and thanks for all the fish');
memmap_object = memmapfile('c:\temp\mem_object.map');
save object_uncompressed -v6 struct_object memmap_object

(*) or can you please at least prepare the saved files for me so that i can use them for debuging

What does matdump report in both cases. If the explicit format specifier does not allow any thing else than numeric types, no char, no cell, no struct, no object, no complexl, than i do suspect that char, cell, struct and object are not supported either when no format is specified.

What does matlab say when you try to reopen the above examples with memmappedfile for reading and what when you use normal load?
Do you get the char, struct and object back?

Which i fear is why they only show examples of numeric arrays and matrices to be mapped and no other data type.

Or matlab does some trick to store object as huge mat_c_uint8 type blob which it backconverts lateron. For classes it mandatorily would need the cassdef on the matlab path. To check and verify this hypothesis i would need at least the resulting saved files (memmap and normal).

[tbeu]

I get the same results as #64 (comment) if I export the MAT-files in R2014a instead of R2015b.

MAT files: mat.zip

[hernot]

Hi Thank you very much. Am Mittwoch, den 15.11.2017, 12:00 +0000 schrieb tbeu:

I get the same results as #64 (comment) comment-338461338 if I export the MAT-files in R2014a instead of R2015b.

Seems as if Matlab stores old style classes/objects defined via class directories @classname differentenly than newstyle classes defined via classdef. The ones we used in the old company were old style objects. Anyway.  Can you upload all your results including Matlab 2014a and Can you do me the favour and serialize a simple matrix using the getByteStreamFromArray(anyData) as follows

…

------------------ snip ------------------- % bests would be if anymatrix would reference the data inside our test % class(es) without the class decoration instead of just the randi  % matrix below  anymatrix = randi(5,5) % if the b for force binary is not recognized any more remove the b fid = fopen('serializeddata.bin','wb')  if fid >= 0     fwrite(fid,getByteStreamFromArray(anymatrix))     fclose(fid) end

---------------------------------------- And send me the result. That would help and simplify to read and interpret the opaque class structure allowing to extract new style class data from mat V5  files. even though not officially documented, i bet it is not much different, excempt removed tags/byte length which seems to be hardcoded and thus implicitly known by Matlab ;-) Best Xristoph     -- Christoph, Elisabeth Hintermüller Eisenhandstraße 33/23 4020 Linz Austria Tel.: +43 650 8827347 mail: christoph@out-world.com mail: elisabeth@out-world.com www: http://www.out-world.com

[tbeu]

Seems as if Matlab stores old style classes/objects defined via class directories @classname differentenly than newstyle classes defined via classdef. The ones we used in the old company were old style objects. Anyway.

Should I rather try to save old style object?

[hernot]

Hi  Am Donnerstag, den 16.11.2017, 10:08 +0000 schrieb tbeu:

> > Seems as if Matlab stores old style classes/objects defined via > class directories @classname differentenly than newstyle classes > defined via classdef. The ones we used in the old company were old > style objects. Anyway. Should I rather try to use old style object?

Hm not sure if they havent changed their storage too, as, as far as i remember, there was an error about invalid parameters for all when loading old style objects in afiak Matlab 2014a or 2015a, not shure which version. And it is likely they now also store the inside the opaque object, as in 2016 a there this error we did not observer any more.  Si i suggest to wait until somebody complaiins about not beeing able to load them form old data < 2014a and now try to figure if consistent loading of objects from opaque object is possible. Best Xristoph

[tbeu]

Is this still being worked on? If not, I propose to close it w/o merge?

[hernot]

Hi  Inline Response (answers below questions asked) Message sent on Samstag, 03.02.2018, 15:25 +0000 by tbeu:

Is this still being worked on? If not, I propose to close it w/o merge?

I hasd to postpone it the last month and the obpaque object, which is suspect is a hardcoded matrix/struct which carries as payload a matlab serialization string as posted above is still pending. I would need some files to continue on this as stated in my earlier posts. But i do not have any idea when i will have time to continue, it is not on higest prio for now. So if you could generated the files for me and sedn them i would be ok with closing afterwards and sending a new one when finally solved the object riddle.  LG Xristoph

[tbeu]

One year later: can we close it?