Fix behaviour of deerload with 2D-datasets

[laenan8466]

The resolves the error that the data was mapped the wrong direction.
Error was found for the attached dataset and need testing for further 2d datasets.
2D_ScanTime.zip

Fixes #82

[luisfabib]

I have locally run some diagnostics and the bug seems to be somewhere else. However, before I push anything since I do not know exactly what to expect from the data: @laenan8466 are these plots what you would expect from the dataset (with dimensions 285x300)?

[laenan8466]

Hey, nope wrong direction.
The time axis has 285 entries and there are 300 scans (from which only 111 contain data).
This script plots them for me (with my fix):

# Load data
dataImport = dict() # introduce as dictionary
dataImport['xyAxisArray'],dataImport['intensity'],dataImport['parameters'] = dl.deerload(filepath,full_output=True)

# Convert into data array
data2d = dict() # dictonary
data2d['time']=dataImport['xyAxisArray'][:,0] # Get first column with time
# remove nan values using numpy.isnan() 
# and numpy.logical_not 
data2d['time'] = data2d['time'][np.logical_not(np.isnan(data2d['time']))] 

data2d['scans'] = np.round(dataImport['xyAxisArray'][:,1]*1000+1) # convert scan id to 1 ... n and as integer

# Delete all zero elements
# mask them
zeroMask = np.ma.masked_equal(dataImport['intensity'],0) # mask all values that are zero

data2d['intensity']=dataImport['intensity'][~zeroMask.mask[:,0],:] # use only row mask to delete all rows (=scans) that are zero
data2d['scans']=data2d['scans'][~zeroMask.mask[:,0]] # also crop scans


plt.plot(data2d['time'],np.transpose(data2d['intensity']))

plt.xlabel('Time / us')
plt.ylabel('Intensity')

For reference: You are looking at a series of LaserIMD data. So there is a plateau in the beginning, a decay and increase and then a plateau again.

In your second picture, one can see the issue: the step seems to be shifted with every scan, but they should align as shown in my figure.

[luisfabib]

Ok, now I understand. Thanks, I'll keep working on it.

[laenan8466]

As reference - if you sum up all data in scan direction one should result s.th. like this:

The decay in the end is not normal for LaserIMD and reason is the used resonator/setup which is not ideal but needed in this case. If you go from 0.4 to roughly 1.7 us you will get a deer trace.

[luisfabib]

The cause of this bug was a line inside the file reader code block that was creating a too large matrix leading to a bug-ish reshaping later.

Additionally, I have improved the interface for the abscissa output. Before N abscissa with M being the largest amount of points was being returned as NxM matrix. The shorter abscissas were filled with NaN values. This is a MATLAB relic from the migration and makes no sense in the context of Python. Now, abscissa is a list of arrays. Each n-th element of the list is the abscissa array of the corresponding n-th dimension of the dataset.

Before:

t1 = abscissa[:,0]
t2 = abscissa[:,1]
t2 = t2[~np.isnan(t2)]

Now:

t1 = abscissa[0]
t2 = abscissa[1]

[luisfabib]

Oh, it seems I had a bug in my test. The reshaping error still persists.

[laenan8466]

I second the new abscissa format. Good choice!

Just text me, if I can assist with the reshaping/import errror.

[luisfabib]

The issue seems to be on the definition of the dimensions. The data are read as a 1D-array and split according to the XPTS and YPTS defined in the DSC file to form an XPTSxYPTS matrix. In this case, the dataset should be 300x285, however, in the DSC file XPTS=285 and YPTS=300.

[laenan8466]

I see.
Does the loading works properly with other 2D-datasets that doesn't use 'scans' on one axis (YNAM= 'unknown')? Do you have another dataset with multiple scans to compare?

Not sure why this problem occurs but I remember similar issues when working with a MATLAB importer library we wrote for our group. Our solution was to analyse the X/YNAM and assign different importers. But parts of our import depended on @stestoll easyspin eprload, where the import from binary data is handled.

So I guess the imported matrix itself should have the correct order. But to me the reading from buffer is s.th. I do not understand completely and that scares me when working with it. Could one read the correct dimensions directly? But I'm not good with reading binary data.

[luisfabib]

I think I have fixed the problem with the last set of commits. With this MWE:

import numpy as np 
import matplotlib.pyplot as plt 
import deerlab as dl 


dataImport = dict() # introduce as dictionary
x,y,par = dl.deerload('LaserIMD_01_DB261_20200621.DSC',full_output=True)

plt.subplot(211)
plt.plot(x[0],np.sum(y,axis=1))
plt.ylabel('V(t1)')
plt.xlabel('t1 [us]')
plt.tight_layout()
plt.show()

I obtain the correct accumulated signal as @laenan8466 described above:

I can obtain satisfactory results as well with other 2D-datasets.

[laenan8466]

Works for me, suggestions in comments above (minor).
Thanks for all the work!

[luisfabib]

Seems I forgot a couple of print there.... Nice catch! Thanks!