Tabs, Batch Processing and 2D data slicing and processing for pr #2181

ru4en · 2022-09-07T13:20:07Z

Tabs, Batch Processing, Plots, Tables and 2D data slicing & processing for Pr

This is the work I’ve done during my summer placement. Most of the changes I have made are in the Inversion Perspective directory. I’ve attached A short document describing the changes/feature I’ve implemented.

Features:

Tabs
1D Batch Processing
2D slicing and Batch Processing of slices
2D plot and 1D slice Plots
Batch Table
CSV exporting and Plotting on the Batch Table window
Minor changes such as fixing the Dropdown menu and remembering Parameters

Find more Information in the PDF bellow.
Notes and Reference Document for Pr interface DRAFT.pdf

issue #1448 - Enable batch processing for P(r) NOTE: This issue is not yet fixed. Feature: Selected data sent from the load data can now be sent to Pr window and a new tab will appear for each data set. Although the goal is batch calculate all data and neatly display in a table the current “non batch” tab method can be used for quick references of calculations for each file. In the nect commit I intend to implement a batch tab. Notes for those are in the bottom. Dev note - 1) Block automatic calculation after data is sent to Pr prospective whilst batch mode is deselected. (Allow users to hit calculate) Or block graphs from showing automatically as it clusters the interface. - 2) Fix batch Mode o When batch mode is selected create only one tab (similar to fitting) o New interface for batch / same interface could be used with the addition of “Fit” / ” Graph” button conjunction with the 1st TODO. 3) Clean up Code. The current files ive modified/created are messy. Also comment code

…w and efficient. also a way to display/export all the data in a table is still required

Currently, Tabs works. Batch still has a little ironing out to do. mainly its interaction with setCurrentData(). Additionally, The data from Batch still needs to be sent to a Table and eventual reported/exported.

files calculated using Pr can now be batch Processed and saved to a table. However, a lot of work is still left to fix optimise and automate specific parts of this feature. things still left to fix - [ ] calculate all files before showing the table - [ ] perform calculations without stop showing the plots as it slows down and clutters the interface - [ ] optimise, clean and comment the code

Files imported into Pr interface can now automatically be processed and outputs displayed in a table. Code still requires to be optimised and cleaned. A better way to update the table is also required. things still left to fix - [x] calculate all files before showing the table - [x] perform calculations without stop showing the plots as it slows down and clutters the interface - [ ] optimise, clean and comment the code

However, im still getting errors with displayChange and setCurrentData even though the calculations and outputs are successful.

Batch Processing still not working properly. Cleaned a bit of the code(That I was working on). 2D data shows a plot of the data however calculations still cannot be made. in the process of coding the bit gets a slice of the 2D data.

when 2D data is loaded into Pr interface a raw data plot should pop up. Using the Pr interface, parameters for the conversions can be inputted and when calculated should update the graphs and calculate the Pr values for those specific slices.

… be plotted on a single plot. However, currently only one of the slices is calculated using Pr, this will be changed (Hopefully) in the next commit so that all sliced can be processed together similarly to 1D Batch and its outputs displayed in a table.

…lices.

the table populates the slices created and the button when pressed need to show plots of the slice. calculate all button need to also be fixed. the data cannot be read as its not in the right object.

…dating the gui and getting/setting parameters.

Furthermore, the data output ont the batch table will be from self._calculator. this means It's flexible for further expansions and is just as reliable as the gui. Although a few Issues still remain during calculation, one being Q range being set to 0 or bellow which causes the data set to not be calculated and the calculations of the previous data to be set for the 'current' data set (Hence you might see duplicate rows with different file names). In my previous commit in the branch 1D data could not be uploaded for individual calculation that bug has now been fixed. Code still needs to be cleaned up and commented.

this is a bit of a hacky way of getting it to work and is just to get an idea of how it might work. things to improve upon are :- - [ ] Getting the table in GridPanel.py to work flexibly for both 1D batch aswell as 2D slices. - [ ] cleaning up slices() - [ ] making showBatchOutput() more portable - [ ] creating batch like datalist support for 2D slices. (so that it can be batch processed just like 1D batch files) - [ ] SORTING OUT THE ERRORS - [ ] (Might need help) Fix the Wierd Parameter issue. (Currently set/default parameters change to estimates which makes it hard to use reliabely. Background, Qmin and Qmax is also not working as expected, it sets itself to a value bellow 0 causing it to not calculate for that slice.)

need to test, clean up code, and make sure that this doesn't affect 1D batch processing

…ionally, Q min and Q Max can be seen in the Batch results panel. However, further testing and Code cleaning is required. Newly created pop up windows such as the batch results, 1D slice plot, and 2D image plots need to be connected with the interface and made to fit in withing the UI.

… Pr fit and I(q) plots. This Version should somewhat work besides the found bugs. Testing and code cleaning still required. - [ ] Batch Processing might not yet be reliable. A row or 2 might duplicate if a parameter is not right eg: Qmax or NoOfTerms. - [ ] Dropdown menu Goes back to previous Item if user decides to change. - [ ] All New Windows Are outside the SasView Interface.

…u. Additionally the dropdown menu used to not set to new selected data which has been fixed. however, in this version 2D data slicing is bracken this is due to some changes with how raw batch data is handles.

… the dataset. when switched back to the dateset (using the dropdown menu) the parameters selected prevously will be set.

…ead to a dir so each data file could be assigned to a thread instance. estimationThread and estimationThreadNT might also need to be changed as such. however more testing is required.

slicing: slices data are turned into model using the createModelItemWithPlot function. this is makes it much easier to move around as it can work just like 1D data. 1D batch: Parameters set before batch processing is remembered and used when processed using batch mode.

…imations". Parameters whilst, Batch Processing is also remembered. Estimated data doesn't mess with actual data unless suggested button is pressed. However, Duplication issue still remains.

…ne using to Calculate all button is still a bit buggy.

Still a bit more tweaking is required for batch processing slices.

…D rawImage and shows slice using phi. Fixed buttons. Minor cleanups. fixed 2D slices from not being batched processed.

for Devs: Plots in Pr can be disabled using self._allowPlots Plot Behaviour: if Batch or Data2D Slices -> hide Plots if Button(Calculate) or Single Data1D -> show Plots

… the lager functions into smaller functions. And Changes the Warnings for GridPanel.py.

lucas-wilkins · 2024-07-27T10:08:02Z

@krzywon should all be fixed.

…w/sasview into 2D-Data-Processing-for-Pr

… seem get to run with the following code ``` GuiUtils.updateModelItemWithPlot(item, new_plot, new_plot.id) self.base.manager.communicator.plotUpdateSignal.emit([new_plot]) self.base.manager.communicator.forcePlotDisplaySignal.emit([item, new_plot]) ``` changes made using feedback from: 1) #2181 (comment) removed as it was something that I was testing out but didn't end up using. 2) #2181 (comment) Moved closeEvent() from InversionWidget.py to InversionPerspective.py. 3) #2181 (comment) changed `from sas.sascalc.dataloader.manipulations import SectorQ` to `from sasdata.data_util.manipulations import SectorQ`

krzywon · 2024-10-25T19:42:20Z

This might be superseded by #3137. Keeping this open for now, but likely will be closed next week.

rozyczko · 2024-10-28T09:49:26Z

One issue (?) found with the range sliders: moving them on the I(q) chart doesn't update the values in the Options tab for Total Q range. The other way - when modified in the Max:` textbox, the range change propagates to the chart

rozyczko · 2024-10-28T09:53:57Z

Another: the widget should not be resizable. There is nothing to be gained by resizing. It only looks bizarre when you maximize this widget.
It's different than the Fitting Perspective, where resizing can help access and view parameters, when the table is large.

rozyczko

small changes requested for consideration

rozyczko · 2024-10-28T10:06:21Z

src/sas/qtgui/MainWindow/DataExplorer.py

@@ -616,8 +616,22 @@ def updatePerspectiveWithProperties(self, key, value):
        if 'pr_params' in value:
            self.cbFitting.setCurrentIndex(self.cbFitting.findText('Inversion'))
            params = value['pr_params']
-            self.sendItemToPerspective(items[0])
-            self._perspective().updateFromParameters(params)
+            # Make the perspective read the rest of the read data


this conditional contains the same content as the branch above (for fitting) and the common code should be merged

rozyczko · 2024-10-28T10:13:02Z

src/sas/qtgui/Perspectives/Inversion/InversionPerspective.py

+        return True
+
+    @property
+    def supports_fitting(self):


this should probably be supports_fitting_menu

The default for supports_fitting_menu is False, so this might be unneccessary.

rozyczko · 2024-10-28T10:17:17Z

src/sas/qtgui/Perspectives/Inversion/InversionPerspective.py

    def communicator(self):
-        return self.communicate
+        return self.filesWidget.communicate


there's only one instance of Communicate - created in GuiManager. Both self.communicate (from the parent, which is GuiManager) or from filesWidget should be the same object. Why change it here?

rozyczko · 2024-10-28T10:26:39Z

src/sas/qtgui/Utilities/GridPanel.py

            key = table.horizontalHeaderItem(column).data(0)
+            for row in range(table.rowCount()):


avoid lazy try..except.

for column in range(table.columnCount()): value = [] key = table.horizontalHeaderItem(column).data(0) for row in range(table.rowCount()): item = table.item(row, column) if item is not None: value.append(item.data(0)) else: value.append(" ")

krzywon

Code wise, this is getting closer to finished.

Functionality wise, I'm seeing the following:

the save state is still broken - attempting to save with two tabs throws an error
the regularization constant and number of terms never seems to be calculated. Maybe it is slow, but that needs to get fixed.
Some of the sector slicer changes are still present. Updating the slicer parameter window updates the view, but not the 1D data. The 2D P(r), which is now disabled, should really be added to the batch slicer window as a fitting option.

@dehoni - Do you need/want help with any of this? I should be able to find time early next calendar year to work on this.

krzywon · 2024-10-29T15:48:06Z

src/sas/sascalc/pr/p_invertor.py

+        if norm == 0:
+            ret_val = 0
+        else:
+            ret_val = np.sqrt(oscill/norm) / np.pi * self.d_max
        return ret_val


Suggested change

if norm == 0:

ret_val = 0

else:

ret_val = np.sqrt(oscill/norm) / np.pi * self.d_max

return ret_val

return 0 if norm == 0 else np.sqrt(oscill/norm) / np.pi * self.d_max

krzywon · 2024-10-29T15:56:37Z

src/sas/sascalc/pr/invertor.py

@@ -161,21 +161,26 @@ def __setattr__(self, name, value):
            return self.set_y(value)
        elif name == 'err':
            value2 = abs(value)
+            if None in value:


If None is in the array, abs() will throw a TypeError and this message will not be reachable.

krzywon · 2024-10-29T17:53:39Z

src/sas/sascalc/pr/calc.py

+    if total == 0:
+        return 0
+    else:
+        return total_pos / total  # dx cancels


I have seen this same paradigm done in three different ways. They could all be simplified to return 0 if a == 0 else x / a

(A)

if a == 0: return 0 else: return x / a

(B)

if a == 0: return 0 return x / a

(C)

if a == 0: value = 0 else: value = x / a return value

krzywon · 2024-10-29T18:02:11Z

src/sas/qtgui/Utilities/GridPanel.py

+        # if output_data is not None:
+        #     # Set a table tooltip describing the model
+        #     model_name = output_data[0][0].model.id
+        #     self.tabWidget.setTabToolTip(0, model_name)


Removing unused code doesn't hurt

krzywon · 2024-10-29T18:09:02Z

src/sas/qtgui/Perspectives/Inversion/InversionWidget.py

+
+# Default Values for inputs
+NUMBER_OF_TERMS = 10
+REGULARIZATION = 0.0


The regularization constant has been 0.0001 in the past. Is there a reason for the change?

krzywon · 2024-10-29T18:23:18Z

src/sas/qtgui/Perspectives/Inversion/InversionPerspective.py

+    def _calculateUpdate(self, params: dict, tab_name):
+        """ Update the perspective using a dictionary of parameters
+        e.g. those loaded via open project or open analysis menu items"""
+        raise NotImplementedError("Update from parameters not implemented yet.")


What is this and why is it here?

krzywon · 2024-10-29T18:25:15Z

src/sas/qtgui/Perspectives/Inversion/InversionPerspective.py

+        return True
+
+    @property
+    def supports_fitting(self):


The default for supports_fitting_menu is False, so this might be unneccessary.

ru4en added 30 commits July 1, 2022 15:09

Added Batch Processing for Pr, However the interface is currently slo…

2f0e64b

…w and efficient. also a way to display/export all the data in a table is still required

Added Tabs and Batch Processing for Pr.

294c3cb

Currently, Tabs works. Batch still has a little ironing out to do. mainly its interaction with setCurrentData(). Additionally, The data from Batch still needs to be sent to a Table and eventual reported/exported.

Tab and Batch mode for Pr is now functional.

eac7699

However, im still getting errors with displayChange and setCurrentData even though the calculations and outputs are successful.

Fixed Batch Tabs for Pr inversion.

264bf8b

Batch Processing still not working properly. Cleaned a bit of the code(That I was working on). 2D data shows a plot of the data however calculations still cannot be made. in the process of coding the bit gets a slice of the 2D data.

buggy 2D to 1D plots fixed. The 1D plot shows the correct number of s…

024e208

…lices.

2D data has a new tab (Ui) with an outputs table.

18e660a

the table populates the slices created and the button when pressed need to show plots of the slice. calculate all button need to also be fixed. the data cannot be read as its not in the right object.

batch results are consistent now however there are a few bugs with up…

916c69b

…dating the gui and getting/setting parameters.

Working on Batch Processing for 2D slices.

f9dabdf

need to test, clean up code, and make sure that this doesn't affect 1D batch processing

Parameters for individual data sets can be set using the dropdown men…

eb99cd2

…u. Additionally the dropdown menu used to not set to new selected data which has been fixed. however, in this version 2D data slicing is bracken this is due to some changes with how raw batch data is handles.

set parameters for items in the dropdown menu of batch will saved for…

9ce2941

… the dataset. when switched back to the dateset (using the dropdown menu) the parameters selected prevously will be set.

Result Duplication Issue has slightly been patched by setting calcThr…

de06f70

…ead to a dir so each data file could be assigned to a thread instance. estimationThread and estimationThreadNT might also need to be changed as such. however more testing is required.

got batch processing to spit out actual calculations rather then "est…

722e27b

…imations". Parameters whilst, Batch Processing is also remembered. Estimated data doesn't mess with actual data unless suggested button is pressed. However, Duplication issue still remains.

Calculations on batch interface is more reliable now. Calculations do…

a878d19

…ne using to Calculate all button is still a bit buggy.

Slicer works a bit smother now.

1436167

Still a bit more tweaking is required for batch processing slices.

Fixed Errors for 2D data and added a few catchers.

480fb7d

fixed exporting from table issue, buttons in slicer now updates the 2…

c8a3502

…D rawImage and shows slice using phi. Fixed buttons. Minor cleanups. fixed 2D slices from not being batched processed.

Plots Behaviour is designed to be more dynamic.

09b3b10

for Devs: Plots in Pr can be disabled using self._allowPlots Plot Behaviour: if Batch or Data2D Slices -> hide Plots if Button(Calculate) or Single Data1D -> show Plots

Cleaned and Added Comments to InversionWidget.py, broke down a few of…

876c36e

… the lager functions into smaller functions. And Changes the Warnings for GridPanel.py.

Cleaned up and worked on some finishing touches ready for PR.

5a608ea

lucas-wilkins and others added 5 commits July 19, 2024 16:26

Unused variable

4b8749b

Alpha check rewrite

ad85c7d

Merge branch 'release_6.0.0' into 2D-Data-Processing-for-Pr

acdaaef

Merge branch 'release_6.0.0' into 2D-Data-Processing-for-Pr

de60d82

Merge branch 'release_6.0.0' into 2D-Data-Processing-for-Pr

4294f80

lucas-wilkins requested a review from krzywon July 27, 2024 10:07

wpotrzebowski added the Discuss At The Call Issues to be discussed at the fortnightly call label Jul 29, 2024

dehoni added 2 commits August 4, 2024 14:51

setting (and keeping) q limits in P(r) options

9987982

Merge branch '2D-Data-Processing-for-Pr' of https://github.com/SasVie…

053cac6

…w/sasview into 2D-Data-Processing-for-Pr

wpotrzebowski removed the Discuss At The Call Issues to be discussed at the fortnightly call label Aug 5, 2024

making QRangeSlidersTest run

bbbf464

wpotrzebowski added Discuss At The Call Issues to be discussed at the fortnightly call and removed Discuss At The Call Issues to be discussed at the fortnightly call labels Aug 12, 2024

dehoni added 6 commits August 19, 2024 10:04

fix optional q-range selection via text fields in Inversion

146b218

fixing incursion

16496f4

catch data with dy=0 crashing Inversion

66f038c

no empty string QMIN and QMAX input defaults

8565ba6

reinstate batch table

9b2272d

plot Pr and data from Batchresult table

befa7a9

wpotrzebowski removed the SasView 6.0.0 Required for 6.0.0 release label Aug 27, 2024

wpotrzebowski mentioned this pull request Oct 25, 2024

Merge 6.0.0 changes into main #3133

Open

7 tasks

krzywon mentioned this pull request Oct 25, 2024

Tabs, Batch Processing and 2D data slicing and processing for pr (rebase) #3137

Draft

rozyczko requested changes Oct 28, 2024

View reviewed changes

krzywon reviewed Oct 29, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Tabs, Batch Processing and 2D data slicing and processing for pr #2181

Tabs, Batch Processing and 2D data slicing and processing for pr #2181

ru4en commented Sep 7, 2022 •

edited

Loading

lucas-wilkins commented Jul 27, 2024

krzywon commented Oct 25, 2024

rozyczko commented Oct 28, 2024

rozyczko commented Oct 28, 2024

rozyczko left a comment

rozyczko Oct 28, 2024

rozyczko Oct 28, 2024

krzywon Oct 29, 2024

rozyczko Oct 28, 2024

rozyczko Oct 28, 2024

krzywon left a comment

krzywon Oct 29, 2024

krzywon Oct 29, 2024

krzywon Oct 29, 2024

krzywon Oct 29, 2024

krzywon Oct 29, 2024

krzywon Oct 29, 2024

krzywon Oct 29, 2024

		key = table.horizontalHeaderItem(column).data(0)
		for row in range(table.rowCount()):

Tabs, Batch Processing and 2D data slicing and processing for pr #2181

Are you sure you want to change the base?

Tabs, Batch Processing and 2D data slicing and processing for pr #2181

Conversation

ru4en commented Sep 7, 2022 • edited Loading

Tabs, Batch Processing, Plots, Tables and 2D data slicing & processing for Pr

Features:

lucas-wilkins commented Jul 27, 2024

krzywon commented Oct 25, 2024

rozyczko commented Oct 28, 2024

rozyczko commented Oct 28, 2024

rozyczko left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

krzywon left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ru4en commented Sep 7, 2022 •

edited

Loading