Enable Out-of-Core #703

mmarineBlueQuartz · 2023-09-26T17:03:29Z

Update filters and unit tests to enable out-of-core data.
Update Application to require static methods for creation and destruction.
Update ParallelAlgorithm classes for disabling onout-of-core data.
Add memory utilities for checking available and total memory.

Naming Conventions

Naming of variables should descriptive where needed. Loop Control Variables can use i if warranted. Most of these conventions are enforced through the clang-tidy and clang-format configuration files. See the file complex/docs/Code_Style_Guide.md for a more in depth explanation.

Filter Checklist

The help file complex/docs/Porting_Filters.md has documentation to help you port or write new filters. At the top is a nice checklist of items that should be noted when porting a filter.

Unit Testing

The idea of unit testing is to test the filter for proper execution and error handling. How many variations on a unit test each filter needs is entirely dependent on what the filter is doing. Generally, the variations can fall into a few categories:

1 Unit test to test output from the filter against known exemplar set of data
1 Unit test to test invalid input code paths that are specific to a filter. Don't test that a DataPath does not exist since that test is already performed as part of the SelectDataArrayAction.

Code Cleanup

No commented out code (rare exceptions to this is allowed..)
No API changes were made (or the changes have been approved)
No major design changes were made (or the changes have been approved)
Added test (or behavior not changed)
Updated API documentation (or API not changed)
Added license to new files (if any)
Added example pipelines that use the filter
Classes and methods are properly documented

joeykleingers

Just some minor things to remove, but otherwise this looks pretty good. I'll be testing Solomon's pipelines again over the next few days and will record filter execution time statistics.

src/Plugins/ComplexCore/src/ComplexCore/Filters/FindArrayStatisticsFilter.cpp

src/Plugins/ComplexCore/test/FindArrayStatisticsTest.cpp

src/Plugins/ComplexCore/test/FindSurfaceFeaturesTest.cpp

...entationAnalysis/src/OrientationAnalysis/Filters/Algorithms/FindKernelAvgMisorientations.cpp

JDuffeyBQ

Joey's comments about some leftover debugging code also should be addressed.

src/complex/Common/Uuid.hpp

src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/GenerateFZQuaternions.cpp

...Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/GenerateIPFColors.cpp

...ns/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/GenerateGBCDPoleFigure.cpp

...s/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/GenerateFaceIPFColoring.cpp

src/Plugins/ComplexCore/src/ComplexCore/Filters/Algorithms/ConvertData.cpp

src/Plugins/ComplexCore/src/ComplexCore/Filters/Algorithms/ConvertColorToGrayScale.cpp

src/complex/Filter/FilterList.cpp

src/Plugins/ComplexCore/src/ComplexCore/Filters/ImportHDF5Dataset.cpp

src/Plugins/OrientationAnalysis/test/CMakeLists.txt

imikejackson · 2023-11-06T23:23:27Z

src/Plugins/ComplexCore/src/ComplexCore/Filters/Algorithms/FindArrayStatistics.cpp

+                                                       inputValues->FindSummation, inputValues->FindHistogram, inputValues->MinRange, inputValues->MaxRange, inputValues->UseFullRange,
+                                                       inputValues->NumBins, inputValues->FindModalBinRanges, mask, featureIds, source, featureHasDataPtr, lengthArrayPtr, minArrayPtr, maxArrayPtr,
+                                                       meanArrayPtr, modeArrayPtr, stdDevArrayPtr, summationArrayPtr, histArrayPtr, mostPopulatedBinPtr, modalBinsArrayPtr, filter));
+    indexAlg.setParallelizationEnabled(false);


This still needs to be validated as to why it is disabled?

imikejackson · 2023-11-06T23:24:13Z

src/Plugins/ComplexCore/src/ComplexCore/Filters/Algorithms/FindArrayStatistics.cpp

+// #ifdef COMPLEX_ENABLE_MULTICORE
+#if 0


@mmarineBlueQuartz Can we validate that this parallelization has to be disabled?

imikejackson · 2023-11-06T23:24:57Z

src/Plugins/ComplexCore/src/ComplexCore/Filters/Algorithms/PartitionGeometry.cpp

@@ -244,6 +246,7 @@ Result<> PartitionGeometry::partitionCellBasedGeometry(const IGridGeometry& inpu

  ParallelData3DAlgorithm dataAlg;
  dataAlg.setRange(dims[0], dims[1], dims[2]);
+  dataAlg.setParallelizationEnabled(false);


@mmarineBlueQuartz Does this need to be disabled? Can we do the same checks for out-of-core arrays versus in-core arrays?

* Update filters and unit tests to enable out-of-core data. * Update Application to require static methods for creation and destruction. * Update ParallelAlgorithm classes for disabling onout-of-core data. * Add memory utilities for checking available and total memory.

* Removed debugging code from NX Runner

Fix IOFormat and DataArrayTest

Signed-off-by: Jared Duffey <jared.duffey@bluequartz.net>

* Added out-of-core array checks. Fixed ConvertColorToGrayScale parallelization code for out-of-core datastores.

Signed-off-by: Michael Jackson <mike.jackson@bluequartz.net>

* Enable Out-of-Core * Update filters and unit tests to enable out-of-core data. * Update Application to require static methods for creation and destruction. * Update ParallelAlgorithm classes for disabling onout-of-core data. * Add memory utilities for checking available and total memory. * Moved out of core check in ITK filters to execute helper function * Added out-of-core array checks. * Fixed ConvertColorToGrayScale parallelization code for out-of-core datastores. --------- Signed-off-by: Jared Duffey <jared.duffey@bluequartz.net> Signed-off-by: Michael Jackson <mike.jackson@bluequartz.net> Co-authored-by: Jared Duffey <jared.duffey@bluequartz.net> Co-authored-by: Michael Jackson <mike.jackson@bluequartz.net>

mmarineBlueQuartz force-pushed the feature/Solomon_OOC branch from f18263a to 4912e35 Compare September 28, 2023 13:10

mmarineBlueQuartz requested a review from joeykleingers September 29, 2023 12:40

imikejackson force-pushed the feature/Solomon_OOC branch from 064a45d to 52b4caa Compare October 6, 2023 22:35

mmarineBlueQuartz requested review from JDuffeyBQ and removed request for joeykleingers October 13, 2023 14:28

joeykleingers approved these changes Oct 19, 2023

View reviewed changes

mmarineBlueQuartz force-pushed the feature/Solomon_OOC branch from 52b4caa to b53f1fe Compare October 24, 2023 16:06

JDuffeyBQ force-pushed the feature/Solomon_OOC branch 3 times, most recently from c588f5c to 349711c Compare October 31, 2023 18:18

JDuffeyBQ requested changes Oct 31, 2023

View reviewed changes

imikejackson requested changes Nov 6, 2023

View reviewed changes

mmarineBlueQuartz and others added 17 commits November 8, 2023 17:05

Clang-format

855124f

Fix Linux MemoryUtilities.cpp

c401657

Removed debugging code

1662a44

* Removed debugging code from NX Runner

Fix python

7adb5e2

Fix complex_test

d0f71cc

Fix IOFormat and DataArrayTest

PR requested changes

9564685

Fix PluginTest

09c5fc2

Moved out of core check in ITK filters to execute helper function

c2110ec

Signed-off-by: Jared Duffey <jared.duffey@bluequartz.net>

Fixed cmake comment

039ec04

Signed-off-by: Jared Duffey <jared.duffey@bluequartz.net>

Reverted extraneous change from ITK filters

4402b14

Signed-off-by: Jared Duffey <jared.duffey@bluequartz.net>

Added newline to EOF

d0a7c30

Signed-off-by: Jared Duffey <jared.duffey@bluequartz.net>

Changed to pass string by ref

c494fad

Signed-off-by: Jared Duffey <jared.duffey@bluequartz.net>

Removed commented out code

5664f58

Signed-off-by: Jared Duffey <jared.duffey@bluequartz.net>

PR Requested Changes

0b30c90

Linux compile fix

6753b7c

PR requested changes

4ab6dc0

* Added out-of-core array checks. Fixed ConvertColorToGrayScale parallelization code for out-of-core datastores.

imikejackson force-pushed the feature/Solomon_OOC branch from 9dc9c4f to 4ab6dc0 Compare November 8, 2023 22:05

Fix compile error and other small tweaks.

fa3970a

Signed-off-by: Michael Jackson <mike.jackson@bluequartz.net>

imikejackson merged commit 5d12251 into BlueQuartzSoftware:develop Nov 10, 2023
7 checks passed

imikejackson deleted the feature/Solomon_OOC branch November 10, 2023 17:04

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Enable Out-of-Core #703

Enable Out-of-Core #703

mmarineBlueQuartz commented Sep 26, 2023 •

edited

Loading

joeykleingers left a comment

JDuffeyBQ left a comment

imikejackson Nov 6, 2023

imikejackson Nov 6, 2023

imikejackson Nov 6, 2023

Enable Out-of-Core #703

Enable Out-of-Core #703

Conversation

mmarineBlueQuartz commented Sep 26, 2023 • edited Loading

Naming Conventions

Filter Checklist

Unit Testing

Code Cleanup

joeykleingers left a comment

Choose a reason for hiding this comment

JDuffeyBQ left a comment

Choose a reason for hiding this comment

imikejackson Nov 6, 2023

Choose a reason for hiding this comment

imikejackson Nov 6, 2023

Choose a reason for hiding this comment

imikejackson Nov 6, 2023

Choose a reason for hiding this comment

mmarineBlueQuartz commented Sep 26, 2023 •

edited

Loading