Dataobject rework - future RAVEN v 1.1 #598

alfoa · 2018-03-21T16:25:46Z

Pull Request Description

What issue does this change request address? (Use "#" before the issue to link it, i.e., #42.)

Closes #182
Closes #363
Closes #225
Closes #568 (obsolete with respect to the new DataObject)
Closes #56 (obsolete with respect to the new DataObject)
Closes #112
Closes #589
Closes #319
Closes #305
Closes #252
Closes #573
Closes #551
Closes #73
Closes #551 (overcome by establish_conda_env.sh script)
Closes #627
Closes #258
Closes #129
Closes #93
Closes #91
Closes #83
Closes #77
Closes #58
Closes #43
Closes #68

What are the significant changes in functionality due to this change request?

This PR addresses multiple Issues.
The following new features have been added:

new Data object structure
addition of DataSet class
standardization of PostProcessors' outputs
New ROMs (time-dependent)
Possibility to handle vector input spaces (not just scalars)
Degradation problems with printing solved
EnsembleModel for Unstructured Input handling
Reached 100% tests' documentation
New Code interfaces (e.g. SCALE)
Library update

THIS PULL REQUEST (AND THE NEW DEVEL) WILL BE THE RAVEN V 1.1 RELEASE

For Change Control Board: Change Request Review

The following review must be completed by an authorized member of the Change Control Board.

1. Review all computer code.
2. If any changes occur to the input syntax, there must be an accompanying change to the user manual and xsd schema. If the input syntax change deprecates existing input files, a conversion script needs to be added (see Conversion Scripts).
3. Make sure the Python code and commenting standards are respected (camelBack, etc.) - See on the wiki for details.
4. Automated Tests should pass, including run_tests, pylint, manual building and xsd tests. If there are changes to Simulation.py or JobHandler.py the qsub tests must pass.
5. If significant functionality is added, there must be tests added to check this. Tests should cover all possible options. Multiple short tests are preferred over one large test. If new development on the internal JobHandler parallel system is performed, a cluster test must be added setting, in XML block, the node <internalParallel> to True.
6. If the change modifies or adds a requirement or a requirement based test case, the Change Control Board's Chair or designee also needs to approve the change. The requirements and the requirements test shall be in sync.
7. The merge request must reference an issue. If the issue is closed, the issue close checklist shall be done.
8. If an analytic test is changed/added is the the analytic documentation updated/added?

* dummy, external model both nominally working * test_Lorentz needs to pass for externalModel; however, we need hdf5 to work as output first * test_Lorentz runs, using ExternalModel, so external model and dummy both running for hist and point set

…into alfoa/dataobject-rework

…taobject-rework

Loading dataset fully from dict

…into alfoa/dataobject-rework

outstreams and performance -> dataobject-rework

* works, but ProbabilityWeight is not always provided by the sampler, like in test_Lorentz using MC * merging in developments * stash * addMetaKeys works, proved with Samplers adding ProbabilityWeight * cleanup

…into dataobject-rework

fixed write...but the history set does not work yet

…into dataobject-rework

…/raven into alfoa/dataobject-rework

…dered and consequentially a spurious diff can happen

…m_exponential PolyExponential, Spline and DMD ROMs

* expand install script for conda 4.4 and beyond * added explanatory comments

* Optimizer inherits from Sampler * first implementation: by default copy value to all entries in vector variable, works * finished test and implementation of simple repeat-value vector variable sampling * added InputSpecs for optimizer, tests pass * got input params working for optimizer * first implementation: by default copy value to all entries in vector variable, works * finished test and implementation of simple repeat-value vector variable sampling * added InputSpecs for optimizer, tests pass * got input params working for optimizer * stash * fixed gradient calculation to include vectors, all non-vector tests passing * fixed gradient calculation to include vectors, all non-vector tests passing, conditional sizing for vector grad * boundary condition checking, all passing * redundant trajectories, all passing * same coordinate check * dot product step sizing * stochastic engine is incorrectly sized; currently each entry in vector is being perturbed identically. Needs work. * working on constraints, convergence is really poor, needs more help * first boundary conditions (internal) working, although type change in precond test * constraints fully done, only precond has a problem still, vector still not converging well * debugging difference between all scalars and vector * vector * time parabola model * fixed initial step size * working, although as a vector is a bit slower than all scalars * vector is faster than scalar, reduced scale of tests (and better solution) * all passing, but precond, which is having the type error still * cleaned up, removed scalar comparison test, fixed precond test * cleanup * last bit of cleanup, all tests passing * stash, it appears customsampler and datasets are not yet compatible * xsd * stash, <what> cannot handle specific requests * reloading from dataset csv works be default * fixed unit test, vector test * xsd * CustomSampler handles Point,History,Data sets * cleanup * cleanup * updated custom sampler description docs * Optimizer uses Custom sampler with vector variables for initial points * unnecessarily-tested file * initial round of review comments * script for disclaimer adding, also added to models in optimizing test dir * increased verbosity for test debug * more verbosity for debugging * gold standard agrees with all test machines, personal cluster profile (my desktop find minimum in traj 1 of 36 instead of 220ish) * new golds

* exposed RNG to RAVEN python...swig * fixed for now dist stoch enviroment

* shape from node to attribute * constants can now be vectors too * necessary Sampler and Optimizer changes * extracted common constant reading for sampler, optimizer * including string custom vector vars * vector constant works in rrr with optimizer

derekstucki

Finishing a review part to make sure it's placing comments in the right places.

derekstucki · 2018-05-03T17:03:36Z

backend_run_tests

+  try:
+    interpolationND = utils.findCrowModule('randomENG')
+    print("randomENG","\n",randomENG)
+  except:


Does the "no except without an exception" rule apply to this file?

derekstucki · 2018-05-03T17:19:42Z

framework/DataObjects/DataSet.py

+    else:
+      # need the sampleTag meta to load histories
+      # BY DEFAULT keep everything needed to reload this entity.  Inheritors can define _neededForReload to specify what that is.
+      keep = set(self._inputs + self._outputs + self._metavars + self._neededForReload


~~1804~~ 1840: Is this [:] needed?

Is this noted on the right line?

Nope, 1840 instead of 1804. That's what I get for trusting my memory.

I don't think it's needed. Removed.

derekstucki · 2018-05-03T17:23:15Z

Any comment I leave in DataSet.py after about line 1400 is placed near line 1400 on this page and not placed at all when viewing the files changed view. I will put line numbers in all my comments, but that is going to be really annoying for anyone reading my review. Any suggestions on how to fix this would be helpful.

derekstucki

Completing partial review to make sure comments in other files go where they should.

derekstucki · 2018-05-03T17:34:00Z

framework/DataObjects/DataSet.py

+    else:
+      # need the sampleTag meta to load histories
+      # BY DEFAULT keep everything needed to reload this entity.  Inheritors can define _neededForReload to specify what that is.
+      keep = set(self._inputs + self._outputs + self._metavars + self._neededForReload


1842: This seems like a lot of conversion steps. Can it be reduced?

Line in question, for reference:

fromColl = list( dict(zip(self._orderedVars,c)) for c in fromColl )

I need a list of realizations that contain hierarchal path endings, where each realization is a dictionary mapping variable names to values for that ending. I'm not sure if there'a a more abbreviated way to do this here, but I'm open to ideas.

derekstucki · 2018-05-03T17:40:37Z

framework/DataObjects/Factory.py

@@ -26,7 +26,8 @@

 ################################################################################
 from utils import utils
-from DataObjects.Data import Data
+from DataObjects.DataObject import DataObject as Data


I imagine this is imported as Data in order to avoid a bunch of name changes throughout the file. Would it be better long term to just make all those changes? A search and replace would make very quick work of it.

All references already removed; so I trivially removed the alias now. Fixed.

derekstucki · 2018-05-03T17:41:45Z

framework/DataObjects/HistorySet.py


-#External Modules------------------------------------------------------------------------------------
+import sys,os


Better separated.

derekstucki · 2018-05-03T17:46:12Z

framework/DataObjects/HistorySet.py

+  __builtin__.profile
+except AttributeError:
+  # profiler not preset, so pass through
+  def profile(func):


This try/except seems off. If __builtin__.profile exists, then it still needs to be called as __builtin__.profile, while if it doens't, profile will exist in the main name space. If profile is already in the main name space from __builtin__, why bother importing __builtin__ to test it instead of just testing the one in the main name space? Am I missing something?

See the accepted answer to this inquiry, where I got it from.

What is in that answer is exactly what I would expect here, but isn't what is here. Either the try needs to read "profile = __builtin__.profile", or the except needs to have "__builtin__.profile = profile". As written, profile is in one of two different namespaces depending on whether the try fails or not. Unless I'm missing something.

I see. Fixed using your first suggestion.

* pre-merge review comments addressed for modules in framework/DataObjects, with the exception of merging DataObject into DataSet * removed hierarchal unecessary use of [:] * remainder of comments addressed

alfoa · 2018-05-17T16:39:45Z

For Change Control Board: Change Request Review

The following review must be completed by an authorized member of the Change Control Board.

1. Review all computer code. THE REVIEW HAS BEEN PERFORMED BY ALL DEVELOPERS
2. If any changes occur to the input syntax, there must be an accompanying change to the user manual and xsd schema. If the input syntax change deprecates existing input files, a conversion script needs to be added (see Conversion Scripts). DONE
3. Make sure the Python code and commenting standards are respected (camelBack, etc.) - See on the wiki for details. OK
4. Automated Tests should pass, including run_tests, pylint, manual building and xsd tests. If there are changes to Simulation.py or JobHandler.py the qsub tests must pass. OK
5. If significant functionality is added, there must be tests added to check this. Tests should cover all possible options. Multiple short tests are preferred over one large test. If new development on the internal JobHandler parallel system is performed, a cluster test must be added setting, in XML block, the node <internalParallel> to True. ALL THE TESTS HAVE BEEN REWORKED
6. If the change modifies or adds a requirement or a requirement based test case, the Change Control Board's Chair or designee also needs to approve the change. The requirements and the requirements test shall be in sync. OK. ALFOA APPROVES!
7. The merge request must reference an issue. If the issue is closed, the issue close checklist shall be done. SEVERAL ISSUES ARE ADDRESSED HERE
8. If an analytic test is changed/added is the the analytic documentation updated/added? NO OUTCOME DIFFERENCES

We addressed what he asked for or postpone the comments to a subsequential PR

PaulTalbot-INL

Targets met, and both internal and external reviews passed. Tests passing. Approving for merge.

alfoa and others added 30 commits November 27, 2017 12:26

improved speed

3be8774

Talbpaul/dummy output (#438)

819e39b

* dummy, external model both nominally working * test_Lorentz needs to pass for externalModel; however, we need hdf5 to work as output first * test_Lorentz runs, using ExternalModel, so external model and dummy both running for hist and point set

Merge branch 'dataobject-rework' of https://github.com/idaholab/raven …

e387ed4

…into alfoa/dataobject-rework

split formatting versus selective realization

3231f54

Merge branch 'dataobject-rework' of github.com:idaholab/raven into da…

69c0150

…taobject-rework

removed call from XPoint to XData for formatting rlz

a723849

checking devel

ec0338a

typo

34ea50d

loading from dictionary

85c5b13

looks like it's working and tested

f939058

cleanup

7469ff5

review comments

5211026

Merge pull request #441 from idaholab/talbpaul/dataset-from-dict

df9b669

Loading dataset fully from dict

Fixed outstreams and performance in printing

d0af30d

Merge branch 'dataobject-rework' of https://github.com/idaholab/raven …

d03fd90

…into alfoa/dataobject-rework

added comment

a3d4575

Merge pull request #442 from idaholab/alfoa/dataobject-rework

1ea5fc3

outstreams and performance -> dataobject-rework

Talbpaul/register metadata (#443)

6c55cc0

* works, but ProbabilityWeight is not always provided by the sampler, like in test_Lorentz using MC * merging in developments * stash * addMetaKeys works, proved with Samplers adding ProbabilityWeight * cleanup

Merge branch 'dataobject-rework' of https://github.com/idaholab/raven …

69ad9b3

…into dataobject-rework

fixed write...but the history set does not work yet

98cba88

Merge pull request #444 from idaholab/alfoa/dataobject-rework

15897e2

fixed write...but the history set does not work yet

fixed return code

9f99821

fixed return code (#445)

e826b3b

Merge branch 'dataobject-rework' of https://github.com/idaholab/raven …

1cc2f1f

…into dataobject-rework

unit tests passing again

eeecaaf

slicing now works

07e1e38

added abstract methods to object base

a194254

starting reworking database

5619cb4

Merge branch 'alfoa/dataobject-rework' of https://github.com/idaholab…

d7fc235

…/raven into alfoa/dataobject-rework

histories working:

d5b6414

alfoa and others added 9 commits April 27, 2018 00:47

remove xml checker for DMD since the eigenvalues are not necessary or…

08382ec

…dered and consequentially a spurious diff can happen

type

4604314

added importe of differential_evolution only where is required

d0a83e4

modified tests

2b1694e

Merge pull request #579 from idaholab/alfoa/dataobject-rework_poly_su…

1d8fd12

…m_exponential PolyExponential, Spline and DMD ROMs

expand install script for conda 4.4 and beyond (#618)

6ed1420

* expand install script for conda 4.4 and beyond * added explanatory comments

exposed RNG to RAVEN python...swig (#630)

7387c4e

* exposed RNG to RAVEN python...swig * fixed for now dist stoch enviroment

Vector constants (#632)

00cf3de

* shape from node to attribute * constants can now be vectors too * necessary Sampler and Optimizer changes * extracted common constant reading for sampler, optimizer * including string custom vector vars * vector constant works in rrr with optimizer

derekstucki suggested changes May 3, 2018

View reviewed changes

derekstucki previously requested changes May 3, 2018

View reviewed changes

PaulTalbot-INL and others added 4 commits May 16, 2018 09:33

pre-merge review comments addressed: framework/DataObjects (#646)

0a30e9f

* pre-merge review comments addressed for modules in framework/DataObjects, with the exception of merging DataObject into DataSet * removed hierarchal unecessary use of [:] * remainder of comments addressed

Conflicts fixed

d5f3d99

Merge branch 'devel' into dataobject-rework

46c0b57

modified test for dataobject rework

9153a35

alfoa mentioned this pull request May 17, 2018

Issues need to be solved after dataobject-rework gets merged #655

Closed

10 tasks

alfoa added 2 commits May 17, 2018 00:19

added documentations for the last 29 tests were not documented

d69ac23

ok

a7d03ab

alfoa changed the title ~~Dataobject rework - preparation of merge~~ Dataobject rework - future RAVEN v 1.1 May 17, 2018

type

5812f6b

alfoa added 4 commits May 17, 2018 11:25

Closes #68

39bc740

added verbosity in all the XSD s

2589141

fixed None global attribute for verbosity setting

c72dada

fixed BaseClass if statment in checking for global attribute argument

a5ad049

PaulTalbot-INL approved these changes May 17, 2018

View reviewed changes

alfoa merged commit 688cdd1 into devel May 17, 2018

alfoa deleted the dataobject-rework branch July 20, 2018 17:24

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Dataobject rework - future RAVEN v 1.1 #598

Dataobject rework - future RAVEN v 1.1 #598

alfoa commented Mar 21, 2018 •

edited

Loading

derekstucki left a comment

derekstucki May 3, 2018

alfoa May 17, 2018

derekstucki May 3, 2018 •

edited

Loading

PaulTalbot-INL May 3, 2018

derekstucki May 3, 2018

PaulTalbot-INL May 3, 2018

derekstucki commented May 3, 2018

derekstucki left a comment

derekstucki May 3, 2018

PaulTalbot-INL May 8, 2018

derekstucki May 3, 2018

PaulTalbot-INL May 8, 2018

derekstucki May 3, 2018

PaulTalbot-INL May 8, 2018

derekstucki May 3, 2018

PaulTalbot-INL May 3, 2018

derekstucki May 4, 2018 •

edited

Loading

PaulTalbot-INL May 8, 2018

alfoa commented May 17, 2018

PaulTalbot-INL left a comment


		#External Modules------------------------------------------------------------------------------------
		import sys,os

Dataobject rework - future RAVEN v 1.1 #598

Dataobject rework - future RAVEN v 1.1 #598

Conversation

alfoa commented Mar 21, 2018 • edited Loading

Pull Request Description

What issue does this change request address? (Use "#" before the issue to link it, i.e., #42.)

What are the significant changes in functionality due to this change request?

For Change Control Board: Change Request Review

derekstucki left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

derekstucki May 3, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

derekstucki commented May 3, 2018

derekstucki left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

derekstucki May 4, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

alfoa commented May 17, 2018

For Change Control Board: Change Request Review

PaulTalbot-INL left a comment

Choose a reason for hiding this comment

alfoa commented Mar 21, 2018 •

edited

Loading

derekstucki May 3, 2018 •

edited

Loading

derekstucki May 4, 2018 •

edited

Loading