Talbpaul/register metadata #443
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…ike in test_Lorentz using MC
…/register-metadata
…lbpaul/register-metadata
…/register-metadata
wangcj05
requested changes
Nov 28, 2017
framework/Models/ExternalModel.py
Outdated
| @@ -250,10 +250,12 @@ def evaluateSample(self, myInput, samplerType, kwargs): | |||
| """ | |||
| Input = self.createNewInput(myInput, samplerType, **kwargs) | |||
| inRun = copy.copy(self._manipulateInput(Input[0][0])) | |||
| print('DEBUG inrun:',inRun) | |||
wangcj05
approved these changes
Nov 28, 2017
PaulTalbot-INL
pushed a commit
that referenced
this pull request
Mar 12, 2018
* new data objects and units tests * supporting structure * touching up * addVariable classifier * pptx for converstion * ppt work * more ppt touchups * typo * prepping to merge in devel * relative import fix * pausing to work on outstreams * test_Grid_Sampler.xml works * initial edits * fixed OutStreams Print * abstract API methods added to base class * improved speed * Talbpaul/dummy output (#438) * dummy, external model both nominally working * test_Lorentz needs to pass for externalModel; however, we need hdf5 to work as output first * test_Lorentz runs, using ExternalModel, so external model and dummy both running for hist and point set * split formatting versus selective realization * removed call from XPoint to XData for formatting rlz * checking devel * typo * loading from dictionary * looks like it's working and tested * cleanup * review comments * Fixed outstreams and performance in printing * added comment * Talbpaul/register metadata (#443) * works, but ProbabilityWeight is not always provided by the sampler, like in test_Lorentz using MC * merging in developments * stash * addMetaKeys works, proved with Samplers adding ProbabilityWeight * cleanup * fixed write...but the history set does not work yet * fixed return code (#445) * unit tests passing again * slicing now works * added abstract methods to object base * histories working: * switched to comprehension * fname to fName * removing variables works * added addReal check * Correct setting of default pivots for history set (#448) * fixed default pivot param setting to be in order * put 'time' back in since we handle that now * edits * progressive CSV writing with appends instead of rewriting each iteration for multiruns * cleanup * check data alignment works (#452) * fixed no-scalar bug, albeit in a non-traditional way. (#455) * Alfoa/dataobject rework (#453) * speeded up group search and allgroup list * finished h5py_interface * working on IOStep * finished database * finished not hirearchical HDF5 and fixed static ROMs * fixed time dep ROMs * pylint problems addressed (#451) * pylint problems addressed * Update TestXPointSet.py * malformed comment blocks in 2 tests * new unordered CSV differ, tested on test_Lorentz * convert code to use new data objects (#457) * initial convert of code for new data object * fix bugs in dataset and continue convert code for accepting the new data object * additional converting * clean up * Sampler Restarts (#458) * started work * sampler restarts working * Adaptive Samplers (plus Dummy IO fix) (#459) * adaptive sparse grid, sobol working * Adaptive sampling, plus Dummy-based rlz updates from input, output * cleanup * fixed prefix as numpy array * convert basic statistics pp to use the new data object (#460) * convert basicStatistics to use the new data objects * convert tests of basicStatisitics * convert more tests * clean up * move addMetaKeys to localInputAndChecks * resolve comments * fix checkIndexAlignment in DataSet * add unit test for checkIndexAlignment * convert metric pp to use new data objects (#462) * convert metric pp to use new data objects * clean up * Mods for new dataObject (#463) * edits * edits * comments review-1 * edits * asDictionary works * added full loop cycle test * dataobject rework: Optimizer (#461) * adaptive sparse grid, sobol working * Adaptive sampling, plus Dummy-based rlz updates from input, output * cleanup * added cluster labeling to outstream print, will be tested with optimizer * optimizers working, but need to fix reading from CSV before merging * stash * fixed prefix as numpy array * loading CSV correctly now, so optimizer can find optimal solns * cleanup * now handling infs and nans * cleanup * added the option to avoid to reprint the files in case their content did not change * reverted 101 tests modified by the conversion scripts (by mistake) * reverted other tests * reverted tests user guide * reverted all the other tests * Update beale.xml removed ```"``` * removed " * removed abstract method that has been removed from the XDataSet * fixed with respect to the new prefix strategy * fixed loading of dataset in case no metadata are found in the xml (finally statement in a try except is always executed) * fixed type in ROM causing a failure * fixed another typo that was making the Dummy.py model to fail * removed whitespace that was added from another commit * updated to use pd.isnull instead of np.isnan in unordered CSV differ * test files for differ * convert CrossValidation PP to use new data objects. (#470) * fix typo in rom * modify tests * convert cross validation pp * convert tests for cross validation pp * clean up * keep the options to dump the cross validation results for each fold * update conversion scripts for cross validation * Fixing no-meta CSV reading (#473) * fixed history set reading from CSV if no XML present, or if present for that matter * fixed up PointSet and DataSet versions, too. * Doing better than "object" typing in DataObject (#478) * typing is now working, but if more samples are taken after asDataset is called, and asDataset is called again, integers will be converted into floats. I submitted a stackOverflow question about this, and we will see. * fixed integer preservation using concat instead of merge * added hierarchical methods (#474) * added hierarchical methods * fixed multiline in databases * added sanity assertion * stash * mechanics in place for hierarchical * hierarchal working with collector OR data * a little more testing * D.O. Rework: Topological PP (#472) * stash * stash * stash * topologicals working * cleanup * review comments * convert importance rank pp to use the new data objects (#477) * convert importance rank pp to use the new data objects * fix comments * convert tests (#480) * LimitSurface adaptive sampler (Standard, Batch, MultiGrid), etc (#469) * fixed return code * starting reworking database * ok * ok * ok * ok * ok * moving ahead * speeded up group search and allgroup list * fixed subgroup * almost done * finished h5py_interface * working on IOStep * dfloat * ok finished database * finished not hirearchical HDF5 and fixed static ROMs * fixed time dep ROMs * addressed Paul's comments * fixed plot * fixed other 2 tests * fixed other 3 tests * regolded another test * initial edits * fixed write...but the history set does not work yet * fixed return code * Update README.md (#439) just removed few sentences * unit tests passing again * slicing now works * added abstract methods to object base * starting reworking database * histories working: * switched to comprehension * fname to fName * removing variables works * added addReal check * Correct setting of default pivots for history set (#448) * fixed default pivot param setting to be in order * put 'time' back in since we handle that now * ok * ok * ok * ok * edits * ok * moving ahead * speeded up group search and allgroup list * fixed subgroup * progressive CSV writing with appends instead of rewriting each iteration for multiruns * cleanup * almost done * finished h5py_interface * working on IOStep * dfloat * ok finished database * finished not hirearchical HDF5 and fixed static ROMs * fixed time dep ROMs * check data alignment works (#452) * fixed no-scalar bug, albeit in a non-traditional way. (#455) * addressed Paul's comments * pylint problems addressed (#451) * pylint problems addressed * Update TestXPointSet.py * malformed comment blocks in 2 tests * new unordered CSV differ, tested on test_Lorentz * convert code to use new data objects (#457) * initial convert of code for new data object * fix bugs in dataset and continue convert code for accepting the new data object * additional converting * clean up * Sampler Restarts (#458) * started work * sampler restarts working * Adaptive Samplers (plus Dummy IO fix) (#459) * adaptive sparse grid, sobol working * Adaptive sampling, plus Dummy-based rlz updates from input, output * cleanup * fixed prefix as numpy array * convert basic statistics pp to use the new data object (#460) * convert basicStatistics to use the new data objects * convert tests of basicStatisitics * convert more tests * clean up * move addMetaKeys to localInputAndChecks * resolve comments * fix checkIndexAlignment in DataSet * add unit test for checkIndexAlignment * Closes #464 * added test * convert metric pp to use new data objects (#462) * convert metric pp to use new data objects * clean up * fixed the tests * added the new gold file for the new test * Mods for new dataObject (#463) * edits * edits * comments review-1 * edits * asDictionary works * added full loop cycle test * fixed plot * fixed other 2 tests * fixed other 3 tests * Update TestXDataSet.py * Update Dummy.py * other tests converted * readded deleted file * almost done * ok * fixed tests * Fixed LimitSurface Postprocessors, added conversion script, added documentation * ok * adapted LimitSurfaceSearch sampler and moved directory for more clearity * almost finished safest point * ok * fixed safest point PP * fixed adaptive batch * modified documentation for SafestPoint * removed Outputhold for Optimizer since is not used and not working * fixed metadata fro samples * removed all * modified unit test * addressed Diego's comments * convert tests (#480) * removed commented part in user manual * addrresed Conjiang's * remove .DS_Store * convert external post processor to use new DataObjects (#479) * convert external pp to use the new data objects * regold tests * fix the DataObjects * address comments * starting reworking database * ok * ok * finished h5py_interface * working on IOStep * dfloat * ok finished database * finished not hirearchical HDF5 and fixed static ROMs * addressed Paul's comments * fixed plot * fixed other 2 tests * fixed other 3 tests * regolded another test * unit tests passing again * slicing now works * starting reworking database * histories working: * removing variables works * ok * ok * progressive CSV writing with appends instead of rewriting each iteration for multiruns * cleanup * finished h5py_interface * working on IOStep * dfloat * ok finished database * finished not hirearchical HDF5 and fixed static ROMs * check data alignment works (#452) * fixed no-scalar bug, albeit in a non-traditional way. (#455) * pylint problems addressed (#451) * pylint problems addressed * Update TestXPointSet.py * malformed comment blocks in 2 tests * new unordered CSV differ, tested on test_Lorentz * Adaptive Samplers (plus Dummy IO fix) (#459) * adaptive sparse grid, sobol working * Adaptive sampling, plus Dummy-based rlz updates from input, output * cleanup * fixed prefix as numpy array * convert basic statistics pp to use the new data object (#460) * convert basicStatistics to use the new data objects * convert tests of basicStatisitics * convert more tests * clean up * move addMetaKeys to localInputAndChecks * resolve comments * fix checkIndexAlignment in DataSet * add unit test for checkIndexAlignment * Closes #464 * convert metric pp to use new data objects (#462) * convert metric pp to use new data objects * clean up * fixed the tests * fixed other 2 tests * other tests converted * readded deleted file * almost done * ok * fixed tests * Fixed LimitSurface Postprocessors, added conversion script, added documentation * ok * adapted LimitSurfaceSearch sampler and moved directory for more clearity * almost finished safest point * ok * fixed safest point PP * fixed adaptive batch * modified documentation for SafestPoint * removed Outputhold for Optimizer since is not used and not working * fixed metadata fro samples * addressed Diego's comments * removed commented part in user manual * addrresed Conjiang's * remove .DS_Store * convert external post processor to use new DataObjects (#479) * convert external pp to use the new data objects * regold tests * fix the DataObjects * address comments * addressed Congjiang's comments * Step forward (#482) * merged Diego's commits for InterfacePostProcessor * Fixed OutStreams printing and regolder foulty ROMs results * modified ARMA tests * fixed reading from CSV of HistorySet (correctly considering the pivotParameter) * fixed * TEMPORARY fix of printing of HISTORY SET * fixed ROM, ARMA, DATABASE (loader), Multiple usage of database * fixed multi target rom and rom trainer test * added the possibility to ask for unfolded XArrays in the realization(...) method * ok * modified other two tests * addressed Paul and Congjiang's comments (no interface PP related) * removed comment that does no apply anymore in HistorySet * fixed another test database * addressed comments * addressed comments 2 * edits * fixed test file * fixed tests (#487) * Mandd/fixed test HS2PS (#488) * fixed tests * edits * edits * edits * Reworked cNDarray, no more replicating indexes (#484) * added a synch checking that is too slow * stash * stash before merging in develop branch * stash * moving back to desktop * got types back * all types including histories now preserved * point set now fixed * dummy fix, mergefix * whitespace * review comment cleanup * Alfoa/continuing work (#486) * regolded XML and added relative tollerance * added warning if no output space has been requested in OutStream Print * Update XHistorySet.py * added selective HDF5 * regolded XML and added relative tollerance * added warning if no output space has been requested in OutStream Print * Update XHistorySet.py * added selective HDF5 * typo endif * fixed HDF5 selective * working on the alias system * fixed alias system tests (but can not be really fixed since the printing of the history set is in random order) * fixed alias system and random history set printing * fixed other 2 tests * Fix the probability weight for ND and 1D distributions (#483) * add regold files * fix probability weight of each sampled variable * address comments * Alfoa/continuing work (#492) * regolded XML and added relative tollerance * added warning if no output space has been requested in OutStream Print * Update XHistorySet.py * added selective HDF5 * regolded XML and added relative tollerance * added warning if no output space has been requested in OutStream Print * Update XHistorySet.py * added selective HDF5 * typo endif * fixed HDF5 selective * working on the alias system * fixed alias system tests (but can not be really fixed since the printing of the history set is in random order) * fixed alias system and random history set printing * fixed other 2 tests * fixed a test * fixed database directory creation (#493) * fixed database directory creation * Update h5py_interface_creator.py removed print statemtnt * Update ExternalModel.py * almost finished ensemble * fix pca related tests. (#496) * fix pca_LHS tests * fix probability weight for sobol and sparse grid related samplers, plus fix the corresponding tests * remove unused gold files * fix ND_test_MC_MVN * fix test Metrics.simpleMetric * fix test EnsembledSampler.EnsembleSamplerConstants * fix probability weight for discrete dist * Alfoa/dataobject rework ensemble model (#498) * Modifications of the ensemble model that fix 8 tests over 12 * fixed test_distribution_combination.xml * added seed in test_distribution_combination.xml and regolded * fixed tests (#490) * fixed tests for TypicalHistory * Mandd/e timporter issues (#501) * fixed tests * edits * edits * edits * edits * edits * edits * edits * edits * edits * edit * edit * edit * edit (#502) * Alfoa/go on ensemble model (#504) * fixed test_output and test_merge_2_databases * fixed hybrid * fixed parser error in tests file * Fixing Tests 1 (#494) * data objects metadata, load csv and history fixed * xml validates * restart csv, mc fixed * improved optimizer infinite test * disabled RavenOutput postprocessor * fixed grid with constants test * cleanup * Add variable probability weights in the Custom Sampler (#505) * add variable probability weight for custom sampler, this will fix the BasicStatistics.timeDependent test * convert test BasicStatistics.timeDependent * fix other tests * fix test framework/Distributions.categorical * add comments * fix comparison statistics (#506) * fix IOStep DataObject to Database * fix comparison statistics * update test test_calc_and_transfer.xml and regold * Non-multilevel Optimizers reworked (#507) * optimizers passing with the new DataObjects except multilevel * Alfoa/data object rework (#509) * fixed CustomSampler * fixed test_output for new version of matplotlib * Closes #517 * small fix in the manual * updated MOOSE id * fixed run_tests * Framework in makefile (#520) * Added hit to all requirements and clean * formatting * Different approach * Now bindings instead of binary * Moves hit.so to usable location * fixed another parsing error in tests file for framework/PostProcessors/TemporalDataMiningPostProcessor/Clustering.GaussianMixture * removed commented code * added new ID of moose with fix on TestHarness * fixed make file for windwos * Revert "Merge branch 'dataobject-rework' into alfoa/testFix" This reverts commit 9840e3e, reversing changes made to adda3cd. * Update run_tests * Added setup.py and a build script for hit. * Added script to makefile. * Dirty hack to get around civet. * Continuation of dirty hack to get around civet. * Testing some verbosity. * Trying cp instead of mv. * More debugging messages. * More verbosity. * Trying different build behavior based on OS. * Yet another way. * Trying to duplicate known good build for linux. * Corrected name of env variable to test for windows. * Revert "More verbosity." Commit only added debugging messages. This reverts commit 32c8a74. * Revert "More debugging messages." Commit only added debugging messages. This reverts commit 05b52db. * Exit code grabbing on hit build. * Added version check for xarray. * Fixed typo. * Specified pandas version. * Updated moose to get fix for hit code. * Revert "Specified pandas version." This reverts commit fe354d7. * Update RavenUtils.py removed xarray and netcdf dependency....this should go on dataobject-rework
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Pull Request Description
Adds a registration system for keeping pointwise metadata.
Every RAVEN entity can call
self.addMetaKeysto register pointwise metadata keys. If a variable in a realization isn't an input, output, or registered metadata, the data object will ignore it.