LSTM Regression #1519

joshua-cogliati-inl · 2021-04-19T15:25:00Z

Pull Request Description

What issue does this change request address?

Closes #1540

What are the significant changes in functionality due to this change request?

Adds a LSTM regression (we have an existing LSTM classifier)

For Change Control Board: Change Request Review

The following review must be completed by an authorized member of the Change Control Board.

1. Review all computer code.
2. If any changes occur to the input syntax, there must be an accompanying change to the user manual and xsd schema. If the input syntax change deprecates existing input files, a conversion script needs to be added (see Conversion Scripts).
3. Make sure the Python code and commenting standards are respected (camelBack, etc.) - See on the wiki for details.
4. Automated Tests should pass, including run_tests, pylint, manual building and xsd tests. If there are changes to Simulation.py or JobHandler.py the qsub tests must pass.
5. If significant functionality is added, there must be tests added to check this. Tests should cover all possible options. Multiple short tests are preferred over one large test. If new development on the internal JobHandler parallel system is performed, a cluster test must be added setting, in XML block, the node <internalParallel> to True.
6. If the change modifies or adds a requirement or a requirement based test case, the Change Control Board's Chair or designee also needs to approve the change. The requirements and the requirements test shall be in sync.
7. The merge request must reference an issue. If the issue is closed, the issue close checklist shall be done.
8. If an analytic test is changed/added is the the analytic documentation updated/added?
9. If any test used as a basis for documentation examples (currently found in raven/tests/framework/user_guide and raven/docs/workshop) have been changed, the associated documentation must be reviewed and assured the text matches the example.

for Prescient requirements.

It makes the training easier for the RNN.

moosebuild · 2021-04-19T16:43:24Z

Job Test mac on c2647a1 : invalidated by @joshua-cogliati-inl

restarted civet

moosebuild · 2021-04-19T16:52:43Z

Job Mingw Test on c2647a1 : invalidated by @joshua-cogliati-inl

restarted civet

wangcj05 · 2021-04-19T19:42:52Z

framework/SupervisedLearning/KerasRegression.py

+from .SupervisedLearning import supervisedLearning
+#Internal Modules End--------------------------------------------------------------------------------
+
+class KerasRegression(supervisedLearning):


I think this class need to be merged with the KerasClassifier class. Most of the functionalities are the same for both Regression and Classifier. I think a new base class can be extracted from these two classes.

I completely agree (this is one of the reasons this is marked Do Not Merge). My plan is to make a KerasBaseClass, and put the common things in KerasRegression and KerasClassifier in it.

I have created a base class and pulled some common features out to it. I am still deciding if I need to work a bit more to get more common functions out or not.

I think I have extracted pretty much all of the common functionality into a base class. So if you want to take a look now, it would be welcome.

moosebuild · 2021-04-28T20:33:25Z

Job Mingw Test on e4a31fa : invalidated by @joshua-cogliati-inl

restarted civet

moosebuild · 2021-05-03T15:27:40Z

Job Test Fedora 31 on dfad1fa : invalidated by @joshua-cogliati-inl

failed tests/framework/ensembleModelTests/testEnsembleModelLateralInsertions

joshua-cogliati-inl · 2021-05-04T17:25:44Z

@wangcj05 or @alfoa This is ready for more review. Thanks.

wangcj05

Some minor comments for your consideration. It is a good start to create the base class for neural network.

framework/SupervisedLearning/KerasBase.py

framework/SupervisedLearning/KerasClassifier.py

framework/SupervisedLearning/KerasRegression.py

framework/SupervisedLearning/__init__.py

tests/framework/ROM/tensorflow_keras/keras_lstm_regression.xml

wangcj05 · 2021-05-04T20:29:06Z

When I compared the KerasLSTMClassifier and KerasLSTMRegression, they share most of the functions. This suggests to me it is probably better to combine these two classes. I suggest we have group discussion. @alfoa @joshua-cogliati-inl

joshua-cogliati-inl · 2021-05-05T19:45:02Z

framework/SupervisedLearning/KerasLSTMRegression.py

+          self.initOptionDict[layerName]['return_sequences'] = True
+          self.raiseAWarning('return_sequences is resetted to True for layer',layerName)
+
+  def _preprocessInputs(self,featureVals):


@wangcj05 Hm, I think I could move _preprocessInputs to KerasBase, if you agree that would be a good idea. I think that removes the major redundancy in KerasLSTMRegression and KerasLSTMCLassifier.

I think this is tied to LSTM. It is not general all neural network.

Hm, then I could just make a LSTM utils, and put the function in that, and use it in both.

Tho' since @alfoa is going to be making changes to this soon, maybe I should just put a comment in both files that it is identical, and after the changes we can decide if it is worth changing.

To me, a comment is fine. I think my comments are more for future. How do we want the DNN classes inheritance look like? @alfoa any more comments? BTW, the issue that I raised can be further discussed and addressed in future PR.

Okay, I added a comment to the source code.

wangcj05 · 2021-05-10T16:45:55Z

@joshua-cogliati-inl qsubs failed, could you take a look?

moosebuild · 2021-05-10T16:56:12Z

Job Test qsubs on 9b7b418 : invalidated by @joshua-cogliati-inl

Failed tests/framework/Samplers/Categorical/RestartMissingVars

moosebuild · 2021-05-10T17:41:43Z

Job Test qsubs on 9b7b418 : invalidated by @joshua-cogliati-inl

Failed tests/framework/CodeInterfaceTests/RAVEN/Basic

wangcj05 · 2021-05-10T19:58:21Z

This PR is good. Checklist is satisfied, and tests are green (OpenSUSE Leap 15 is down for now.).

joshua-cogliati-inl added 12 commits April 13, 2021 11:52

Starting creating a test for lstm regression.

291b8f1

Adding initial KerasLSTMRegression class (copy of KerasLSTMClassifier)

c24709f

Specifying statsmodel=0.12 causes unsolvability of dependencies on OSX

a4df355

for Prescient requirements.

Adding copy of KerasClassifier called KerasRegression

8b354b1

Start work on revising to do regression.

57f91db

LSTM should always return_sequences

2b5b9a8

Fixed shape of arrays.

03bbcb7

Fixing evaluate to work with keras.

f37c5c1

Attempt to sample with SingleRun

5b28467

Trying with CustomSampler.

f0a8eac

Workaround for CustomSampler bug.

6252683

Adding scaling of the target values since that is needed.

c2647a1

It makes the training easier for the RNN.

joshua-cogliati-inl added the Do Not Merge label Apr 19, 2021

joshua-cogliati-inl changed the title ~~Cogljj/lstm regression~~ LSTM Regression Apr 19, 2021

wangcj05 reviewed Apr 19, 2021

View reviewed changes

Making common base class for Keras Neural Network tools.

e4a31fa

joshua-cogliati-inl force-pushed the cogljj/lstm_regression branch from ff10494 to e4a31fa Compare April 28, 2021 20:24

joshua-cogliati-inl added 4 commits April 30, 2021 13:39

Merge remote-tracking branch 'origin/devel' into cogljj/lstm_regression

561ea3f

Adding a regression test for the lstm regression.

b5abc7e

Switching SingleRun on non-Code to an error.

0f2bc2f

The LSTM varies depending on computer a bit, so adding rel_err

dfad1fa

joshua-cogliati-inl force-pushed the cogljj/lstm_regression branch from 4cda313 to dfad1fa Compare May 3, 2021 14:13

joshua-cogliati-inl added 4 commits May 3, 2021 13:25

Simplifying some code in KerasClassifier.

ed004e9

Moving train and layer adding functions to KerasBase.

5728c3b

Moving more functions to KerasBase.

eb41829

Allowing 25% greater ARMA relative error.

e750817

Adding documentation on KerasLSTMRegression.

e33571c

joshua-cogliati-inl removed the Do Not Merge label May 4, 2021

wangcj05 requested changes May 4, 2021

View reviewed changes

joshua-cogliati-inl added 2 commits May 4, 2021 16:39

Fixing some of the review comments.

29cc3d1

Additional code review comments.

8e85690

joshua-cogliati-inl commented May 5, 2021

View reviewed changes

Fewer epochs and other minor changes.

b6b21a3

joshua-cogliati-inl force-pushed the cogljj/lstm_regression branch from 4ded24b to b6b21a3 Compare May 5, 2021 19:58

Adding comment that _preprocessInputs in duplicated.

9b7b418

wangcj05 approved these changes May 10, 2021

View reviewed changes

wangcj05 merged commit cf0c4f9 into devel May 10, 2021

wangcj05 deleted the cogljj/lstm_regression branch May 10, 2021 19:58

wangcj05 added the RAVENv2.1 All tasks and defects that will go in RAVEN v2.1 label May 12, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

LSTM Regression #1519

LSTM Regression #1519

joshua-cogliati-inl commented Apr 19, 2021 •

edited by wangcj05

Loading

moosebuild commented Apr 19, 2021

moosebuild commented Apr 19, 2021

wangcj05 Apr 19, 2021

joshua-cogliati-inl Apr 20, 2021

joshua-cogliati-inl Apr 30, 2021

joshua-cogliati-inl May 3, 2021

moosebuild commented Apr 28, 2021

moosebuild commented May 3, 2021

joshua-cogliati-inl commented May 4, 2021

wangcj05 left a comment

wangcj05 commented May 4, 2021

joshua-cogliati-inl May 5, 2021

wangcj05 May 5, 2021

joshua-cogliati-inl May 5, 2021

joshua-cogliati-inl May 5, 2021

wangcj05 May 5, 2021

joshua-cogliati-inl May 10, 2021

wangcj05 commented May 10, 2021

moosebuild commented May 10, 2021

moosebuild commented May 10, 2021

wangcj05 commented May 10, 2021

LSTM Regression #1519

LSTM Regression #1519

Conversation

joshua-cogliati-inl commented Apr 19, 2021 • edited by wangcj05 Loading

Pull Request Description

What issue does this change request address?

What are the significant changes in functionality due to this change request?

For Change Control Board: Change Request Review

moosebuild commented Apr 19, 2021

moosebuild commented Apr 19, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

moosebuild commented Apr 28, 2021

moosebuild commented May 3, 2021

joshua-cogliati-inl commented May 4, 2021

wangcj05 left a comment

Choose a reason for hiding this comment

wangcj05 commented May 4, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

wangcj05 commented May 10, 2021

moosebuild commented May 10, 2021

moosebuild commented May 10, 2021

wangcj05 commented May 10, 2021

joshua-cogliati-inl commented Apr 19, 2021 •

edited by wangcj05

Loading