ensemble model parallelism #1342

alfoa · 2020-10-13T16:34:55Z

Pull Request Description

What issue does this change request address? (Use "#" before the issue to link it, i.e., #42.)

Closes #1341

What are the significant changes in functionality due to this change request?

Changed the parallelism strategy in case of ensemble model.

For Change Control Board: Change Request Review

The following review must be completed by an authorized member of the Change Control Board.

1. Review all computer code.
2. If any changes occur to the input syntax, there must be an accompanying change to the user manual and xsd schema. If the input syntax change deprecates existing input files, a conversion script needs to be added (see Conversion Scripts). (No changes to input syntax)
3. Make sure the Python code and commenting standards are respected (camelBack, etc.) - See on the wiki for details. (Yes)
4. Automated Tests should pass, including run_tests, pylint, manual building and xsd tests. If there are changes to Simulation.py or JobHandler.py the qsub tests must pass. (yes)
5. If significant functionality is added, there must be tests added to check this. Tests should cover all possible options. Multiple short tests are preferred over one large test. If new development on the internal JobHandler parallel system is performed, a cluster test must be added setting, in XML block, the node <internalParallel> to True. (no significant functionality added)
6. If the change modifies or adds a requirement or a requirement based test case, the Change Control Board's Chair or designee also needs to approve the change. The requirements and the requirements test shall be in sync. (needs second)
7. The merge request must reference an issue. If the issue is closed, the issue close checklist shall be done. (done)
8. If an analytic test is changed/added is the the analytic documentation updated/added? (no change)
9. If any test used as a basis for documentation examples (currently found in raven/tests/framework/user_guide and raven/docs/workshop) have been changed, the associated documentation must be reviewed and assured the text matches the example. (no change)

…e involved

alfoa · 2020-10-13T16:36:14Z

@JiaZhou-PU

moosebuild · 2020-10-14T19:59:53Z

Job Mingw Test on 32960e5 : invalidated by @joshua-cogliati-inl

test civet

moosebuild · 2020-10-14T20:04:10Z

Job Mingw Test on 32960e5 : invalidated by @joshua-cogliati-inl

test civet

alfoa · 2020-10-20T16:55:43Z

@joshua-cogliati-inl ready to review

joshua-cogliati-inl

There are some things that need to be fixed. I would prefer if this was split into two pull requests, but that is not necessary. I also had some questions.

There are definitely improvements in this pull request :)

framework/JobHandler.py

framework/Models/EnsembleModel.py

framework/SupervisedLearning/ARMA.py

framework/utils/randomUtils.py

tests/framework/CodeInterfaceTests/Instant/IAEA2D/sample/1/alias.xml

alfoa · 2020-10-22T16:51:38Z

@joshua-cogliati-inl Josh I addressed your comments

moosebuild · 2020-10-22T20:41:35Z

Job Test Ubuntu 18-2 Python 3 on da1cd4f : invalidated by @joshua-cogliati-inl

failed tests/framework/Optimizers/SimulatedAnnealing/ExponentialEggHolder

PaulTalbot-INL · 2020-10-22T22:28:27Z

Any hope for a end-of-week miracle merge on this?

alfoa · 2020-10-22T22:43:04Z

@PaulTalbot-INL looks like Josh is on it...so you called the miracle ehhehe

joshua-cogliati-inl · 2020-10-22T22:45:31Z

I approve, but this changes a requirements test so needs a second approval from the CCB chair or designee.

joshua-cogliati-inl · 2020-10-22T22:49:01Z

@PaulTalbot-INL @alfoa This changes a requirements test, so it needs a second approver.

alfoa · 2020-10-22T23:06:37Z

As chair of COB I approve the requirement test modifications.

alfoa · 2020-10-22T23:07:02Z

@PaulTalbot-INL can you also approve the PR?

PaulTalbot-INL

I'm okay with all these changes, as well.

PaulTalbot-INL · 2020-10-22T23:49:03Z

framework/Models/EnsembleModel.py

-    kwargsKeys.pop(kwargsKeys.index("jobHandler"))
-    kwargsToKeep = { keepKey: kwargs[keepKey] for keepKey in kwargsKeys}
-    jobHandler = kwargs['jobHandler']
+    kwargsToKeep = { keepKey: kwargs[keepKey] for keepKey in list(kwargs.keys())}


Is this just kwargsToKeep == copy.deepcopy(kwargs)? No change needed, I see how this could be useful in the future if we have keys we don't want to keep.

ops...yes.. i will address this in next PR

alfoa added 7 commits October 2, 2020 09:40

change in paradigm

8c59500

Merge branch 'devel' into alfoa/few_ray_mods

042888c

ensemble

8a32208

a

38e4761

a

a5126d5

Merge branch 'devel' into alfoa/few_ray_mods

7f0f86a

changed paradigm for parallelization of ensemble model if no Codes ar…

eefdd36

…e involved

alfoa changed the title ~~Alfoa/few ray mods~~ ensemble model parallelism Oct 13, 2020

Update EnsembleModel.py

d9a909e

alfoa requested a review from joshua-cogliati-inl October 13, 2020 16:36

fix version of RAY

32960e5

alfoa added 16 commits October 15, 2020 12:27

fixed bug in library_handler

09c5de7

new statsmodel and change of the api

384c855

removed some tests that require executable that we do not compile

5cbcbc5

fix

976dc50

removed additional test with required exe

e1e4ad9

restored files from Conjian's test of the interface

3f032ec

fixed tests

9f64a69

moved env variable

9f1e6dc

a

13175b4

added Choice

7e88b84

added unit test

b9f9011

deepcopy in the right place

7bf653a

deepcopy in the right place

6eac6e1

local fis

5274717

updated versions

c9e8864

removed local fix for VPN

ed1acba

Merge branch 'devel' into alfoa/few_ray_mods

05eb82d

joshua-cogliati-inl requested changes Oct 21, 2020

View reviewed changes

addresssed Josh's comments

da1cd4f

joshua-cogliati-inl approved these changes Oct 22, 2020

View reviewed changes

PaulTalbot-INL approved these changes Oct 22, 2020

View reviewed changes

PaulTalbot-INL merged commit 580af97 into devel Oct 22, 2020

alfoa deleted the alfoa/few_ray_mods branch October 22, 2020 23:51

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ensemble model parallelism #1342

ensemble model parallelism #1342

alfoa commented Oct 13, 2020 •

edited by PaulTalbot-INL

Loading

alfoa commented Oct 13, 2020

moosebuild commented Oct 14, 2020

moosebuild commented Oct 14, 2020

alfoa commented Oct 20, 2020

joshua-cogliati-inl left a comment

alfoa commented Oct 22, 2020

moosebuild commented Oct 22, 2020

PaulTalbot-INL commented Oct 22, 2020

alfoa commented Oct 22, 2020

joshua-cogliati-inl commented Oct 22, 2020

joshua-cogliati-inl commented Oct 22, 2020

alfoa commented Oct 22, 2020

alfoa commented Oct 22, 2020

PaulTalbot-INL left a comment

PaulTalbot-INL Oct 22, 2020

alfoa Oct 22, 2020

ensemble model parallelism #1342

ensemble model parallelism #1342

Conversation

alfoa commented Oct 13, 2020 • edited by PaulTalbot-INL Loading

Pull Request Description

What issue does this change request address? (Use "#" before the issue to link it, i.e., #42.)

What are the significant changes in functionality due to this change request?

For Change Control Board: Change Request Review

alfoa commented Oct 13, 2020

moosebuild commented Oct 14, 2020

moosebuild commented Oct 14, 2020

alfoa commented Oct 20, 2020

joshua-cogliati-inl left a comment

Choose a reason for hiding this comment

alfoa commented Oct 22, 2020

moosebuild commented Oct 22, 2020

PaulTalbot-INL commented Oct 22, 2020

alfoa commented Oct 22, 2020

joshua-cogliati-inl commented Oct 22, 2020

joshua-cogliati-inl commented Oct 22, 2020

alfoa commented Oct 22, 2020

alfoa commented Oct 22, 2020

PaulTalbot-INL left a comment

Choose a reason for hiding this comment

PaulTalbot-INL Oct 22, 2020

Choose a reason for hiding this comment

alfoa Oct 22, 2020

Choose a reason for hiding this comment

alfoa commented Oct 13, 2020 •

edited by PaulTalbot-INL

Loading