Numba montecarlo #929

wkerzendorf · 2019-05-04T16:26:29Z

Things to check if they are done

Things to do

Line estimators are slowing the process down - maybe a useful optimization strategy

Final Testing

There are 4 code versions. There is upstream/master there is numba_montecarlo both of which have different seeding strategies and will never be numerically equal (those we will summarize as main). There are two versions that have additional code that will allow logging - those we will call c_compare and numba_compare (and will call the compare branch).

virtual - only compare - sica
virtual - only compare - ddc10
integrated - main only - sica
integrated - main only - ddc10
estimators - compare only -sica
estimators - compare only -ddc10
iterations - (compare/main) -sica
iterations - (compare/main) -ddc10
Need to fix negative boundary distance issue for vpackets: specific case occurs when a packet reaches a boundary with a close line so distance_trace_line = 0 and distance_boundary can be negative (shell ID does not advance)
The models that shall be run is low velocity Si/Ca and DDC10.

The last step is to get numba to produce a new refdata and look at this in the notebook.

If all these models run then we agree that numba montecarlo can be merged.

Merging Strategy

talk to TARDIS core
if all tests pass - update the version number to 4.0 with the merged numba montecarlo
merge

Post-merge

Start writing unit tests for the numba version
Improve speed.

wkerzendorf · 2019-06-04T17:14:11Z

There is a sensible discussion about setting r when doing a move_packet_across_shell_boundary - do we move like we normally do - or just set r to the shell boundary??

stuartarchibald · 2020-03-16T20:06:06Z

tardis/montecarlo/montecarlo_numba/__init__.py

@@ -0,0 +1,7 @@
+from llvmlite import binding
+binding.set_option("tmp", "-non-global-value-max-name-size=2048")


Wondering what this is for? IIRC this patch against LLVM "fixed", well, prevented, it from being an issue https://github.com/numba/llvmlite/blob/master/conda-recipes/0001-Revert-Limit-size-of-non-GlobalValue-name.patch .

good to know. Thanks - haven't worked on it in a while.

Co-authored-by: Christian Vogl <cvogl@mpa-garching.mpg.de> Co-authored-by: Alice Harpole <aliceharpole@gmail.com> Co-authored-by: Yssavo Camacho-Neves <yic6@physics.rutgers.edu>

Co-authored-by: Christian Vogl <cvogl@mpa-garching.mpg.de>

Values are stored in the vpacket collection for each vpacket. Vpacket collections are dumped into a huge list right now.

Need to set the configuration value to True in montecarlo_configuration.py Needs to be an install option again

Can now be activated as a configuration settings rather than compile-time. Output should now be a proper array of arrays rather than a list of arrays.

tardis/io/schemas/montecarlo.yml

tardis/montecarlo/__init__.py

wkerzendorf · 2020-11-13T21:21:21Z

tardis/montecarlo/montecarlo_numba/__init__.py

@@ -0,0 +1,7 @@
+from llvmlite import binding
+binding.set_option("tmp", "-non-global-value-max-name-size=2048")


Suggested change

binding.set_option("tmp", "-non-global-value-max-name-size=2048")

tardis/montecarlo/montecarlo_numba/base.py

tardis/montecarlo/montecarlo_numba/tests/test_single_packet_loop.py

tardis/montecarlo/montecarlo_numba/tests/test_vpacket.py

tardis/montecarlo/montecarlo_numba/vpacket.py

tardis/scripts/debug/run_numba_single.run.xml

Docstrings from Wolfgang Co-authored-by: Wolfgang Kerzendorf <wkerzendorf@gmail.com>

Co-authored-by: Wolfgang Kerzendorf <wkerzendorf@gmail.com>

codecov · 2020-11-14T23:08:57Z

Codecov Report

Merging #929 (d14025e) into master (071a48e) will decrease coverage by 10.32%.
The diff coverage is n/a.

@@             Coverage Diff             @@
##           master     #929       +/-   ##
===========================================
- Coverage   81.57%   71.24%   -10.33%     
===========================================
  Files          47       66       +19     
  Lines        3804     5077     +1273     
===========================================
+ Hits         3103     3617      +514     
- Misses        701     1460      +759

Impacted Files	Coverage Δ
tardis/tardis/montecarlo/formal_integral.py	`15.78% <0.00%> (-78.81%)`	⬇️
tardis/tardis/widgets/line_info.py	`75.17% <0.00%> (-20.69%)`	⬇️
tardis/tardis/widgets/tests/test_line_info.py	`80.85% <0.00%> (-19.15%)`	⬇️
tardis/tardis/montecarlo/base.py	`84.61% <0.00%> (-2.89%)`	⬇️
tardis/tardis/widgets/util.py	`82.92% <0.00%> (-2.44%)`	⬇️
tardis/tardis/io/config_reader.py	`86.02% <0.00%> (ø)`
tardis/tardis/montecarlo/struct.py	`100.00% <0.00%> (ø)`
tardis/tardis/widgets/shell_info.py	`96.00% <0.00%> (ø)`
tardis/tardis/io/config_validator.py	`89.47% <0.00%> (ø)`
tardis/tardis/montecarlo/spectrum.py	`70.00% <0.00%> (ø)`
... and 24 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 071a48e...d14025e. Read the comment docs.

Mostly to force a new test run

This is a year worth of work from many people and hopefully will make TARDIS easier to use and develop on. Co-authored-by: Andrew Fullard <andrewgfullard@gmail.com> Co-authored-by: Jack O'Brien <jobrien585@gmail.com> Co-authored-by: Christian Vogl <cvogl@mpa-garching.mpg.de> Co-authored-by: Andreas Flörs <afloers@mpa-garching.mpg.de> Co-authored-by: Luke Shingles <luke.shingles@gmail.com> Co-authored-by: Alice Harpole <harpolea@gmail.com> Co-authored-by: Marc Williamson <marxwillia@gmail.com> Co-authored-by: Yssa Camacho <yssavo.camacho@gmail.com> Co-authored-by: Arjun Savel <asavel@berkeley.edu> Co-authored-by: Stuart Archibald <stuart.archibald@googlemail.com> Co-authored-by: Siu Kwan Lam <michael.lam.sk@gmail.com>

This is a year worth of work from many people and hopefully will make TARDIS easier to use and develop on. Co-authored-by: Andrew Fullard <andrewgfullard@gmail.com> Co-authored-by: Jack O'Brien <jobrien585@gmail.com> Co-authored-by: Christian Vogl <cvogl@mpa-garching.mpg.de> Co-authored-by: Andreas Flörs <afloers@mpa-garching.mpg.de> Co-authored-by: Luke Shingles <luke.shingles@gmail.com> Co-authored-by: Alice Harpole <aliceharpole@gmail.com> Co-authored-by: Marc Williamson <marxwillia@gmail.com> Co-authored-by: Yssa Camacho <yssavo.camacho@gmail.com> Co-authored-by: Arjun Savel <asavel@berkeley.edu> Co-authored-by: Stuart Archibald <stuart.archibald@googlemail.com> Co-authored-by: Siu Kwan Lam <michael.lam.sk@gmail.com>

Numba montecarlo

This is a year worth of work from many people and hopefully will make TARDIS easier to use and develop on. Co-authored-by: Andrew Fullard <andrewgfullard@gmail.com> Co-authored-by: Jack O'Brien <jobrien585@gmail.com> Co-authored-by: Christian Vogl <cvogl@mpa-garching.mpg.de> Co-authored-by: Andreas Flörs <afloers@mpa-garching.mpg.de> Co-authored-by: Luke Shingles <luke.shingles@gmail.com> Co-authored-by: Alice Harpole <aliceharpole@gmail.com> Co-authored-by: Marc Williamson <marxwillia@gmail.com> Co-authored-by: Yssa Camacho <yssavo.camacho@gmail.com> Co-authored-by: Arjun Savel <asavel@berkeley.edu> Co-authored-by: Stuart Archibald <stuart.archibald@googlemail.com> Co-authored-by: Siu Kwan Lam <michael.lam.sk@gmail.com>

wkerzendorf force-pushed the numba_montecarlo branch 4 times, most recently from 4854f6c to f3494c2 Compare June 12, 2019 16:50

wkerzendorf force-pushed the numba_montecarlo branch from 4f6f86c to a110200 Compare June 14, 2019 18:59

wkerzendorf force-pushed the numba_montecarlo branch from a110200 to ad246be Compare June 27, 2019 17:00

wkerzendorf force-pushed the numba_montecarlo branch from bb7dd8f to 581365a Compare December 1, 2019 21:28

wkerzendorf force-pushed the numba_montecarlo branch 3 times, most recently from 9923c54 to ace4eda Compare January 25, 2020 23:37

wkerzendorf mentioned this pull request Mar 16, 2020

Suggestion for Numba/TARDIS-SN GSoC project numba/numba#5394

Closed

stuartarchibald reviewed Mar 16, 2020

View reviewed changes

wkerzendorf and others added 17 commits March 17, 2020 23:07

commenting out the montecarlo part

883c605

Add new numba stuff

059cbff

Co-authored-by: Christian Vogl <cvogl@mpa-garching.mpg.de> Co-authored-by: Alice Harpole <aliceharpole@gmail.com> Co-authored-by: Yssavo Camacho-Neves <yic6@physics.rutgers.edu>

add more numba conversions

532b414

Co-authored-by: Christian Vogl <cvogl@mpa-garching.mpg.de> Co-authored-by: Alice Harpole <aliceharpole@gmail.com> Co-authored-by: Yssavo Camacho-Neves <yic6@physics.rutgers.edu>

getting closer - TARDIS now fails

7df78ae

Co-authored-by: Christian Vogl <cvogl@mpa-garching.mpg.de> Co-authored-by: Alice Harpole <aliceharpole@gmail.com> Co-authored-by: Yssavo Camacho-Neves <yic6@physics.rutgers.edu>

final numba test - seems to work.

90bfb22

Co-authored-by: Christian Vogl <cvogl@mpa-garching.mpg.de> Co-authored-by: Alice Harpole <aliceharpole@gmail.com> Co-authored-by: Yssavo Camacho-Neves <yic6@physics.rutgers.edu>

all sorts of nonsense

d22a1c8

Co-authored-by: Christian Vogl <cvogl@mpa-garching.mpg.de> Co-authored-by: Alice Harpole <aliceharpole@gmail.com> Co-authored-by: Yssavo Camacho-Neves <yic6@physics.rutgers.edu>

commit stuff

394167f

Co-authored-by: Christian Vogl <cvogl@mpa-garching.mpg.de> Co-authored-by: Alice Harpole <aliceharpole@gmail.com> Co-authored-by: Yssavo Camacho-Neves <yic6@physics.rutgers.edu>

several edits later

f6b0be3

Co-authored-by: Christian Vogl <cvogl@mpa-garching.mpg.de> Co-authored-by: Alice Harpole <aliceharpole@gmail.com> Co-authored-by: Yssavo Camacho-Neves <yic6@physics.rutgers.edu>

still working on fixing numba_test

0449387

Co-authored-by: Christian Vogl <cvogl@mpa-garching.mpg.de>

add numba stuff

c53425f

branch point to diverge from c

b26410b

working numba implementation

38aae41

numba rewrite

7ce0bf0

Co-authored-by: Christian Vogl <cvogl@mpa-garching.mpg.de>

finished simple implementation

c10a239

Co-authored-by: Christian Vogl <cvogl@mpa-garching.mpg.de>

cleaned up version

9702834

Co-authored-by: Christian Vogl <cvogl@mpa-garching.mpg.de>

rewrite problem

bfd571d

further simplified r_packet

a16f23d

Fixed C compare, added legacy mode flag

1123fa1

andrewfullard marked this pull request as ready for review November 3, 2020 20:28

andrewfullard added 9 commits November 4, 2020 11:10

Cleaned out old fixtures

f1953b4

Vpacket logging partially restored

6513d18

Values are stored in the vpacket collection for each vpacket. Vpacket collections are dumped into a huge list right now.

Logging into HDF, probably

b69986d

Need to set the configuration value to True in montecarlo_configuration.py Needs to be an install option again

Improved virtual packet logging

dba60f3

Can now be activated as a configuration settings rather than compile-time. Output should now be a proper array of arrays rather than a list of arrays.

Only add vpackets that have data to the log

e68ce56

Simplified output of vpacket logging

b72ed62

Fix for 1D arrays

4458145

Fix Vpacket collection spec

884c55a

Static seed for random test

900cd81

wkerzendorf commented Nov 13, 2020

View reviewed changes

andrewfullard and others added 8 commits November 13, 2020 17:37

Apply Wolfgang's suggestions from code review

dee956b

Docstrings from Wolfgang Co-authored-by: Wolfgang Kerzendorf <wkerzendorf@gmail.com>

Update tardis/io/schemas/montecarlo.yml

ab039d2

Co-authored-by: Wolfgang Kerzendorf <wkerzendorf@gmail.com>

Update tardis/montecarlo/__init__.py

8ba9ed8

Co-authored-by: Wolfgang Kerzendorf <wkerzendorf@gmail.com>

Merge branch 'master' into numba_montecarlo

44a14ed

Fix syntax errors

ecc5779

black reformatting for numba montecarlo

59a0cf7

Fix vpacket collection set properties test

cb9791c

xfail tests with reasons

76dead8

andrewfullard added 2 commits November 16, 2020 15:45

Test fixes

1285575

Update debug_numba.rst

d14025e

Mostly to force a new test run

wkerzendorf merged commit 229b676 into master Nov 17, 2020

atharva-2001 pushed a commit to atharva-2001/tardis that referenced this pull request Oct 1, 2021

Merge pull request tardis-sn#929 from tardis-sn/numba_montecarlo

2f295f1

Numba montecarlo

andrewfullard deleted the numba_montecarlo branch July 20, 2022 15:58

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Numba montecarlo #929

Numba montecarlo #929

wkerzendorf commented May 4, 2019 •

edited

Loading

wkerzendorf commented Jun 4, 2019

stuartarchibald Mar 16, 2020

wkerzendorf Mar 16, 2020

wkerzendorf Nov 13, 2020

codecov bot commented Nov 14, 2020 •

edited

Loading

		@@ -0,0 +1,7 @@
		from llvmlite import binding
		binding.set_option("tmp", "-non-global-value-max-name-size=2048")

Numba montecarlo #929

Numba montecarlo #929

Conversation

wkerzendorf commented May 4, 2019 • edited Loading

Things to check if they are done

Final Testing

Merging Strategy

Post-merge

wkerzendorf commented Jun 4, 2019

stuartarchibald Mar 16, 2020

Choose a reason for hiding this comment

wkerzendorf Mar 16, 2020

Choose a reason for hiding this comment

wkerzendorf Nov 13, 2020

Choose a reason for hiding this comment

codecov bot commented Nov 14, 2020 • edited Loading

Codecov Report

wkerzendorf commented May 4, 2019 •

edited

Loading

codecov bot commented Nov 14, 2020 •

edited

Loading