Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

seisflows refactor and overhaul (v2.1.0) #139

Merged
merged 207 commits into from
Oct 18, 2022
Merged

seisflows refactor and overhaul (v2.1.0) #139

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
207 commits
Select commit Hold shift + click to select a range
bba3fd1
fixed failing seisflows and config tests. updated test parameter files
bch0w Jun 1, 2022
16adfd9
working on solver tests, making solver definitions more explicit by a…
bch0w Jun 2, 2022
1237df3
cleaning up some solver calls to remove redundant code
bch0w Jun 2, 2022
a0a9e16
fixing typos and docstrings in optimization module
bch0w Jun 14, 2022
7f351f4
cleaning up more docstrings
bch0w Jun 14, 2022
1054492
moved 'call_solver' into solver.base and OUT of the specfem tool as i…
bch0w Jun 15, 2022
9bcb66d
working on test framework which runs some simple functions to test ou…
bch0w Jun 15, 2022
d1288e2
workstation now writes function run outputs to log files rather than …
bch0w Jun 15, 2022
2010f3c
removing unused tool functions
bch0w Jun 15, 2022
cceea64
working on adding rosenbrock optimization test
bch0w Jun 16, 2022
adbd664
removed seisflows-super, moved all super modules back into main packa…
bch0w Jun 16, 2022
fc8c19b
cleaning up unused functions and docstrings
bch0w Jun 16, 2022
f1ae1c8
removed 'base' line search, bracketing line search is now the base. a…
bch0w Jun 17, 2022
b14fc3b
rearranging config and seisflows cli functions for better organizatio…
bch0w Jun 17, 2022
b1a4343
removing some agency from line search and giving it to optimize to ke…
bch0w Jun 17, 2022
36a6c39
fixing tests after refactor, specfem2d example also works
bch0w Jun 18, 2022
bac59bc
Merge pull request #122 from adjtomo/refactor
bch0w Jun 18, 2022
e841218
bug fix optimize was trying to load PATH.OPTIMIZE before it was insta…
bch0w Jun 20, 2022
9e9c645
bug fix syntax comparison in slurm system
bch0w Jun 21, 2022
1afa661
working on rosenbrock optimization test problem
bch0w Jun 21, 2022
9e21c5b
setting default solver values to reduce the number of required parame…
bch0w Jun 21, 2022
351566c
removed PAR.NT, PAR.DT and PAR.F0 from the list of required solver pa…
bch0w Jun 21, 2022
285b5e7
optimization rosenbrock test now working on test workflow
bch0w Jun 22, 2022
78f0c85
created a new core.py file which contains all the core SeisFlows clas…
bch0w Jun 24, 2022
156f820
added some additional comments to the Base class to explain how thing…
bch0w Jun 24, 2022
de08f53
getting rid of PAR and PATH indirection in config which were just var…
bch0w Jun 27, 2022
0294257
updatiedclass structure for optimization module, key points are that:…
bch0w Jun 27, 2022
c6e08c2
finished updating optimize classes to new architecture
bch0w Jun 28, 2022
f7133a2
updated preprocessing classes to new code style
bch0w Jun 28, 2022
4a2781e
renamed solver.base to solver.specfem to signify that this is really …
bch0w Jun 28, 2022
f9ffb1f
specefem2d and specfem3d now have the new code styling. Trying to con…
bch0w Jun 28, 2022
3f7c34f
finished condensing and rewiriting SOLVER and SYSTEM modules. Removed…
bch0w Jun 28, 2022
983e684
making Forward workflow class the base workflow class, building every…
bch0w Jun 28, 2022
88cedb1
added frontera work in progress system with new submit and run calls
bch0w Jun 29, 2022
ba7118f
updated workflow classes to new code structure. inversion workflow no…
bch0w Jun 29, 2022
a2694c6
rearranging main directory structure to be more easily navigable. mov…
bch0w Jun 30, 2022
5bfc552
moved logging establishment into the core.base class, rather than as …
bch0w Jun 30, 2022
6a1ac53
removed the explicit calls to parent functions that did not change an…
bch0w Jun 30, 2022
6f0576b
specfem2d example now working
bch0w Jun 30, 2022
5076b52
updated test parameter files and fixed all tests with new code structure
bch0w Jun 30, 2022
1df4c65
new cli function seisflows swap to change out module parameters on an…
bch0w Jun 30, 2022
d700407
removing system self call causing recursion error, removing config lo…
bch0w Jul 1, 2022
ac223a1
updating maui run scripts to get 2d example working on cluster
bch0w Jul 1, 2022
9087e86
merging fixes
bch0w Jul 1, 2022
bf36afc
fixing logging as it isnt working on cluster, trying to revert back t…
bch0w Jul 5, 2022
3b47faa
logger fixed for submitted processes on system as well as main logger.
bch0w Jul 5, 2022
317dc00
removing Minmax and Container classes which served very minimal purpo…
bch0w Jul 5, 2022
9a63fa5
fix small bug in solver trying to load model values into a non-instan…
bch0w Jul 5, 2022
7a56723
starting to separate solver model manipulation functions from the cla…
bch0w Jul 5, 2022
6fc60ca
bug fixing missing imports
bch0w Jul 5, 2022
91c4876
Merge pull request #124 from adjtomo/testing
bch0w Jul 6, 2022
918ef00
working to separate solver from model manipulation functinoalities
bch0w Jul 6, 2022
458b1db
created a new specfem Model class which is in charge of reading, chec…
bch0w Jul 7, 2022
dc4aac3
starting to replace the old model implementation with the new one thr…
bch0w Jul 7, 2022
7b97ac3
finished new Model class which has ability to IO and manipulate SPECF…
bch0w Jul 7, 2022
12fda4a
fixing tests, fully changed out new model implementation throughout p…
bch0w Jul 8, 2022
521b028
scrapping the 'postprocess' module as it was really not carrying its …
bch0w Jul 8, 2022
03189f6
testing out a new workflow schema involving functions and a script, N…
bch0w Jul 8, 2022
32c0ccc
gradient and LBFGS now have new implementation of parameter system. i…
bch0w Jul 8, 2022
906a5bc
finished NLCG new parameter system, will need to go back and make sur…
bch0w Jul 8, 2022
4f5a8d6
converted default postprocess module to new paramter system
bch0w Jul 8, 2022
6d903e2
preprocess default converrted to new paramter system but will require…
bch0w Jul 8, 2022
d2474af
solver up to specfem3d_globe converted to new parameter system
bch0w Jul 8, 2022
b17bc73
finished refactoring all system modules up to maui (what it inherited…
bch0w Jul 9, 2022
832a675
refactoring solver and preprocess to fit the new workflow style where…
bch0w Jul 12, 2022
6774984
refactoring solver.specfem to be more pythonic, moving a lot of the l…
bch0w Jul 13, 2022
de40cfc
starting solver test suite
bch0w Jul 13, 2022
2034150
renamed 'wrappers' to 'utils' and cleared out a number of unnused sma…
bch0w Jul 13, 2022
ccc7cee
completed solver tests and preprocess tests after refactoring both co…
bch0w Jul 14, 2022
64d8798
workin g on new forward workflow which is given in script form rather…
bch0w Jul 14, 2022
0800eeb
reworking workflows as classes but with a bookkeeping mechanic to mor…
bch0w Jul 16, 2022
2983584
added inversion workflow supered on top of migration workflow. Need t…
bch0w Jul 18, 2022
d1e94ff
removed CFGPATHS and postprocessing module, rolled postprocessing fun…
bch0w Jul 19, 2022
2ddd9dc
line search currently working, improved log messages for line search …
bch0w Jul 20, 2022
a3cc003
adding doc inheritance for future seisflows configure command, udpati…
bch0w Jul 20, 2022
9513931
implementing a new cluster run system which saves individual methods …
bch0w Jul 21, 2022
b7c17eb
removing agency from optimization module. optimization no longer keep…
bch0w Jul 21, 2022
0063408
adjusting docstrings to fit the new seisflows configure command which…
bch0w Jul 21, 2022
ed7c34b
fixed some check statements in solver
bch0w Jul 21, 2022
147e15b
fixed up seisflows CLI tools based on the new workflow/parameter system
bch0w Jul 23, 2022
487675a
inversion workflow now working with checkpointing of optimization mod…
bch0w Jul 25, 2022
55d054b
moved config.py to tools/ rather than main directory. Moved some of t…
bch0w Jul 25, 2022
eda9b84
working on optimization test problem
bch0w Jul 25, 2022
19637c4
Model class can now set a model attribute and derive internal attribu…
bch0w Jul 26, 2022
a47928d
created a relatively mature optimization test module which runs both …
bch0w Jul 27, 2022
7a4aecd
adding to-do tests and updated NLCG formatting for new Model approach…
bch0w Jul 27, 2022
6402084
rolled in 'core' utilities to the 'config' tool script. changed impor…
bch0w Jul 27, 2022
ad92c21
removing solver_io plugin, this has been completely replaced by the M…
bch0w Jul 27, 2022
43eb704
added a copy() function to Model class for specfem models, whenever n…
bch0w Jul 27, 2022
a4651b7
finished optimization test suite, working on system test suite. Fixin…
bch0w Jul 28, 2022
b95ade6
removing old test workflow, this is now replaced by unit tests
bch0w Jul 28, 2022
ca7c33a
removing out of date seisflows cmd line tool tests
bch0w Jul 28, 2022
eeb4b56
trying to incorporate pyatoa preprocessing. shifting workload from py…
bch0w Jul 28, 2022
3d4189c
working on preprocessing pyatoa tests and fixing up pyatoa class
bch0w Jul 30, 2022
a6d4632
pyatoa preprocessing splitting off sion misfit quantification as a se…
bch0w Aug 1, 2022
082496d
bug fix: during line search wrong alpha was being used to calculate t…
bch0w Aug 1, 2022
c557275
small optimize test update
bch0w Aug 1, 2022
43078f1
fixing some test assertions that were coming up False
bch0w Aug 1, 2022
0990b8f
pyaflowa preprocessing module now working with example data and probl…
bch0w Aug 2, 2022
021abac
fixed a small nuisance where user had to input T0 for Pyatoa themselv…
bch0w Aug 2, 2022
3f48706
small typo fix pyaflowa
bch0w Aug 2, 2022
ad4c7c2
new test data for testing Pyaflowa preprocessing workflow, matches Ta…
bch0w Aug 3, 2022
11b4506
removing test data and slimming down preprocess integration test beca…
bch0w Aug 3, 2022
4d62b1b
seisflows configure now works thanks to more strict docstring formatt…
bch0w Aug 3, 2022
991fcba
changed default preprocess inputs to match pyaflowa call structure
bch0w Aug 4, 2022
b7a3d90
unix nproc command was failing on mac, replaced with os.cpu_count() w…
bch0w Aug 4, 2022
cf80dcf
hiding specfem2d f0 variable as it should not be set in par file
bch0w Aug 11, 2022
b2ca7a3
fixed failing seisflows cli tests which was due to changed cli archit…
bch0w Aug 11, 2022
cbed51f
fixing preprocess tests, all existing tests pass npow
bch0w Aug 11, 2022
56a4745
fixing example problems and tests, will need to write more tests to c…
bch0w Aug 12, 2022
5c3e429
small bugfix optimization wasnt able to access checkpoint for editing
bch0w Aug 12, 2022
21d5ddb
updating docs to match latest version of seisflows, finished command …
bch0w Aug 15, 2022
ced2be3
bugfix inversion was updating all model parameter rather than those c…
bch0w Aug 15, 2022
e8ea3f7
updating doc notebooks to reflect change to seisflows (from seisflows…
bch0w Aug 16, 2022
4988dbd
finished updating docs notebooks and fixed weird ls coloring issue an…
bch0w Aug 16, 2022
94b23d1
updated parameter change in example to match density forced type bool
bch0w Aug 16, 2022
b353175
Merge branch 'master' into refactor_workflow_and_parameter_system
bch0w Aug 16, 2022
0fff014
Merge pull request #125 from adjtomo/refactor_workflow_and_parameter_…
bch0w Aug 16, 2022
6276494
created notebook for extending seisflows
bch0w Aug 17, 2022
23ae5ac
updated slurm system design such that run and submit headers are sepa…
bch0w Aug 17, 2022
aa48609
added a test_system workflow which simply attempts to use the 'run' f…
bch0w Aug 17, 2022
013bfe4
bugfix configure was writing docstrings for Null modules which is unn…
bch0w Aug 17, 2022
9e9a170
tempfix logger printing twice as stdout and log statements were both …
bch0w Aug 18, 2022
238a704
bugfix slurm system not actually replacing single proc calls
bch0w Aug 18, 2022
1a463c2
improving test_flow workflow for testing modules in a live environment
bch0w Aug 18, 2022
c0f23a0
bugfix slurm system not exiting when jobs failed due to incorrect che…
bch0w Aug 18, 2022
975aecd
fixed up 'seisflows swap' command to replace a module in an existing …
bch0w Aug 18, 2022
db1d4d9
updating template parameter file docstring
bch0w Aug 18, 2022
3eeb564
updated some docstrings to make parameter file more readable
bch0w Aug 18, 2022
46ae429
Merge pull request #126 from adjtomo/feature_hpc_support
bch0w Aug 18, 2022
e6a3e81
slurm query job status now has an internal recursive check function i…
bch0w Aug 19, 2022
5bfe916
bugfix seisflows 'swap' sets NoneType as null, before they were input…
bch0w Aug 19, 2022
d6cf1b3
slurm system function to evaluate job id from the subprocess.run stdo…
bch0w Aug 19, 2022
622c7b4
bugfix seisflows clean wasn't actually deleting files
bch0w Aug 19, 2022
058ff4c
frontera added more checks for partition and forced partition choices…
bch0w Aug 20, 2022
27b1f03
testflow slightly working on frontera with ssh option, need to find w…
bch0w Aug 20, 2022
07d1981
Frontera module now defines its own run() function which side-steps t…
bch0w Aug 20, 2022
36317cf
Merge pull request #127 from adjtomo/feature_frontera_support
bch0w Aug 22, 2022
ccf0bc0
re-instantes logging to stream (stdout) but turns off stream logging …
bch0w Aug 23, 2022
f6845ba
updating 2D examples
bch0w Aug 23, 2022
f510a1a
bugfix seisflows examples print statement was trying to evaluate the …
bch0w Aug 23, 2022
48817dd
editing install requirements and pinning deps that werent previously …
bch0w Aug 23, 2022
bd262ae
remove template parameter file (#131)
bch0w Aug 23, 2022
9c763c5
Update documentation (#132)
bch0w Aug 24, 2022
32a5d47
unpinning lowest scipy version due to Conda UnsatisfiedError package …
bch0w Aug 24, 2022
a418d41
extended the seisflows containers docs page with working examples for…
bch0w Aug 25, 2022
0f9e901
added a completion log message for finished example problems
bch0w Aug 25, 2022
f80bef5
splitting 2D example docs into the automated runs which will now incl…
bch0w Aug 26, 2022
a3889d5
Feature plot2d (#134)
bch0w Aug 28, 2022
a5477e8
Merge branch 'devel' into workshop_container
bch0w Aug 28, 2022
5942760
bugfix seisflows plot2d picking up non model directories for plotting
bch0w Aug 29, 2022
1c6d4f3
added plot2d to command line tool docs page
bch0w Aug 29, 2022
1b76e32
updated specfem2d example with what user should expect when finishing…
bch0w Aug 29, 2022
7524379
bugfix: specfem2d smoothing was not providing smoothing lengths (h an…
bch0w Aug 29, 2022
2913cba
updated notebook convert function to do some bookkeeping after creati…
bch0w Aug 29, 2022
39130db
plot2d now plots zero midpoint for gradient/kernel figures
bch0w Aug 29, 2022
7de6d46
bugfix: step_count_max was exceeded by 1 step count, changed a '<=' l…
bch0w Aug 29, 2022
e7e1745
refactored SEISFLOWS EXAMPLES
bch0w Aug 30, 2022
b3e2573
bugfix pyaflowa wasnt writing adjoint sources for misfit ==0
bch0w Aug 30, 2022
e76cc39
bugfix solver.specfem setup not copying correct model files
bch0w Aug 30, 2022
c893c99
updating specfem2d example problems to match the new examples 1 and 2
bch0w Aug 30, 2022
06c947c
bugfix forward workflow failing check with no 'data' even though prep…
bch0w Aug 30, 2022
7d483b5
updating docstrings
bch0w Aug 30, 2022
d744eeb
bugfix forward workflow w/o preprocessing module kept hitting errors …
bch0w Aug 30, 2022
70635db
feature: added a very simple stream plotter to look at output waveforms
bch0w Aug 30, 2022
259f567
forward workflow example working, fixed up the stream plotter, fixing…
bch0w Aug 30, 2022
a8c87fa
reworking example problems to be more dynamic and modular
bch0w Sep 2, 2022
6a5c0c8
bugfix specfem model not able to get min or max val from multi proces…
bch0w Sep 3, 2022
e4e6236
feature allow tape2007 example problems to set event id (with '--even…
bch0w Sep 3, 2022
01a5914
small docstring change specfem model tool
bch0w Sep 7, 2022
1d4b869
bugfix pyaflowa preprocessing failing when not set as it was trying …
bch0w Sep 9, 2022
99702a4
updated specfem2d example notebook and docs page up to Example 3 with…
bch0w Sep 9, 2022
b1ba4a6
slightly modified example 2a docs page to show a better kernel match …
bch0w Sep 9, 2022
780ed7b
updated specfem2d examples after final proofread
bch0w Sep 9, 2022
44e617f
added plotst to command line tool docs page
bch0w Sep 9, 2022
dac4c45
fixing preprocess test failing due to wrong paths
bch0w Sep 9, 2022
ae89405
Merge pull request #135 from adjtomo/workshop_container
bch0w Sep 9, 2022
67edb3f
Update README.md
bch0w Sep 9, 2022
66fd5c4
feature allow user to specify mpi executable during example problems
bch0w Sep 9, 2022
7863061
Merge branch 'devel' of github.com:adjtomo/seisflows into devel
bch0w Sep 9, 2022
4ff549c
docs updated index so that the top heading is 'seisflows documentatio…
bch0w Sep 10, 2022
8d29f65
examples added MPI capabilities natively to base example, allow user …
bch0w Sep 10, 2022
e79b3d2
updated other examples and seisflows CLI tool to accept mpiexec and n…
bch0w Sep 10, 2022
a9b9df0
added a small section in example docs about using MPI
bch0w Sep 10, 2022
7975e61
updated permissions on submit workflow script
bch0w Sep 10, 2022
315271c
prototype chinook singularity system interface kludge to test inside …
bch0w Sep 10, 2022
bda5657
bugfix run tasks spawned by Cluster.run() now log to separate log fil…
bch0w Sep 11, 2022
a400da3
feature: seisflows plot2d adds model and parameter name to title for …
bch0w Sep 13, 2022
227752b
examples: removed symlinking of specfem2d to current working director…
bch0w Sep 13, 2022
6da4f97
Merge branch 'devel' of github.com:adjtomo/seisflows into devel
bch0w Sep 13, 2022
cd5bbea
bugfix: typo in specfem3d executable check
bch0w Sep 13, 2022
567b4de
fixed some assertion error log messages for solver specfem
bch0w Sep 14, 2022
e83cf4e
bugfix specfem3d pointing too unavailable model database location
bch0w Sep 14, 2022
6722bb8
feature specfem3d now wraps 'xcombine_vol_data_vtk' for making VTK files
bch0w Sep 14, 2022
18d57ee
bugfix specfem3d wasnt deleting prune scratch files
bch0w Sep 14, 2022
b3395fe
missing parenthesis typo
bch0w Sep 14, 2022
32d3775
removed sf example print statement telling user that no existing spec…
bch0w Sep 15, 2022
ba96465
small typo bug fixes in specfem3d implementation
bch0w Sep 16, 2022
d9af4e3
add plot_2d_image and use it for specfem2d.Model.plot2d
rdno Oct 7, 2022
fd66e90
Merge pull request #137 from rdno/plot_2d_image
bch0w Oct 14, 2022
f97c008
updated chinook and slurm system and testflow now works on new Chinook
bch0w Oct 15, 2022
f5d0dcd
addressing #138 broke out 'Model' class from the specfem tools into i…
bch0w Oct 15, 2022
ff2f97e
slurm system adding wait time afer successful run() to allow filesyst…
bch0w Oct 15, 2022
daacea4
added logger warning message to model when no path is given
bch0w Oct 15, 2022
e557285
Added more wait time in slurm run calls as python was running past th…
bch0w Oct 15, 2022
60eb16b
shortened slurm wait time by a few seconds
bch0w Oct 15, 2022
b1de09f
bumping version number for version release
bch0w Oct 18, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ SeisFlows
`Documentation` can be found on Read the Docs: https://seisflows.readthedocs.io (in development)


[![SCOPED](https://img.shields.io/endpoint?url=https://runkit.io/wangyinz/scoped/branches/master/Pyatoa)](https://github.com/SeisSCOPED/container/pkgs/container/pyatoa)
[![SCOPED](https://img.shields.io/endpoint?url=https://runkit.io/wangyinz/scoped/branches/master/adjTomo)](https://github.com/SeisSCOPED/container/pkgs/container/adjtomo)
[![Documentation Status](https://readthedocs.org/projects/seisflows/badge/?version=latest)](https://seisflows.readthedocs.io/en/latest/?badge=latest)
---

Expand Down
Loading