SeisFlows Code Development Plan

[bch0w]

Last Update 8/6/24

This Code Development Plan (CDP) supersedes the CDP on the documentation and lays out the roadmap of features that are needed, requested or desired by the users and developers, as well as known bugs that should be fixed.

New developers who are interested in looking for tasks can use this list to determine what the needs of the community are. Development plans are organized by sections and modules for easier accessibility.

If you are interested in taking on a task and require more information to tackle it, feel free to either comment here with the task and a request for more details, or open an issue (https://github.com/adjtomo/seisflows/issues) with the task. Relevant issues will be linked back to this document for tracking purposes.

Note that any changes implemented in this list will likely go to the devel branch and be released officially in periodic version releases.

---

Priority of tasks are denoted by:

• ❗Critical: whatever it is, it needs to be addressed as soon as possible
• 🔥 Important: features or issues that should be addressed sooner rather than later
• 💨 Basic: this should be probably be addressed before the next version release
• 🔵 Blue Sky: would be nice to have in the future

Difficulty for tasks are denoted by:

• 🐔 Advanced: Requires extensive knowledge of the codebase (this is gonna be a tough one)
• 🐤 Intermediate: Requires basic understanding of codebase (if you're confident with Python you can probably handle this)
• 🐣 Beginner: Requires minimal understanding of codebase (easy first issues, you could knock this out over a coffee or three)

e.g., ❗🐣 means this is easy but needs to be addressed ASAP!

---

Unsorted:

Things on our radar

• Minimum model update threshold to stop line search (i.e., Wolfe Conditions)
• Ensure CIG 'Best Practice' standards

Known Bugs

Issues that have been pointed out by Users/Developers that should be fixed prior to implementing anything new

• 🔥 🐤 seisflows clean does not remove the directory included in path_data
• 🔥 🐔 seisflows state file does not populate a failed state (-1) even though a function clearly failed
• 🔥 🐔 SU and default preprocessing module are broken/unfinished (Fix Seismic Unix (SU) format support and bugfix default preprocessing seisflows#174)
• 🔥 🐔 Certain System sub modules have no ntask_max mechanisms (system parameter ntask_max is not honored for certain subclasses seisflows#200)
• 🔥 🐔 Some systems do not have consistent cores-per-node across a given partition, which may affect the currently implemented partition system
• 🔥 🐤 Default preprocessing scales waveforms by largest synthetic not largest waveform, leading to some mis-scaled plots
• 💨 🐤 Preprocessing parameter unit_output does not show up in parameters.yaml file during configure, but it should
• 🔥 🐔 Fujitsu System class does not currently honor parameter ntask_max, though it is included in the parameter set

Planned Features

Big planned feature additions that may require modifying some of SeisFlows' internal structure

• 💨 🐔 Dynamic mini batching line searches
• 💨 🐔 ADIOS file format compatibility
• 💨 🐔 SPECFEM ASDF file format compability
• 💨 🐔 Improved weighting functionality for misfit function (geographical weighting, kernel weighting, period band weighting)
• 💨 🐔 Provide more User control over misfit quantification procedure (e.g., manually selecting time windows, manually removing stations)
• 💨 🐔 Use SPECFEM's simultaneous_runs feature to improve storage requirements for large runs
• 💨 🐔 Implement alternative System strategy that deals with long queue times and preference for large job reservations
• 🔵 🐤 Mark parameters for different levels of requirements, e.g., "required", "extended" and "advanced" (and even "developer") that expose different parameters to 'seisflows configure', such that new users are inundated with too many parameters right off the bat
• 🔵 🐔 Wfdiff workflow to automatically determine the minimum resolvable period of a mesh (https://github.com/adjtomo/wfdiff)
• 🔵 🐔 Resolution testing workflow for point spread functions and zeroth moment tests

Package/Version Control/GitHub

• 🔥🐤 Improve unit and integration testing (this would be a great job for GitHub Copilot!)
• 🔥 🐣 Pre-commit framework (Flake8, Black, pre-commit, codecoverage)
• 🔥 🐣 GitHub actions on pushes to master
• 🔥 🐣 Code version citation (Zenodo)

Package Structure

• 💨 🐔Move plugins into respective directories (line_search -> optimize and misfit / adjoint -> preprocess) for better accessibility

Namespace Changes

• 💨 🐣 Change parameter ntask -> nevents because ntask can get confused with System parameters
• 💨 🐤 Change the name of the default preprocessing module, sort of confusing w.r.t e.g., default parameters

Documentation/Logging:

Updates or changes to the documentation on ReadTheDocs

• ❗🐣 Remove CDP page and point to this discussion for a more dynamic CDP document
• 💨 🐤FAQs based on previous issues

How Tos

• 💨 🐤 How to set up your directory structure (data naming formats etc.)
• 💨 🐤 How to use each of the workflows (forward, migration, inversion)
• 💨 🐤 How to user SeisFlows debug mode for more manual control of the workflow
• 💨 🐤 How to reset or roll back a line search
• 💨 🐤 How to manually rerun failed tasks and functions
• 💨 🐤 How to run a research-grade problem with SPECFEM + SeisFlows + Pyatoa
• 💨 🐤 How can new developers get started and help with the code base

Parameter File

• 🔥 🐤 Improved docstring implementation in parameter file because the current format may confuse new Users into thinking some parameters are not used

Logging (sflog.txt):

• 🔥 🐣 Better Workflow module log messages that directly print the name of function being run in the header
• 🔥 🐔 seisflows check should log out what checks are being run and whether or not they pass, so that Users know what parts of the code are being checked (less black boxy)
• 🔥 🐤 Better demarcation when new seisflows commands are run (e.g., submit, debug), so that the main log file can be visually parsed easier
• 🔥 🐤 Restructuring of log message levels, I think too many things are in debug now such that debug must be on if you want relevant information. Info should provide a useable level of inforamtion

Module Upgrades

Specific changes, updates or additions to individual modules with the package

Workflow:

• 💨 🐔 Improve Thrifty behavior to be a bit more intelligent w.r.t current iteration number
• 💨 🐔 Remove parameter iteration, not really necessary if Thrifty behavior is fixed

Solver:

• 💨 🐔 Improved schema for parameters, e.g., explicitely naming the updated parameters (vp, vs), rather than putting this behind names (acoustic, elastic), or both
• 💨 🐔Allow different meshing options for true model when using generate_data
• 💨 🐔 More checks for empty directories which usually happen when SPECFEM fails silently
• 💨 🐤 Waveform traces, currently stored in 'scratch/solver/<event_id>/traces' should really be stored in 'scratch/eval_grad' or 'scratch/eval_func' to match the organizational structure used for kernels/gradient
• 💨 🐤 Check for SPECFEM3D_GLOBE attenuation parameters set if forward wavefield will be saved, as SPECFEM will complain if not
• 💨 🐔 Allow different smoothing lengths for different parameters, currently a blanket smoothing is applied to all parameters

Preprocessing:

• 🔥 🐔 Increase transparency and User fine control over the misfit quantification process, including (customizable preprocessing functionality, manual control of windows)
• 🔥 🐔Pyaflowa should only save inversion results from the final selected step count, currently it saves everything but this is not really useful except in fringe cases, and ends up ballooning storage costs
• 🔥 🐔Controllable number of max workers when preprocessing on concurrent futures because this can easily surpass the total number of system cores
• 🔥🐤 Better error messages with traceback when processing fails and throws critical log messages. Currently only the error message is printed out but it would be nice to get full traceback

System:

• 💨 🐔 Source level checkpointing system
• 🔥 🐔Option for rerun ignored if all jobs failed because clearly something is wrong with the function setup
• 💨 🐤 If System jobs fail, try to provide more information about the failing job (e.g., the node number/name)
• 💨 🐔 Introduce a system to allow users to modify tasktimes for individual tasks, currently one tasktime is shared among all processes
• 💨 🐤 If parameter submit_to == 'direct' (or 'login') submit master job directly to login node. This can currently be achieved with seisflows submit --direct
• 💨 🐔 Improve job failure forwarding, currently failure of child process compute jobs can fail silently causing main job crash further down the line rather than at the exact process that failed (e.g., simulation fails but SeisFlows treats the job as complete)
• 💨 🐔 If more than some % of ntasks job fail, kill the main job. This should signify that something is wrong and the System should not attempt to rerun failed jobs or continue the workflow
• 💨 🐔 For 'direct' login node main jobs, model handling tasks should be handle by single-process compute jobs and not the login node

Optimize

• 🔵 🐔 Optimize the Model class which stores SPECFEM GLL model as Numpy array (use xarray or some data schema with compression like HDF5)
• 🔥 🐔 Expose line search reset and restart functions at a public and accessible level
• 🔥 🐤 Save optimization figures to a sub-directory rather than directly to the output directory
• 🔵 🐔 Remove Model class dependence on using NumPy.savez for model arrays as I/O is unnecessarily heavy (replace with HDF5? or Memmaps?)

SeisFlows Command Line Tool

• 🔵 🐔 new CL tool seisflows state for editing the State file so that Users don't have to manually edit it
• 💨 🐤 'seisflows print tasks' short descriptions of each of the sub modules
• 🔥 🐤 seisflows check should have a thumbs up when all checks pass
• 💨 🐤 seisflows unset reset parameter in parameter file to its default value
• 💨 🐤 seisflows clean --limited do not do a full clean but only delete largest scratch files for storage savings
• 💨 🐤 seisflows reset line_search should save the old line search files somewhere incase this needs to be undone