IndexError while running an acoustic example

[raulleoncz]

Hello everyone.
I've been trying to run an acoustic inversion problem using specfem2d as my solver and 'su' as the data format but I'm encountering the same error. Below there's a ss of the error.

I've trying with different parameters thinking that maybe I was forgetting something but I got the same error. Here are some of the parameters that I'm using:

workflow: inversion
data_case: synthetic
stop_after: evaluate_initial_misfit
syn_data_format: su
materials: acoustic
obs_data_format: su
stop
misfit: traveltime

I hope you can help me to understand what I'm doing wrong. Thank you.

[bch0w]

Hi @raulleoncz, thanks for raising this issue. At the moment the SU data format is underdeveloped with respect to the ASCII format option, and I suspect that may the source of your issue (see related issue: #174). I have had it on my to do list to bugfix the SU capabilities but have not had the time.

Can you try setting the data format to ASCII and seeing if you run into issues? Also if you could provide your parameter.yaml file, that might help me recreate the issue and see if I can put out a fix to get you past this error.

[raulleoncz]

Thank you so much for the quick answer!
Ok. I ran the same simulation following your advice and it worked. It appears that the problem could be related to what you already said is a bug or maybe related to the components I specified. The parameters that I used for the ascii format are these:

Modules:
workflow: inversion
system: workstation
solver: specfem2d
preprocess: default
optimize: gradient

Forward and Inversion:
data_case: synthetic
stop_after: evaluate_initial_misfit
export_traces: False
export_residuals: False
export_gradient: True
export_kernels: False
start: 1
end: 2
export_model: True
thrifty: False
iteration: 1

Workstation:
ntask: 1
nproc: 1
mpiexec: null
log_level: DEBUG
verbose: False

Solver:
syn_data_format: ascii
materials: acoustic
density: False
attenuation: False
smooth_h: 0.0
smooth_v: 0.0
components: Z
source_prefix: SOURCE
multiples: False

Preprocess:
obs_data_format: ascii
unit_output: disp
misfit: traveltime
adjoint: traveltime
normalize: []
filter: null
min_period: null
max_period: null
min_freq: null
max_freq: null
mute: []
early_slope: null
early_const: null
late_slope: null
late_const: null
short_dist: null
long_dist: null

Optimization:
preconditioner: null
step_count_max: 5
step_len_init: 0.05
step_len_max: 0.5
line_search_method: bracket

Paths:
path_model_init: /home/rleon/Example/specfem2d_workdir/OUTPUT_FILES_INIT
path_model_true: /home/rleon/Example/specfem2d_workdir/OUTPUT_FILES_TRUE
path_specfem_bin: /home/rleon/Example/specfem2d_workdir/bin
path_specfem_data: /home/rleon/Example/specfem2d_workdir/DATA

For the simulation using SU, I just changed the syn and obs format to 'su' and components to 'd', this because I was following the structure of the output files 'Ux_file_single_d' (maybe this could be my error).
Thank you again for your help.

[bch0w]

Thanks for the update @raulleoncz, glad that ASCII format works for you. Yes this is likely coming from the underdeveloped SU format option, and I think in your specific case you are correct that the filenames incorrectly searched for by SeisFlows. I hope I can carve out some time soon to work through this as it looks to be a useful feature for many.

[raulleoncz]

Update.
Hello @bch0w, sorry for the inconvenience. I've been trying to run the next step of the workflow which should stop after evaluating the gradient from kernels but the process had been failed at the last part "Generating/Processing Misfit Kernel". At the beginning I though it was something related to my specfem version which seems to have problems reading tomo files but I ran the same simulation on a cluster with a version that works well I still got the same error. I show you the error in the following images:

I checked the misfit folder and it is indeed empty. The parameters that I used are practically the same, obviously I change the system to cluster and I specified mpiexec as mpirun. I hope you can help me.
Thank you.

[bch0w]

Hi @raulleoncz, missing kernels might be explained by a few different things so we might have to poke around to find the issue. I would check that your adjoint simulations are running successfully, and if they are, check that your adjoint sources are present.

To look at the adjoint simulations you will have some log files named something like:
scratch/solver/mainsolver/adj_solver.log
scratch/solver/mainsolver/OUTPUT_FILES/output_solver.txt
The first should be empty, and the second should indicate that the adjoint simulation ran successfully

If those do not show any issue, then perhaps the adjoint sources in:
scratch/solver/mainsolver/traces/adj
There should be a number of traces in there, equal to the amount of synthetic waveforms you have in syn.

If that is also alright, then let me know and I can suggest some other places to look!

P.S. if we need to troubleshoot some more I may ask you to start a new issue just to keep a separate record

[bch0w]

Hi @raulleoncz, I am closing this issue as I think we migrated the discussion to #185 and now to #202. Please re-open if I am mistaken, thanks