specfem3d_globe output to asdf very slow #16
Comments
|
Thanks for reporting. Can you send me the STATIONS file also? I will test it on our cluster and see if I can figure out why it is so slow for this number of receivers. |
|
yep! Here it is. Laura Ermert Sonneggstrasse 5, NO H 39.3 On 24 Nov 2015, at 20:18, James Smith <notifications@github.commailto:notifications@github.com> wrote: Thanks for reporting. Can you send me the STATIONS file also? I will test it on our cluster and see if I can figure out why it is so slow for this number of receivers. � |
|
Sorry but can you send it again? I don't see the STATIONS file Also, are you using the development version from CIG? If that's the case, then your example won't work as I know output is broken if number of time steps < NSTEP_BETWEEN_OUTPUT_SEISMOS (based on your Par_file settings) for that version. The development verison on my repository fixes this issue (and quite a few other issues with the ASDF writer). It will be pushed to the CIG repository this week after I receive feedback from another user. But if I can get the STATIONS file I can test the version from my repository to make sure it works and has acceptable performance as I haven't run an example with that number of stations before. Thanks. |
|
hmm, yes, I am using the devel branch from github. However, my nr. of time steps in total (1200) is > than nstep between output seismos (1000). Laura Ermert Sonneggstrasse 5, NO H 39.3 On 24 Nov 2015, at 20:32, James Smith <notifications@github.commailto:notifications@github.com> wrote: Sorry but can you send it again? I don't see the STATIONS file Also, are you using the development version from CIG? If that's the case, then your example won't work as I know output is broken if number of time steps < NSTEP_BETWEEN_OUTPUT_SEISMOS (based on your Par_file settings) for that version. The development verison on my repository fixes this issue (and quite a few other issues with the ASDF writer). It will be pushed to the CIG repository this week after I receive feedback from another user. But if I can get the STATIONS file I can test the version from my repository to make sure it works and has acceptable performance as I haven't run an example with that number of stations before. Thanks. � 00000000 SRC -89.3208 -180.0000 0.0 0.0 |
|
Ah, yes I meant that nstep between output seismos < total then it won't work with the devel branch from github (unless you use my branch). I will test your example later today and see how it goes. Thanks. |
|
Also that STATIONS file you pasted cut off after 1565 Stations. I can just make my own file with a script so I'll just do that. |
|
Hi hi, I just tried writing everything only at the end, just to make sure that wasn’t the problem, but it’s still writing very slowly. Thanks in advance, anyway. I appreciate any suggestion. Laura Ermert Sonneggstrasse 5, NO H 39.3 On 24 Nov 2015, at 20:42, James Smith <notifications@github.commailto:notifications@github.com> wrote: Ah, yes I meant that nstep between output seismos < total then it won't work with the devel branch from github (unless you use my branch). I will test your example later today and see how it goes. Thanks. — |
|
A quick update. I ran specfem3d_globe for 25000 stations using ASDF writer and it completed in 3 minutes and 12 seconds (traces were 0.5 minutes versus 2 minutes in your case). ASDF output looks fine in ObsPy. When I attempt >30000 stations, I run into memory exceeded issues on my cluster (I'm using 8 GB/node--should be plenty). Again thanks for pointing this out. It seems there is a problem with the ASDF writer using too much memory for this number of stations. When I use SAC binary, the solver works fine so this is something I should fix. |
|
Hi James! Happy new year! I wanted to follow up on the question of writing ~50000 seismograms from specfem3Dglobe to ASDF file…I think Lion said that this issue was fixed. Now I was wondering what I need to reinstall to make it work on the cluster here. Just pull the newest specfem development branch? Best, Laura Ermert Sonneggstrasse 5, NO H 39.3 On 25 Nov 2015, at 00:22, James Smith <notifications@github.commailto:notifications@github.com> wrote: A quick update. I ran specfem3d_globe for 25000 stations using ASDF writer and it completed in 3 minutes and 12 seconds (traces were 0.5 minutes versus 2 minutes in your case). ASDF output looks fine in ObsPy. When I attempt >30000 stations, I run into memory exceeded issues on my cluster (I'm using 8 GB/node--should be plenty). Again thanks for pointing this out. It seems there is a problem with the ASDF writer using too much memory for this number of stations. When I use SAC binary, the solver works fine so this is something I should fix. — |
|
Hi Laura, Thanks! It's too early to say that this issue is fixed, but I now get reasonable performance on 384 cores on ~6000 seismograms for 60 minute duration. I'm going to push my changes in the next day or so and I'll update you when I do. I need to test on a larger number of receivers and a larger number of cores. Cheers, From: lermert [notifications@github.com] Hi James! Happy new year! I wanted to follow up on the question of writing ~50000 seismograms from specfem3Dglobe to ASDF file…I think Lion said that this issue was fixed. Now I was wondering what I need to reinstall to make it work on the cluster here. Just pull the newest specfem development branch? Best, Laura Ermert Sonneggstrasse 5, NO H 39.3 On 25 Nov 2015, at 00:22, James Smith <notifications@github.commailto:notifications@github.com> wrote: A quick update. I ran specfem3d_globe for 25000 stations using ASDF writer and it completed in 3 minutes and 12 seconds (traces were 0.5 minutes versus 2 minutes in your case). ASDF output looks fine in ObsPy. When I attempt >30000 stations, I run into memory exceeded issues on my cluster (I'm using 8 GB/node--should be plenty). Again thanks for pointing this out. It seems there is a problem with the ASDF writer using too much memory for this number of stations. When I use SAC binary, the solver works fine so this is something I should fix. — — |
|
Hi Laura, I’ve uploaded a newer asdf-lib with the latest library fix. There is still heavy memory usage when I tasted it, so it may not work for your case yet. If you have time to try it, let me know if there are any improvements. I’ll continue working on this as time permits. Cheers, On Jan 4, 2016, at 10:55 AM, lermert <notifications@github.commailto:notifications@github.com> wrote: Hi James! Happy new year! I wanted to follow up on the question of writing ~50000 seismograms from specfem3Dglobe to ASDF file…I think Lion said that this issue was fixed. Now I was wondering what I need to reinstall to make it work on the cluster here. Just pull the newest specfem development branch? Best, Laura Ermert Sonneggstrasse 5, NO H 39.3 On 25 Nov 2015, at 00:22, James Smith <notifications@github.commailto:notifications@github.commailto:notifications@github.com> wrote: A quick update. I ran specfem3d_globe for 25000 stations using ASDF writer and it completed in 3 minutes and 12 seconds (traces were 0.5 minutes versus 2 minutes in your case). ASDF output looks fine in ObsPy. When I attempt >30000 stations, I run into memory exceeded issues on my cluster (I'm using 8 GB/node--should be plenty). Again thanks for pointing this out. It seems there is a problem with the ASDF writer using too much memory for this number of stations. When I use SAC binary, the solver works fine so this is something I should fix. — — |
|
Hi James, I have a short question…I have trouble compiling with ADIOS support; is that needed for ASDF? Otherwise I’ll be easier off without. Also, our compiler here keeps complaining about the -Wl flag in this line: SPECFEM_LINK_FLAGS += $(ASDF_LIBS) -lhdf5hl_fortran -lhdf5_hl -lhdf5 -Wl -lstdc++ In the src/specfem3D/rules.mk, so I removed this flag; not sure if that is a problem. Thanks and best, LauraLaura Ermert Sonneggstrasse 5, NO H 39.3 On 05 Jan 2016, at 23:06, James Smith <notifications@github.commailto:notifications@github.com> wrote: Hi Laura, I’ve uploaded a newer asdf-lib with the latest library fix. There is still heavy memory usage when I tasted it, so it may not work for your case yet. If you have time to try it, let me know if there are any improvements. I’ll continue working on this as time permits. Cheers, On Jan 4, 2016, at 10:55 AM, lermert <notifications@github.commailto:notifications@github.commailto:notifications@github.com> wrote: Hi James! Happy new year! I wanted to follow up on the question of writing ~50000 seismograms from specfem3Dglobe to ASDF file…I think Lion said that this issue was fixed. Now I was wondering what I need to reinstall to make it work on the cluster here. Just pull the newest specfem development branch? Best, Laura Ermert Sonneggstrasse 5, NO H 39.3 On 25 Nov 2015, at 00:22, James Smith <notifications@github.commailto:notifications@github.commailto:notifications@github.commailto:notifications@github.com> wrote: A quick update. I ran specfem3d_globe for 25000 stations using ASDF writer and it completed in 3 minutes and 12 seconds (traces were 0.5 minutes versus 2 minutes in your case). ASDF output looks fine in ObsPy. When I attempt >30000 stations, I run into memory exceeded issues on my cluster (I'm using 8 GB/node--should be plenty). Again thanks for pointing this out. It seems there is a problem with the ASDF writer using too much memory for this number of stations. When I use SAC binary, the solver works fine so this is something I should fix. — — — |
|
Hi Laura, ADIOS isn’t need for ASDF. The -WI option should be for warnings that violate fortran. I didn’t get any problems on my end. I’m using intel-13.1 for compiling, so maybe there’s a problem with other compilers. Thanks for letting me know, On Jan 6, 2016, at 11:31 AM, lermert <notifications@github.commailto:notifications@github.com> wrote: Hi James, I have a short question…I have trouble compiling with ADIOS support; is that needed for ASDF? Otherwise I’ll be easier off without. Also, our compiler here keeps complaining about the -Wl flag in this line: SPECFEM_LINK_FLAGS += $(ASDF_LIBS) -lhdf5hl_fortran -lhdf5_hl -lhdf5 -Wl -lstdc++ In the src/specfem3D/rules.mk, so I removed this flag; not sure if that is a problem. Thanks and best, LauraLaura Ermert Sonneggstrasse 5, NO H 39.3 On 05 Jan 2016, at 23:06, James Smith <notifications@github.commailto:notifications@github.commailto:notifications@github.com> wrote: Hi Laura, I’ve uploaded a newer asdf-lib with the latest library fix. There is still heavy memory usage when I tasted it, so it may not work for your case yet. If you have time to try it, let me know if there are any improvements. I’ll continue working on this as time permits. Cheers, On Jan 4, 2016, at 10:55 AM, lermert <notifications@github.commailto:notifications@github.commailto:notifications@github.commailto:notifications@github.com> wrote: Hi James! Happy new year! I wanted to follow up on the question of writing ~50000 seismograms from specfem3Dglobe to ASDF file…I think Lion said that this issue was fixed. Now I was wondering what I need to reinstall to make it work on the cluster here. Just pull the newest specfem development branch? Best, Laura Ermert Sonneggstrasse 5, NO H 39.3 On 25 Nov 2015, at 00:22, James Smith <notifications@github.commailto:notifications@github.commailto:notifications@github.commailto:notifications@github.commailto:notifications@github.com> wrote: A quick update. I ran specfem3d_globe for 25000 stations using ASDF writer and it completed in 3 minutes and 12 seconds (traces were 0.5 minutes versus 2 minutes in your case). ASDF output looks fine in ObsPy. When I attempt >30000 stations, I run into memory exceeded issues on my cluster (I'm using 8 GB/node--should be plenty). Again thanks for pointing this out. It seems there is a problem with the ASDF writer using too much memory for this number of stations. When I use SAC binary, the solver works fine so this is something I should fix. — — — — |
|
Also, please update the asdf-lib to the latest version before testing the ASDF writer. I added a fix yesterday for the library. On Jan 6, 2016, at 11:40 AM, James A. Smith <jas11@exchange.Princeton.EDUmailto:jas11@exchange.princeton.edu> wrote: Hi Laura, ADIOS isn’t need for ASDF. The -WI option should be for warnings that violate fortran. I didn’t get any problems on my end. I’m using intel-13.1 for compiling, so maybe there’s a problem with other compilers. Thanks for letting me know, On Jan 6, 2016, at 11:31 AM, lermert <notifications@github.commailto:notifications@github.com> wrote: Hi James, I have a short question…I have trouble compiling with ADIOS support; is that needed for ASDF? Otherwise I’ll be easier off without. Also, our compiler here keeps complaining about the -Wl flag in this line: SPECFEM_LINK_FLAGS += $(ASDF_LIBS) -lhdf5hl_fortran -lhdf5_hl -lhdf5 -Wl -lstdc++ In the src/specfem3D/rules.mk, so I removed this flag; not sure if that is a problem. Thanks and best, LauraLaura Ermert Sonneggstrasse 5, NO H 39.3 On 05 Jan 2016, at 23:06, James Smith <notifications@github.commailto:notifications@github.commailto:notifications@github.com> wrote: Hi Laura, I’ve uploaded a newer asdf-lib with the latest library fix. There is still heavy memory usage when I tasted it, so it may not work for your case yet. If you have time to try it, let me know if there are any improvements. I’ll continue working on this as time permits. Cheers, On Jan 4, 2016, at 10:55 AM, lermert <notifications@github.commailto:notifications@github.commailto:notifications@github.commailto:notifications@github.com> wrote: Hi James! Happy new year! I wanted to follow up on the question of writing ~50000 seismograms from specfem3Dglobe to ASDF file…I think Lion said that this issue was fixed. Now I was wondering what I need to reinstall to make it work on the cluster here. Just pull the newest specfem development branch? Best, Laura Ermert Sonneggstrasse 5, NO H 39.3 On 25 Nov 2015, at 00:22, James Smith <notifications@github.commailto:notifications@github.commailto:notifications@github.commailto:notifications@github.commailto:notifications@github.com> wrote: A quick update. I ran specfem3d_globe for 25000 stations using ASDF writer and it completed in 3 minutes and 12 seconds (traces were 0.5 minutes versus 2 minutes in your case). ASDF output looks fine in ObsPy. When I attempt >30000 stations, I run into memory exceeded issues on my cluster (I'm using 8 GB/node--should be plenty). Again thanks for pointing this out. It seems there is a problem with the ASDF writer using too much memory for this number of stations. When I use SAC binary, the solver works fine so this is something I should fix. — — — — |
|
Hi Hi James! It took me a while to recompile everything — had to remember how to do that, which should be faster now in the future. Unfortunately when I ran it on a 20-minute seismogram test, I received a bunch of error messages: The synthetics.h5 file is opened, but then the waveforms not successfully written, although the station names are successfully included. Let me know if I can change something to make it work. HDF5-DIAG: Error detected in HDF5 (1.8.14) MPI-process 2HDF5-DIAG: Error detected in HDF5 (1.8.14) : ….. HDF5-DIAG: Error detected in HDF5 (1.8.14) MPI-process 16: minor: Unable to close fileLaura Ermert Sonneggstrasse 5, NO H 39.3 |
|
Hi Laura, It looks like my merge of the original_ASDF branch that has the working writer to the devel branch on github didn’t work. Sorry about that. It looks like the old writer is still on the devel branch. I will fix it later today and update you. Cheers, On Jan 7, 2016, at 11:22 AM, lermert <notifications@github.commailto:notifications@github.com> wrote: Hi Hi James! It took me a while to recompile everything — had to remember how to do that, which should be faster now in the future. Unfortunately when I ran it on a 20-minute seismogram test, I received a bunch of error messages: The synthetics.h5 file is opened, but then the waveforms not successfully written, although the station names are successfully included. Let me know if I can change something to make it work. HDF5-DIAG: Error detected in HDF5 (1.8.14) MPI-process 2HDF5-DIAG: Error detected in HDF5 (1.8.14) : ….. HDF5-DIAG: Error detected in HDF5 (1.8.14) MPI-process 16: minor: Unable to close fileLaura Ermert Sonneggstrasse 5, NO H 39.3 — |
|
No problem! Thanks a lot :) (at least I know again how to compile, haha) Laura Ermert Sonneggstrasse 5, NO H 39.3 On 07 Jan 2016, at 18:51, James Smith <notifications@github.commailto:notifications@github.com> wrote: Hi Laura, It looks like my merge of the original_ASDF branch that has the working writer to the devel branch on github didn’t work. Sorry about that. It looks like the old writer is still on the devel branch. I will fix it later today and update you. Cheers, On Jan 7, 2016, at 11:22 AM, lermert <notifications@github.commailto:notifications@github.commailto:notifications@github.com> wrote: Hi Hi James! It took me a while to recompile everything — had to remember how to do that, which should be faster now in the future. Unfortunately when I ran it on a 20-minute seismogram test, I received a bunch of error messages: The synthetics.h5 file is opened, but then the waveforms not successfully written, although the station names are successfully included. Let me know if I can change something to make it work. HDF5-DIAG: Error detected in HDF5 (1.8.14) MPI-process 2HDF5-DIAG: Error detected in HDF5 (1.8.14) : ….. HDF5-DIAG: Error detected in HDF5 (1.8.14) MPI-process 16: minor: Unable to close fileLaura Ermert Sonneggstrasse 5, NO H 39.3 — — |
|
Can you try again? Also, please double check you compiled and linked to the latest asdf-lib. Thanks, On Jan 7, 2016, at 12:55 PM, lermert <notifications@github.commailto:notifications@github.com> wrote: No problem! Thanks a lot :) (at least I know again how to compile, haha) Laura Ermert Sonneggstrasse 5, NO H 39.3 On 07 Jan 2016, at 18:51, James Smith <notifications@github.commailto:notifications@github.commailto:notifications@github.com> wrote: Hi Laura, It looks like my merge of the original_ASDF branch that has the working writer to the devel branch on github didn’t work. Sorry about that. It looks like the old writer is still on the devel branch. I will fix it later today and update you. Cheers, On Jan 7, 2016, at 11:22 AM, lermert <notifications@github.commailto:notifications@github.commailto:notifications@github.commailto:notifications@github.com> wrote: Hi Hi James! It took me a while to recompile everything — had to remember how to do that, which should be faster now in the future. Unfortunately when I ran it on a 20-minute seismogram test, I received a bunch of error messages: The synthetics.h5 file is opened, but then the waveforms not successfully written, although the station names are successfully included. Let me know if I can change something to make it work. HDF5-DIAG: Error detected in HDF5 (1.8.14) MPI-process 2HDF5-DIAG: Error detected in HDF5 (1.8.14) : ….. HDF5-DIAG: Error detected in HDF5 (1.8.14) MPI-process 16: minor: Unable to close fileLaura Ermert Sonneggstrasse 5, NO H 39.3 — — — |
|
Hi James! Just to be sure I get the right one, I am cloning the master branch on https://github.com/SeismicData/asdf-library...correct me if I need another branch. I'll try again first thing tomorrow! Institut f�r Geophysik, ETH Z�rich From: James Smith [notifications@github.com] Can you try again? Also, please double check you compiled and linked to the latest asdf-lib. Thanks, On Jan 7, 2016, at 12:55 PM, lermert <notifications@github.commailto:notifications@github.com> wrote: No problem! Thanks a lot :) (at least I know again how to compile, haha) Laura Ermert Sonneggstrasse 5, NO H 39.3 On 07 Jan 2016, at 18:51, James Smith <notifications@github.commailto:notifications@github.commailto:notifications@github.com> wrote: Hi Laura, It looks like my merge of the original_ASDF branch that has the working writer to the devel branch on github didn�t work. Sorry about that. It looks like the old writer is still on the devel branch. I will fix it later today and update you. Cheers, On Jan 7, 2016, at 11:22 AM, lermert <notifications@github.commailto:notifications@github.commailto:notifications@github.commailto:notifications@github.com> wrote: Hi Hi James! It took me a while to recompile everything � had to remember how to do that, which should be faster now in the future. Unfortunately when I ran it on a 20-minute seismogram test, I received a bunch of error messages: The synthetics.h5 file is opened, but then the waveforms not successfully written, although the station names are successfully included. Let me know if I can change something to make it work. HDF5-DIAG: Error detected in HDF5 (1.8.14) MPI-process 2HDF5-DIAG: Error detected in HDF5 (1.8.14) : �.. HDF5-DIAG: Error detected in HDF5 (1.8.14) MPI-process 16: minor: Unable to close fileLaura Ermert Sonneggstrasse 5, NO H 39.3 � � � � |
|
Hi James! Just to be sure I get the right one, I am cloning the master branch on https://github.com/SeismicData/asdf-library...just correct me if I need another branch. I'll try again first thing tomorrow morning! Institut f�r Geophysik, ETH Z�rich From: James Smith [notifications@github.com] Can you try again? Also, please double check you compiled and linked to the latest asdf-lib. Thanks, On Jan 7, 2016, at 12:55 PM, lermert <notifications@github.commailto:notifications@github.com> wrote: No problem! Thanks a lot :) (at least I know again how to compile, haha) Laura Ermert Sonneggstrasse 5, NO H 39.3 On 07 Jan 2016, at 18:51, James Smith <notifications@github.commailto:notifications@github.commailto:notifications@github.com> wrote: Hi Laura, It looks like my merge of the original_ASDF branch that has the working writer to the devel branch on github didn�t work. Sorry about that. It looks like the old writer is still on the devel branch. I will fix it later today and update you. Cheers, On Jan 7, 2016, at 11:22 AM, lermert <notifications@github.commailto:notifications@github.commailto:notifications@github.commailto:notifications@github.com> wrote: Hi Hi James! It took me a while to recompile everything � had to remember how to do that, which should be faster now in the future. Unfortunately when I ran it on a 20-minute seismogram test, I received a bunch of error messages: The synthetics.h5 file is opened, but then the waveforms not successfully written, although the station names are successfully included. Let me know if I can change something to make it work. HDF5-DIAG: Error detected in HDF5 (1.8.14) MPI-process 2HDF5-DIAG: Error detected in HDF5 (1.8.14) : �.. HDF5-DIAG: Error detected in HDF5 (1.8.14) MPI-process 16: minor: Unable to close fileLaura Ermert Sonneggstrasse 5, NO H 39.3 � � � � |
|
Yup, it’s the master branch. Sounds good! On Jan 7, 2016, at 4:56 PM, lermert <notifications@github.commailto:notifications@github.com> wrote: Hi James! Just to be sure I get the right one, I am cloning the master branch on https://github.com/SeismicData/asdf-library...just correct me if I need another branch. I'll try again first thing tomorrow morning! Institut f�r Geophysik, ETH Z�rich From: James Smith [notifications@github.commailto:notifications@github.com] Can you try again? Also, please double check you compiled and linked to the latest asdf-lib. Thanks, On Jan 7, 2016, at 12:55 PM, lermert <notifications@github.commailto:notifications@github.commailto:notifications@github.com> wrote: No problem! Thanks a lot :) (at least I know again how to compile, haha) Laura Ermert Sonneggstrasse 5, NO H 39.3 On 07 Jan 2016, at 18:51, James Smith <notifications@github.commailto:notifications@github.commailto:notifications@github.commailto:notifications@github.com> wrote: Hi Laura, It looks like my merge of the original_ASDF branch that has the working writer to the devel branch on github didn�t work. Sorry about that. It looks like the old writer is still on the devel branch. I will fix it later today and update you. Cheers, On Jan 7, 2016, at 11:22 AM, lermert <notifications@github.commailto:notifications@github.commailto:notifications@github.commailto:notifications@github.commailto:notifications@github.com> wrote: Hi Hi James! It took me a while to recompile everything � had to remember how to do that, which should be faster now in the future. Unfortunately when I ran it on a 20-minute seismogram test, I received a bunch of error messages: The synthetics.h5 file is opened, but then the waveforms not successfully written, although the station names are successfully included. Let me know if I can change something to make it work. HDF5-DIAG: Error detected in HDF5 (1.8.14) MPI-process 2HDF5-DIAG: Error detected in HDF5 (1.8.14) : �.. HDF5-DIAG: Error detected in HDF5 (1.8.14) MPI-process 16: minor: Unable to close fileLaura Ermert Sonneggstrasse 5, NO H 39.3 � � � � — |
|
Hi James, unfortunately, still the same. I used the master branch at asdf-library, deleted my old executable, compiled, linked against this one during specfem compilation, compiled specfem, but no change. Do you think it could be related to the settings I use for compilation? I pasted the module list below, the last few are the ones I loaded for compiling asdf. Laura Ermert Sonneggstrasse 5, NO H 39.3
On 07 Jan 2016, at 23:13, James Smith <notifications@github.commailto:notifications@github.com> wrote: Yup, it’s the master branch. Sounds good! On Jan 7, 2016, at 4:56 PM, lermert <notifications@github.commailto:notifications@github.commailto:notifications@github.com> wrote: Hi James! Just to be sure I get the right one, I am cloning the master branch on https://github.com/SeismicData/asdf-library...just correct me if I need another branch. I'll try again first thing tomorrow morning! Institut f�r Geophysik, ETH Z�rich From: James Smith [notifications@github.commailto:notifications@github.commailto:notifications@github.com] Can you try again? Also, please double check you compiled and linked to the latest asdf-lib. Thanks, On Jan 7, 2016, at 12:55 PM, lermert <notifications@github.commailto:notifications@github.commailto:notifications@github.commailto:notifications@github.com> wrote: No problem! Thanks a lot :) (at least I know again how to compile, haha) Laura Ermert Sonneggstrasse 5, NO H 39.3 On 07 Jan 2016, at 18:51, James Smith <notifications@github.commailto:notifications@github.commailto:notifications@github.commailto:notifications@github.commailto:notifications@github.com> wrote: Hi Laura, It looks like my merge of the original_ASDF branch that has the working writer to the devel branch on github didn�t work. Sorry about that. It looks like the old writer is still on the devel branch. I will fix it later today and update you. Cheers, On Jan 7, 2016, at 11:22 AM, lermert <notifications@github.commailto:notifications@github.commailto:notifications@github.commailto:notifications@github.commailto:notifications@github.commailto:notifications@github.com> wrote: Hi Hi James! It took me a while to recompile everything � had to remember how to do that, which should be faster now in the future. Unfortunately when I ran it on a 20-minute seismogram test, I received a bunch of error messages: The synthetics.h5 file is opened, but then the waveforms not successfully written, although the station names are successfully included. Let me know if I can change something to make it work. HDF5-DIAG: Error detected in HDF5 (1.8.14) MPI-process 2HDF5-DIAG: Error detected in HDF5 (1.8.14) : �.. HDF5-DIAG: Error detected in HDF5 (1.8.14) MPI-process 16: minor: Unable to close fileLaura Ermert Sonneggstrasse 5, NO H 39.3 � � � � — — |
|
Hi Laura, I’ve seen this error before when I used an older version of write_output_ASDF.f90, so I thought it was related to that. I’ll test a fresh clone of the repo here. I’m using an older version of hdf5 here (1.8.10), but I don’t think that is a problem. The other test I can try is on a different cluster. I should have time today to try it. James On Jan 8, 2016, at 5:39 AM, lermert <notifications@github.commailto:notifications@github.com> wrote: Hi James, unfortunately, still the same. I used the master branch at asdf-library, deleted my old executable, compiled, linked against this one during specfem compilation, compiled specfem, but no change. Do you think it could be related to the settings I use for compilation? I pasted the module list below, the last few are the ones I loaded for compiling asdf. Laura Ermert Sonneggstrasse 5, NO H 39.3
On 07 Jan 2016, at 23:13, James Smith <notifications@github.commailto:notifications@github.commailto:notifications@github.com> wrote: Yup, it’s the master branch. Sounds good! On Jan 7, 2016, at 4:56 PM, lermert <notifications@github.commailto:notifications@github.commailto:notifications@github.commailto:notifications@github.com> wrote: Hi James! Just to be sure I get the right one, I am cloning the master branch on https://github.com/SeismicData/asdf-library...just correct me if I need another branch. I'll try again first thing tomorrow morning! Institut f�r Geophysik, ETH Z�rich From: James Smith [notifications@github.commailto:notifications@github.commailto:notifications@github.commailto:notifications@github.com] Can you try again? Also, please double check you compiled and linked to the latest asdf-lib. Thanks, On Jan 7, 2016, at 12:55 PM, lermert <notifications@github.commailto:notifications@github.commailto:notifications@github.commailto:notifications@github.commailto:notifications@github.com> wrote: No problem! Thanks a lot :) (at least I know again how to compile, haha) Laura Ermert Sonneggstrasse 5, NO H 39.3 On 07 Jan 2016, at 18:51, James Smith <notifications@github.commailto:notifications@github.commailto:notifications@github.commailto:notifications@github.commailto:notifications@github.commailto:notifications@github.com> wrote: Hi Laura, It looks like my merge of the original_ASDF branch that has the working writer to the devel branch on github didn�t work. Sorry about that. It looks like the old writer is still on the devel branch. I will fix it later today and update you. Cheers, On Jan 7, 2016, at 11:22 AM, lermert <notifications@github.commailto:notifications@github.commailto:notifications@github.commailto:notifications@github.commailto:notifications@github.commailto:notifications@github.commailto:notifications@github.com> wrote: Hi Hi James! It took me a while to recompile everything � had to remember how to do that, which should be faster now in the future. Unfortunately when I ran it on a 20-minute seismogram test, I received a bunch of error messages: The synthetics.h5 file is opened, but then the waveforms not successfully written, although the station names are successfully included. Let me know if I can change something to make it work. HDF5-DIAG: Error detected in HDF5 (1.8.14) MPI-process 2HDF5-DIAG: Error detected in HDF5 (1.8.14) : �.. HDF5-DIAG: Error detected in HDF5 (1.8.14) MPI-process 16: minor: Unable to close fileLaura Ermert Sonneggstrasse 5, NO H 39.3 � � � � — — — |
|
Hi James! I think I might have an idea of the problem. Actually, as I watched the configuration of asdf I saw it does not use the cray-hdf5-parallel, but another hdf5 located in my anaconda directory. Is there a way to tell cmake not to use that hdf5, but rather the cray-hdf5-parallel? For now I will just remove and reinstall anaconda. Laura Ermert Sonneggstrasse 5, NO H 39.3 On 08 Jan 2016, at 16:53, James Smith <notifications@github.commailto:notifications@github.com> wrote: Hi Laura, I’ve seen this error before when I used an older version of write_output_ASDF.f90, so I thought it was related to that. I’ll test a fresh clone of the repo here. I’m using an older version of hdf5 here (1.8.10), but I don’t think that is a problem. The other test I can try is on a different cluster. I should have time today to try it. James On Jan 8, 2016, at 5:39 AM, lermert <notifications@github.commailto:notifications@github.commailto:notifications@github.com> wrote: Hi James, unfortunately, still the same. I used the master branch at asdf-library, deleted my old executable, compiled, linked against this one during specfem compilation, compiled specfem, but no change. Do you think it could be related to the settings I use for compilation? I pasted the module list below, the last few are the ones I loaded for compiling asdf. Laura Ermert Sonneggstrasse 5, NO H 39.3
On 07 Jan 2016, at 23:13, James Smith <notifications@github.commailto:notifications@github.commailto:notifications@github.commailto:notifications@github.com> wrote: Yup, it’s the master branch. Sounds good! On Jan 7, 2016, at 4:56 PM, lermert <notifications@github.commailto:notifications@github.commailto:notifications@github.commailto:notifications@github.commailto:notifications@github.com> wrote: Hi James! Just to be sure I get the right one, I am cloning the master branch on https://github.com/SeismicData/asdf-library...just correct me if I need another branch. I'll try again first thing tomorrow morning! Institut f�r Geophysik, ETH Z�rich From: James Smith [notifications@github.commailto:notifications@github.commailto:notifications@github.commailto:notifications@github.commailto:notifications@github.com] Can you try again? Also, please double check you compiled and linked to the latest asdf-lib. Thanks, On Jan 7, 2016, at 12:55 PM, lermert <notifications@github.commailto:notifications@github.commailto:notifications@github.commailto:notifications@github.commailto:notifications@github.commailto:notifications@github.com> wrote: No problem! Thanks a lot :) (at least I know again how to compile, haha) Laura Ermert Sonneggstrasse 5, NO H 39.3 On 07 Jan 2016, at 18:51, James Smith <notifications@github.commailto:notifications@github.commailto:notifications@github.commailto:notifications@github.commailto:notifications@github.commailto:notifications@github.commailto:notifications@github.com> wrote: Hi Laura, It looks like my merge of the original_ASDF branch that has the working writer to the devel branch on github didn�t work. Sorry about that. It looks like the old writer is still on the devel branch. I will fix it later today and update you. Cheers, On Jan 7, 2016, at 11:22 AM, lermert <notifications@github.commailto:notifications@github.commailto:notifications@github.commailto:notifications@github.commailto:notifications@github.commailto:notifications@github.commailto:notifications@github.commailto:notifications@github.com> wrote: Hi Hi James! It took me a while to recompile everything � had to remember how to do that, which should be faster now in the future. Unfortunately when I ran it on a 20-minute seismogram test, I received a bunch of error messages: The synthetics.h5 file is opened, but then the waveforms not successfully written, although the station names are successfully included. Let me know if I can change something to make it work. HDF5-DIAG: Error detected in HDF5 (1.8.14) MPI-process 2HDF5-DIAG: Error detected in HDF5 (1.8.14) : �.. HDF5-DIAG: Error detected in HDF5 (1.8.14) MPI-process 16: minor: Unable to close fileLaura Ermert Sonneggstrasse 5, NO H 39.3 � � � � — — — — |
|
Hi again, sorry, my last question was probably stupid. I guess I can just prepend the path of the hdf5 I would like. I compiled now with an older hdf5 library, but still get the same. Let me know what you find out…cheers Laura Ermert Sonneggstrasse 5, NO H 39.3 On 08 Jan 2016, at 16:53, James Smith <notifications@github.commailto:notifications@github.com> wrote: Hi Laura, I’ve seen this error before when I used an older version of write_output_ASDF.f90, so I thought it was related to that. I’ll test a fresh clone of the repo here. I’m using an older version of hdf5 here (1.8.10), but I don’t think that is a problem. The other test I can try is on a different cluster. I should have time today to try it. James On Jan 8, 2016, at 5:39 AM, lermert <notifications@github.commailto:notifications@github.commailto:notifications@github.com> wrote: Hi James, unfortunately, still the same. I used the master branch at asdf-library, deleted my old executable, compiled, linked against this one during specfem compilation, compiled specfem, but no change. Do you think it could be related to the settings I use for compilation? I pasted the module list below, the last few are the ones I loaded for compiling asdf. Laura Ermert Sonneggstrasse 5, NO H 39.3
On 07 Jan 2016, at 23:13, James Smith <notifications@github.commailto:notifications@github.commailto:notifications@github.commailto:notifications@github.com> wrote: Yup, it’s the master branch. Sounds good! On Jan 7, 2016, at 4:56 PM, lermert <notifications@github.commailto:notifications@github.commailto:notifications@github.commailto:notifications@github.commailto:notifications@github.com> wrote: Hi James! Just to be sure I get the right one, I am cloning the master branch on https://github.com/SeismicData/asdf-library...just correct me if I need another branch. I'll try again first thing tomorrow morning! Institut f�r Geophysik, ETH Z�rich From: James Smith [notifications@github.commailto:notifications@github.commailto:notifications@github.commailto:notifications@github.commailto:notifications@github.com] Can you try again? Also, please double check you compiled and linked to the latest asdf-lib. Thanks, On Jan 7, 2016, at 12:55 PM, lermert <notifications@github.commailto:notifications@github.commailto:notifications@github.commailto:notifications@github.commailto:notifications@github.commailto:notifications@github.com> wrote: No problem! Thanks a lot :) (at least I know again how to compile, haha) Laura Ermert Sonneggstrasse 5, NO H 39.3 On 07 Jan 2016, at 18:51, James Smith <notifications@github.commailto:notifications@github.commailto:notifications@github.commailto:notifications@github.commailto:notifications@github.commailto:notifications@github.commailto:notifications@github.com> wrote: Hi Laura, It looks like my merge of the original_ASDF branch that has the working writer to the devel branch on github didn�t work. Sorry about that. It looks like the old writer is still on the devel branch. I will fix it later today and update you. Cheers, On Jan 7, 2016, at 11:22 AM, lermert <notifications@github.commailto:notifications@github.commailto:notifications@github.commailto:notifications@github.commailto:notifications@github.commailto:notifications@github.commailto:notifications@github.commailto:notifications@github.com> wrote: Hi Hi James! It took me a while to recompile everything � had to remember how to do that, which should be faster now in the future. Unfortunately when I ran it on a 20-minute seismogram test, I received a bunch of error messages: The synthetics.h5 file is opened, but then the waveforms not successfully written, although the station names are successfully included. Let me know if I can change something to make it work. HDF5-DIAG: Error detected in HDF5 (1.8.14) MPI-process 2HDF5-DIAG: Error detected in HDF5 (1.8.14) : �.. HDF5-DIAG: Error detected in HDF5 (1.8.14) MPI-process 16: minor: Unable to close fileLaura Ermert Sonneggstrasse 5, NO H 39.3 � � � � — — — — |
|
Hi Laura, The fresh clone works for me. I am having some issues with gnu compilers, but that doesn’t explain your error. Are you using the default Par_fie and STATIONS file? Thanks, On Jan 8, 2016, at 12:13 PM, lermert <notifications@github.commailto:notifications@github.com> wrote: Hi again, sorry, my last question was probably stupid. I guess I can just prepend the path of the hdf5 I would like. I compiled now with an older hdf5 library, but still get the same. Let me know what you find out…cheers Laura Ermert Sonneggstrasse 5, NO H 39.3 On 08 Jan 2016, at 16:53, James Smith <notifications@github.commailto:notifications@github.commailto:notifications@github.com> wrote: Hi Laura, I’ve seen this error before when I used an older version of write_output_ASDF.f90, so I thought it was related to that. I’ll test a fresh clone of the repo here. I’m using an older version of hdf5 here (1.8.10), but I don’t think that is a problem. The other test I can try is on a different cluster. I should have time today to try it. James On Jan 8, 2016, at 5:39 AM, lermert <notifications@github.commailto:notifications@github.commailto:notifications@github.commailto:notifications@github.com> wrote: Hi James, unfortunately, still the same. I used the master branch at asdf-library, deleted my old executable, compiled, linked against this one during specfem compilation, compiled specfem, but no change. Do you think it could be related to the settings I use for compilation? I pasted the module list below, the last few are the ones I loaded for compiling asdf. Laura Ermert Sonneggstrasse 5, NO H 39.3
On 07 Jan 2016, at 23:13, James Smith <notifications@github.commailto:notifications@github.commailto:notifications@github.commailto:notifications@github.commailto:notifications@github.com> wrote: Yup, it’s the master branch. Sounds good! On Jan 7, 2016, at 4:56 PM, lermert <notifications@github.commailto:notifications@github.commailto:notifications@github.commailto:notifications@github.commailto:notifications@github.commailto:notifications@github.com> wrote: Hi James! Just to be sure I get the right one, I am cloning the master branch on https://github.com/SeismicData/asdf-library...just correct me if I need another branch. I'll try again first thing tomorrow morning! Institut f�r Geophysik, ETH Z�rich From: James Smith [notifications@github.commailto:notifications@github.commailto:notifications@github.commailto:notifications@github.commailto:notifications@github.commailto:notifications@github.com] Can you try again? Also, please double check you compiled and linked to the latest asdf-lib. Thanks, On Jan 7, 2016, at 12:55 PM, lermert <notifications@github.commailto:notifications@github.commailto:notifications@github.commailto:notifications@github.commailto:notifications@github.commailto:notifications@github.commailto:notifications@github.com> wrote: No problem! Thanks a lot :) (at least I know again how to compile, haha) Laura Ermert Sonneggstrasse 5, NO H 39.3 On 07 Jan 2016, at 18:51, James Smith <notifications@github.commailto:notifications@github.commailto:notifications@github.commailto:notifications@github.commailto:notifications@github.commailto:notifications@github.commailto:notifications@github.commailto:notifications@github.com> wrote: Hi Laura, It looks like my merge of the original_ASDF branch that has the working writer to the devel branch on github didn�t work. Sorry about that. It looks like the old writer is still on the devel branch. I will fix it later today and update you. Cheers, On Jan 7, 2016, at 11:22 AM, lermert <notifications@github.commailto:notifications@github.commailto:notifications@github.commailto:notifications@github.commailto:notifications@github.commailto:notifications@github.commailto:notifications@github.commailto:notifications@github.commailto:notifications@github.com> wrote: Hi Hi James! It took me a while to recompile everything � had to remember how to do that, which should be faster now in the future. Unfortunately when I ran it on a 20-minute seismogram test, I received a bunch of error messages: The synthetics.h5 file is opened, but then the waveforms not successfully written, although the station names are successfully included. Let me know if I can change something to make it work. HDF5-DIAG: Error detected in HDF5 (1.8.14) MPI-process 2HDF5-DIAG: Error detected in HDF5 (1.8.14) : �.. HDF5-DIAG: Error detected in HDF5 (1.8.14) MPI-process 16: minor: Unable to close fileLaura Ermert Sonneggstrasse 5, NO H 39.3 � � � � — — — — — |
Hej there, I am trying to output specfem 3D globe synthetics in large number, like 100 000 seismograms per component. Writing these to binary sac in parallel is very fast, (for a short, very low frequency test setup about 10 minutes) but the handling of the data becomes extremely cumbersome. Thus, asdf would be perfect for my purposes.
I am running specfem3D_globe on Piz Daint (http://www.cscs.ch/computers/piz_daint_piz_dora/index.html), we managed to compile specfem with asdf, but when running the model with my 100000 receivers the writing stalled. I could watch the size of the synthetics.h5 go up megabyte by megabyte. After two hours I cancelled the job (of which the actual calculation takes a couple of minutes at most). I append the compilation file, Parameters and solver output. I had it write after 1000 time steps, that's why the output stops there. Maybe you can look into this, it would be great if you have an idea what's causing the trouble :) cheers
mk_daint.txt
output_solver.asdf.txt
Par_file.asdf.txt
The text was updated successfully, but these errors were encountered: