Skip to content
This repository was archived by the owner on Jul 11, 2023. It is now read-only.

Commit 6bec580

Browse files
emthompson-usgsemthompson-usgs
authored
Merge pull request #560 from emthompson-usgs/ci
azure
2 parents 9630996 + 49be093 commit 6bec580

25 files changed

+394
-316
lines changed

‎.gitignore

+2
Original file line numberDiff line numberDiff line change
@@ -17,3 +17,5 @@ docs/Gemfile.lock
1717
docs/figs/*.aux
1818
docs/figs/*.log
1919
**_version.py
20+
.vscode/
21+
dask-worker-space/

‎.travis.yml

-25
This file was deleted.

‎README.rst

+16-22
Original file line numberDiff line numberDiff line change
@@ -20,52 +20,46 @@ Build info
2020
+ +------------------+-----------------+------------+
2121
| | |AzureM1015P38| | OSX 10.15 | Python 3.8 |
2222
+ +------------------+-----------------+------------+
23-
| | |AzureM1014P37| | OSX 10.14 | Python 3.7 |
23+
| | |AzureWP37| | Windows-latest | Python 3.7 |
2424
+ +------------------+-----------------+------------+
25-
| | |AzureM1014P38| | OSX 10.14 | Python 3.8 |
25+
| | |AzureWP38| | Windows-latest | Python 3.8 |
2626
+ +------------------+-----------------+------------+
27-
| | |AzureLP37| | ubuntu | Python 3.7 |
27+
| | |AzureLP37| | ubuntu-latest | Python 3.7 |
2828
+ +------------------+-----------------+------------+
29-
| | |AzureLP38| | ubuntu | Python 3.8 |
30-
+---------+------------------+-----------------+------------+
31-
| Travis | |Travis| | ubuntu | Python 3.7 |
29+
| | |AzureLP38| | ubuntu-latest | Python 3.8 |
3230
+---------+------------------+-----------------+------------+
3331
| Codacy | |Codacy| |
3432
+---------+-------------------------------------------------+
3533
| CodeCov | |CodeCov| |
3634
+---------+-------------------------------------------------+
3735

38-
.. |Travis| image:: https://travis-ci.com/usgs/groundmotion-processing.svg?branch=master
39-
:target: https://travis-ci.org/usgs/groundmotion-processing
40-
:alt: Travis Build Status
41-
4236
.. |Codacy| image:: https://api.codacy.com/project/badge/Grade/582cbceabb814eca9f708e37d6af9479
4337
:target: https://www.codacy.com/app/mhearne-usgs/groundmotion-processing?utm_source=github.com&utm_medium=referral&utm_content=usgs/groundmotion-processing&utm_campaign=Badge_Grade
4438

4539
.. |CodeCov| image:: https://codecov.io/gh/usgs/groundmotion-processing/branch/master/graph/badge.svg
4640
:target: https://codecov.io/gh/usgs/groundmotion-processing
4741
:alt: Code Coverage Status
4842

49-
.. |AzureM1015P37| image:: https://dev.azure.com/GHSC-ESI/USGS-groundmotion-processing/_apis/build/status/usgs.groundmotion-processing?branchName=master&jobName=gmprocess&configuration=gmprocess%20MacOS_10_15_Python37
43+
.. |AzureM1015P37| image:: https://dev.azure.com/GHSC-ESI/USGS-groundmotion-processing/_apis/build/status/usgs.groundmotion-processing?branchName=master&jobName=gmprocess&configuration=gmprocess%20MacOS_py37
5044
:target: https://dev.azure.com/GHSC-ESI/USGS-groundmotion-processing/_build/latest?definitionId=5&branchName=master
51-
:alt: Azure DevOps Build Status
45+
:alt: Build Status: Mac 10.15, python 3.7
5246

53-
.. |AzureM1015P38| image:: https://dev.azure.com/GHSC-ESI/USGS-groundmotion-processing/_apis/build/status/usgs.groundmotion-processing?branchName=master&jobName=gmprocess&configuration=gmprocess%20MacOS_10_15_Python38
47+
.. |AzureM1015P38| image:: https://dev.azure.com/GHSC-ESI/USGS-groundmotion-processing/_apis/build/status/usgs.groundmotion-processing?branchName=master&jobName=gmprocess&configuration=gmprocess%20MacOS_py38
5448
:target: https://dev.azure.com/GHSC-ESI/USGS-groundmotion-processing/_build/latest?definitionId=5&branchName=master
55-
:alt: Azure DevOps Build Status
49+
:alt: Build Status: Mac 10.15, python 3.8
5650

57-
.. |AzureM1014P37| image:: https://dev.azure.com/GHSC-ESI/USGS-groundmotion-processing/_apis/build/status/usgs.groundmotion-processing?branchName=master&jobName=gmprocess&configuration=gmprocess%20MacOS_10_14_Python37
51+
.. |AzureWP37| image:: https://dev.azure.com/GHSC-ESI/USGS-groundmotion-processing/_apis/build/status/usgs.groundmotion-processing?branchName=master&jobName=gmprocess&configuration=gmprocess%20Windows_py37
5852
:target: https://dev.azure.com/GHSC-ESI/USGS-groundmotion-processing/_build/latest?definitionId=5&branchName=master
59-
:alt: Azure DevOps Build Status
53+
:alt: Build Status: windows-latest, python 3.7
6054

61-
.. |AzureM1014P38| image:: https://dev.azure.com/GHSC-ESI/USGS-groundmotion-processing/_apis/build/status/usgs.groundmotion-processing?branchName=master&jobName=gmprocess&configuration=gmprocess%20MacOS_10_14_Python38
55+
.. |AzureWP38| image:: https://dev.azure.com/GHSC-ESI/USGS-groundmotion-processing/_apis/build/status/usgs.groundmotion-processing?branchName=master&jobName=gmprocess&configuration=gmprocess%20Windows_py38
6256
:target: https://dev.azure.com/GHSC-ESI/USGS-groundmotion-processing/_build/latest?definitionId=5&branchName=master
63-
:alt: Azure DevOps Build Status
57+
:alt: Build Status: windows-latest, python 3.8
6458

65-
.. |AzureLP37| image:: https://dev.azure.com/GHSC-ESI/USGS-groundmotion-processing/_apis/build/status/usgs.groundmotion-processing?branchName=master&jobName=gmprocess&configuration=gmprocess%20Linux_Python37
59+
.. |AzureLP37| image:: https://dev.azure.com/GHSC-ESI/USGS-groundmotion-processing/_apis/build/status/usgs.groundmotion-processing?branchName=master&jobName=gmprocess&configuration=gmprocess%20Linux_py37
6660
:target: https://dev.azure.com/GHSC-ESI/USGS-groundmotion-processing/_build/latest?definitionId=5&branchName=master
67-
:alt: Azure DevOps Build Status
61+
:alt: Build Status: ubuntu-latest, python 3.7
6862

69-
.. |AzureLP38| image:: https://dev.azure.com/GHSC-ESI/USGS-groundmotion-processing/_apis/build/status/usgs.groundmotion-processing?branchName=master&jobName=gmprocess&configuration=gmprocess%20Linux_Python38
63+
.. |AzureLP38| image:: https://dev.azure.com/GHSC-ESI/USGS-groundmotion-processing/_apis/build/status/usgs.groundmotion-processing?branchName=master&jobName=gmprocess&configuration=gmprocess%20Linux_py38
7064
:target: https://dev.azure.com/GHSC-ESI/USGS-groundmotion-processing/_build/latest?definitionId=5&branchName=master
71-
:alt: Azure DevOps Build Status
65+
:alt: Build Status: ubuntu-latest, python 3.8

‎azure-pipelines.yml

+36-11
Original file line numberDiff line numberDiff line change
@@ -16,23 +16,23 @@ jobs:
1616
timeoutInMinutes: 120
1717
strategy:
1818
matrix:
19-
Linux_Python37:
19+
Linux_py37:
2020
imageName: 'ubuntu-latest'
2121
python.version: '3.7'
22-
Linux_Python38:
22+
Linux_py38:
2323
imageName: 'ubuntu-latest'
2424
python.version: '3.8'
25-
MacOS_10_15_Python37:
25+
MacOS_py37:
2626
imageName: 'macOS-10.15'
2727
python.version: '3.7'
28-
MacOS_10_15_Python38:
28+
MacOS_py38:
2929
imageName: 'macOS-10.15'
3030
python.version: '3.8'
31-
MacOS_10_14_Python37:
32-
imageName: 'macOS-10.14'
31+
Windows_py37:
32+
imageName: 'windows-latest'
3333
python.version: '3.7'
34-
MacOS_10_14_Python38:
35-
imageName: 'macOS-10.14'
34+
Windows_py38:
35+
imageName: 'windows-latest'
3636
python.version: '3.8'
3737

3838
pool:
@@ -48,6 +48,11 @@ jobs:
4848

4949
- bash: echo "##vso[task.prependpath]$CONDA/bin"
5050
displayName: Add conda to path
51+
condition: ne( variables['Agent.OS'], 'Windows_NT' )
52+
53+
- powershell: Write-Host "##vso[task.prependpath]$env:CONDA\Scripts"
54+
displayName: Add conda to PATH
55+
condition: eq( variables['Agent.OS'], 'Windows_NT' )
5156

5257
- bash: |
5358
if [[ $(Agent.OS) = 'Darwin' ]]
@@ -65,23 +70,43 @@ jobs:
6570
echo $(Agent.NAME)
6671
echo $(python.version)
6772
bash install.sh -p $(python.version)
68-
displayName: Create environment
73+
displayName: Create conda environment and install gmprocess (mac/linux)
74+
condition: ne( variables['Agent.OS'], 'Windows_NT' )
75+
76+
- script: conda create --yes --name gmprocess --file requirements.txt --strict-channel-priority -c conda-forge -v
77+
displayName: Create conda environment (Windows)
78+
condition: eq( variables['Agent.OS'], 'Windows_NT' )
79+
80+
- script: |
81+
call activate gmprocess
82+
pip install -e . --no-deps --force-reinstall -vv
83+
displayName: Install gmprocess packages (Windows)
84+
condition: eq( variables['Agent.OS'], 'Windows_NT' )
6985
7086
- bash: conda init bash
7187
displayName: Init conda for bash
88+
condition: ne( variables['Agent.OS'], 'Windows_NT' )
7289

7390
- bash: |
7491
source activate gmprocess
7592
export PYTHONPATH="."
7693
py.test --cov=. --cov-report=xml
7794
failOnStderr: true
78-
displayName: Run tests
95+
displayName: Run tests (mac/linux)
7996
name: RunTests
97+
condition: ne( variables['Agent.OS'], 'Windows_NT' )
8098
99+
- script: |
100+
call activate gmprocess
101+
pytest -s -rxs -vv
102+
displayName: Run tests (Windows)
103+
condition: eq( variables['Agent.OS'], 'Windows_NT' )
104+
81105
- bash: |
82106
pip install codecov codacy-coverage
83107
codecov
84108
coverage xml
85109
python-codaccy-coverage -r coverage.xml
86110
bash <(curl -s https://codecov.io/bash)
87-
displayName: Get coverage
111+
displayName: Get coverage (Linux)
112+
condition: eq( variables['Agent.OS'], 'Linux' )

‎tests/conftest.py ‎conftest.py

File renamed without changes.

‎gmprocess/_version.py

+1-1
Original file line numberDiff line numberDiff line change
@@ -1 +1 @@
1-
__version__ = "1.1.4.dev3+g8651539.d20201218"
1+
__version__ = "1.1.7.dev10+g75557a5.d20201220"

‎gmprocess/bin/gmprocess.py

+49-39
Original file line numberDiff line numberDiff line change
@@ -14,7 +14,7 @@
1414
# third party imports
1515
import pandas as pd
1616
from h5py.h5py_warnings import H5pyDeprecationWarning
17-
import numpy as np
17+
from dask.distributed import Client, as_completed
1818

1919
# local imports
2020
from gmprocess.utils.args import add_shared_args
@@ -75,10 +75,9 @@ def append_file(files_created, tag, filename):
7575
files_created[tag] = [filename]
7676

7777

78-
def process_event(outdir, event, pcommands,
78+
def process_event(event, outdir, pcommands,
7979
config, input_directory,
80-
process_tag, logfile,
81-
files_created, output_format,
80+
process_tag, files_created, output_format,
8281
status, recompute_metrics, export_dir=None):
8382

8483
# setup logging to write to the input logfile
@@ -88,6 +87,7 @@ def process_event(outdir, event, pcommands,
8887

8988
logger = logging.getLogger()
9089
stream_handler = logger.handlers[0]
90+
logfile = os.path.join(outdir, '%s.log' % event.id)
9191
fhandler = logging.FileHandler(logfile)
9292
logger.removeHandler(stream_handler)
9393
logger.addHandler(fhandler)
@@ -305,10 +305,10 @@ def process_event(outdir, event, pcommands,
305305
logging.info(
306306
'Creating provenance table for event %s...' % event.id)
307307
with warnings.catch_warnings():
308-
warnings.simplefilter("ignore",
309-
category=H5pyDeprecationWarning)
310-
provdata = workspace.getProvenance(event.id,
311-
labels=[process_tag])
308+
warnings.simplefilter(
309+
"ignore", category=H5pyDeprecationWarning)
310+
provdata = workspace.getProvenance(
311+
event.id, labels=[process_tag])
312312
if output_format == 'csv':
313313
csvfile = os.path.join(event_dir, 'provenance.csv')
314314
append_file(files_created, 'Provenance', csvfile)
@@ -583,45 +583,55 @@ def main():
583583
if len(process_commands.intersection(set(pcommands))) > 0:
584584
if args.num_processes:
585585
# parallelize processing on events using forked processes
586-
eventids = [event.id for event in events]
587-
eventdict = dict(zip(eventids, events))
588-
chunks = np.array_split(eventids, args.num_processes)
589-
for i in range(0, len(chunks)):
590-
try:
591-
pid = os.fork()
592-
except OSError:
593-
sys.stderr.write("Could not create a child process\n")
594-
continue
595-
596-
if pid == 0:
597-
chunk = chunks[i]
598-
logfile = os.path.join(outdir, logfmt % os.getpid())
599-
for eventid in chunk:
600-
event = eventdict[eventid]
601-
workname = process_event(
602-
outdir, event, pcommands, config,
603-
input_directory, process_tag, logfile,
604-
files_created, args.format, args.status,
605-
args.recompute_metrics,
606-
export_dir=args.export_dir)
607-
workspace_files.append(workname)
608-
os._exit(0)
609-
else:
610-
print("Parent: created child process %i." % pid)
611-
612-
for i in range(0, len(chunks)):
613-
child_id, _ = os.waitpid(0, 0)
614-
print('Child process %i has finished.' % child_id)
586+
try:
587+
client = Client(n_workers=args.num_processes)
588+
except OSError:
589+
sys.stderr.write("Could not create a dask client.\n")
590+
sys.exit(1)
591+
592+
# Need a dict holding all args that do not change across calls
593+
_argdict_ = {
594+
'outdir': outdir,
595+
'pcommands': pcommands,
596+
'config': config,
597+
'input_directory': input_directory,
598+
'process_tag': process_tag,
599+
'files_created': files_created,
600+
'output_format': args.format,
601+
'status': args.status,
602+
'recompute_metrics': args.recompute_metrics,
603+
'export_dir': args.export_dir
604+
}
605+
606+
def dask_process_event(event):
607+
"""
608+
Wrapper function for multiprocessing of process_event method.
609+
"""
610+
workname = process_event(event, **_argdict_)
611+
return event, workname
612+
613+
futures = client.map(dask_process_event, events)
614+
615+
for future, result in as_completed(futures, with_results=True):
616+
print(
617+
'Completed event: %s, %s' %
618+
(result[0].id, str(result[1]))
619+
)
620+
615621
else:
616622
logfile = os.path.join(outdir, logfmt % os.getpid())
617623
for event in events:
618624
workname = process_event(
619-
outdir, event, pcommands,
625+
event, outdir, pcommands,
620626
config, input_directory, process_tag,
621-
logfile, files_created, args.format, args.status,
627+
files_created, args.format, args.status,
622628
args.recompute_metrics,
623629
export_dir=args.export_dir)
624630
workspace_files.append(workname)
631+
print(
632+
'Completed event: %s, %s' %
633+
(event.id, str(workname))
634+
)
625635

626636
# logging
627637
logger = None

‎gmprocess/data/config_production.yml

+3-3
Original file line numberDiff line numberDiff line change
@@ -220,9 +220,9 @@ processing:
220220
detrending_method: demean
221221

222222
- compute_snr:
223-
# Presense of this check says to do the signa-to-noise ratio check. Requires
224-
# minimum SNR of `threshold` between `min_freq` and `max_freq` using
225-
# Konno-Omachi smoothed spectra with `bandwidth` parameter.
223+
# Presense of this check says to do the signa-to-noise ratio check.
224+
# Requires minimum SNR of `threshold` between `min_freq` and `max_freq`
225+
# using Konno-Omachi smoothed spectra with `bandwidth` parameter.
226226
bandwidth: 20.0
227227
check:
228228
threshold: 3.0

‎gmprocess/io/fetch_utils.py

+23-18
Original file line numberDiff line numberDiff line change
@@ -204,28 +204,33 @@ def draw_stations_map(pstreams, event, event_dir):
204204
ax.scatter(lons, lats, c=status, marker='^', edgecolors='k',
205205
transform=mmap.geoproj, zorder=100, s=48)
206206

207-
passed_marker = mlines.Line2D([], [], color=PASSED_COLOR, marker='^',
208-
markeredgecolor='k', markersize=12,
209-
label='Passed station', linestyle='None')
210-
failed_marker = mlines.Line2D([], [], color=FAILED_COLOR, marker='^',
211-
markeredgecolor='k', markersize=12,
212-
label='Failed station', linestyle='None')
213-
earthquake_marker = mlines.Line2D([], [], color='red', marker='*',
214-
markersize=12,
215-
label='Earthquake Epicenter',
216-
linestyle='None')
207+
passed_marker = mlines.Line2D(
208+
[], [], color=PASSED_COLOR, marker='^',
209+
markeredgecolor='k', markersize=12,
210+
label='Passed station', linestyle='None')
211+
failed_marker = mlines.Line2D(
212+
[], [], color=FAILED_COLOR, marker='^',
213+
markeredgecolor='k', markersize=12,
214+
label='Failed station', linestyle='None')
215+
earthquake_marker = mlines.Line2D(
216+
[], [], color='red', marker='*',
217+
markersize=12,
218+
label='Earthquake Epicenter',
219+
linestyle='None')
217220
ax.legend(handles=[passed_marker, failed_marker, earthquake_marker],
218221
fontsize=12)
219222

220223
scale = '50m'
221-
land = cfeature.NaturalEarthFeature(category='physical',
222-
name='land',
223-
scale=scale,
224-
facecolor=LAND_COLOR)
225-
ocean = cfeature.NaturalEarthFeature(category='physical',
226-
name='ocean',
227-
scale=scale,
228-
facecolor=OCEAN_COLOR)
224+
land = cfeature.NaturalEarthFeature(
225+
category='physical',
226+
name='land',
227+
scale=scale,
228+
facecolor=LAND_COLOR)
229+
ocean = cfeature.NaturalEarthFeature(
230+
category='physical',
231+
name='ocean',
232+
scale=scale,
233+
facecolor=OCEAN_COLOR)
229234
ax.add_feature(land)
230235
ax.add_feature(ocean)
231236
ax.coastlines(resolution=scale, zorder=10, linewidth=1)

0 commit comments

Comments
 (0)