Problems downloading surrogates with fresh install

[moble]

I installed from conda-forge and tried to run a few lines from @vijayvarma392, but got a couple errors when it tried to download the surrogate data. The first error was because the output directory didn't exist. So I suggest following this line

gwsurrogate/gwsurrogate/catalog.py

Line 233 in d8bb961

sdir = os.path.abspath(sdir)

with

  if not os.path.isdir(sdir):
    os.makedirs(sdir)

The second error was because, although I have wget installed on my machine, the conda environment the code was running in doesn't. But because wget was called with os.system, when the shell errored out, nothing happened, and gwsurrogates continued as if it had succeeded — which made this bug pretty hard to track down. Python now suggests using something from the subprocess module instead of os.system, so I suggest replacing this line

gwsurrogate/gwsurrogate/catalog.py

Line 252 in d8bb961

os.system('wget -q --directory-prefix='+sdir+' '+surr_url)

with

    print(subprocess.check_output('wget -q --directory-prefix='+sdir+' '+surr_url, shell=True, stderr=subprocess.STDOUT))

(along with import subprocess somewhere up above). This should error out if wget fails, and it will show what happened.

[duetosymmetry]

Maybe we should move away from relying on wget and use the built in urllib.urlretrieve (in py2) or urllib.request.urlretrieve (in py3) instead?

[moble]

The docs describe urllib.request.urlretrieve as a legacy interface that may be deprecated in the future; urlopen is presumably preferred. But all of these urllib* functions are pretty finicky — not dealing nicely with encodings, not detecting subtle errors, etc. If you want to stay in python, requests will be more comparable to wget.

[sfield17]

Proposed fix:

• this will be fixed when we address issue 20