Data download for individual tools #488

hover2pi · 2021-04-14T05:09:09Z

Resolves #386 by allowing ExoCTK data to be downloaded as needed for each tool.

…running each tool

pep8speaks · 2021-04-14T05:09:14Z

Hello @hover2pi, Thank you for updating !

In the file exoctk/contam_visibility/field_simulator.py:

Line 68:1: W293 blank line contains whitespace
Line 152:24: W503 line break before binary operator
Line 154:24: W503 line break before binary operator
Line 159:75: W504 line break after binary operator
Line 169:37: W504 line break after binary operator
Line 257:5: E265 block comment should start with '# '
Line 276:5: E265 block comment should start with '# '
Line 281:30: E226 missing whitespace around arithmetic operator
Line 300:5: E265 block comment should start with '# '
Line 335:5: E265 block comment should start with '# '
Line 344:22: E226 missing whitespace around arithmetic operator
Line 346:22: E226 missing whitespace around arithmetic operator
Line 378:5: E265 block comment should start with '# '
Line 389:5: E265 block comment should start with '# '
Line 409:39: E502 the backslash is redundant between brackets
Line 488:5: E265 block comment should start with '# '
Line 511:5: E265 block comment should start with '# '
Line 530:5: E265 block comment should start with '# '
Line 535:30: E226 missing whitespace around arithmetic operator
Line 554:5: E265 block comment should start with '# '
Line 589:5: E265 block comment should start with '# '
Line 598:22: E226 missing whitespace around arithmetic operator
Line 600:22: E226 missing whitespace around arithmetic operator
Line 631:5: E265 block comment should start with '# '
Line 642:5: E265 block comment should start with '# '
Line 658:39: E502 the backslash is redundant between brackets
Line 772:5: E265 block comment should start with '# '

In the file exoctk/forward_models/forward_models.py:

Line 20:1: E266 too many leading '#' for block comment
Line 39:1: E266 too many leading '#' for block comment
Line 41:1: E302 expected 2 blank lines, found 1
Line 77:41: W504 line break after binary operator
Line 78:17: E128 continuation line under-indented for visual indent
Line 80:5: E722 do not use bare 'except'
Line 117:79: W504 line break after binary operator
Line 118:71: W504 line break after binary operator
Line 280:44: E241 multiple spaces after ','
Line 313:57: E231 missing whitespace after ','
Line 327:9: E303 too many blank lines (2)
Line 328:33: E261 at least two spaces before inline comment
Line 329:39: E261 at least two spaces before inline comment
Line 329:40: E262 inline comment should start with '# '
Line 331:28: E261 at least two spaces before inline comment
Line 332:31: E261 at least two spaces before inline comment
Line 342:67: E226 missing whitespace around arithmetic operator
Line 351:64: E226 missing whitespace around arithmetic operator
Line 352:97: E502 the backslash is redundant between brackets
Line 354:35: E127 continuation line over-indented for visual indent
Line 359:32: E127 continuation line over-indented for visual indent
Line 359:48: E226 missing whitespace around arithmetic operator
Line 359:65: E502 the backslash is redundant between brackets
Line 360:74: E502 the backslash is redundant between brackets
Line 368:50: E226 missing whitespace around arithmetic operator
Line 378:17: E128 continuation line under-indented for visual indent

In the file exoctk/modelgrid.py:

Line 292:54: W504 line break after binary operator
Line 293:54: W504 line break after binary operator
Line 294:54: W504 line break after binary operator
Line 295:54: W504 line break after binary operator
Line 296:52: W504 line break after binary operator
Line 302:62: W504 line break after binary operator
Line 303:62: W504 line break after binary operator
Line 310:61: W504 line break after binary operator
Line 311:61: W504 line break after binary operator
Line 564:57: W504 line break after binary operator
Line 565:57: W504 line break after binary operator
Line 566:57: W504 line break after binary operator
Line 567:57: W504 line break after binary operator
Line 568:55: W504 line break after binary operator
Line 588:63: W504 line break after binary operator
Line 590:63: W504 line break after binary operator
Line 592:61: W504 line break after binary operator

In the file exoctk/utils.py:

Line 50:5: E123 closing bracket does not match indentation of opening bracket's line

Comment last updated at 2021-04-27 19:17:24 UTC

hover2pi · 2021-04-14T05:09:57Z

There is exactly a 0% chance the tests pass on this one on my first try.

bourque

After initial review, the code changes look good to me. However, importing exoctk crashes upon testing (see comment on utils.py).

bourque · 2021-04-22T14:19:46Z

exoctk/contam_visibility/field_simulator.py

+# EXOCTK_DATA = os.environ.get('EXOCTK_DATA')
+# if not EXOCTK_DATA:
+#     print('WARNING: The $EXOCTK_DATA environment variable is not set. '
+#           'Contamination overlap will not work. Please set the '
+#           'value of this variable to point to the location of the exoctk_data '
+#           'download folder.  Users may retreive this folder by clicking the '
+#           '"ExoCTK Data Download" button on the ExoCTK website, or by using '
+#           'the exoctk.utils.download_exoctk_data() function.'
+#           )
+#     TRACES_PATH = None
+# else:
+#     TRACES_PATH = os.path.join(EXOCTK_DATA, 'exoctk_contam', 'traces')


Could we just remove this instead of leaving it in and commented out?

bourque · 2021-04-22T14:23:16Z

exoctk/utils.py

 ON_TRAVIS_OR_RTD = HOME_DIR == '/home/travis' or HOME_DIR == '/Users/travis' or HOME_DIR == '/home/docs'
 if not ON_TRAVIS_OR_RTD:
    if not EXOCTK_DATA:
        print(
            'WARNING: The $EXOCTK_DATA environment variable is not set.  Please set the '
            'value of this variable to point to the location of the exoctk_data '
-            'download folder.  Users may retreive this folder by clicking the '
+            'download folder.  Users may retrieve this folder by clicking the '


Nice catch 😆 . "I before E except after C"

bourque · 2021-04-22T14:34:32Z

exoctk/utils.py

@@ -27,21 +27,38 @@

 # Supported filters
 FILTERS = svo.filters()
-NON_JWST = [row['Band'] for row in FILTERS if row['Instrument'] not in ['NIRISS', 'NIRCam', 'NIRSpec', 'MIRI']]
+NON_JWST = [filt for filt in FILTERS if not filt.startswith('NIRISS') and not filt.startswith('NIRCam') and not filt.startswith('NIRSpec') and not filt.startswith('MIRI')]


@hover2pi This line appears to be crashing when trying to import exoctk:

>>> import exoctk pandeia not installed. Functionality limited. Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/Users/bourque/Desktop/repositories/exoctk/exoctk/__init__.py", line 10, in <module> from . import modelgrid File "/Users/bourque/Desktop/repositories/exoctk/exoctk/modelgrid.py", line 24, in <module> from . import utils File "/Users/bourque/Desktop/repositories/exoctk/exoctk/utils.py", line 30, in <module> NON_JWST = [filt for filt in FILTERS if not filt.startswith('NIRISS') and not filt.startswith('NIRCam') and not filt.startswith('NIRSpec') and not filt.startswith('MIRI')] File "/Users/bourque/Desktop/repositories/exoctk/exoctk/utils.py", line 30, in <listcomp> NON_JWST = [filt for filt in FILTERS if not filt.startswith('NIRISS') and not filt.startswith('NIRCam') and not filt.startswith('NIRSpec') and not filt.startswith('MIRI')] AttributeError: 'Row' object has no attribute 'startswith'

hover2pi · 2021-04-27T15:10:20Z

@bourque Ok, I realized I didn't update the env files. Thanks!

bourque · 2021-04-27T17:30:28Z

@hover2pi FYI, I pulled you most recent changes and I am still seeing the same error as before:

>>> import exoctk
pandeia not installed. Functionality limited.
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/bourque/Desktop/repositories/exoctk/exoctk/__init__.py", line 10, in <module>
    from . import modelgrid
  File "/Users/bourque/Desktop/repositories/exoctk/exoctk/modelgrid.py", line 24, in <module>
    from . import utils
  File "/Users/bourque/Desktop/repositories/exoctk/exoctk/utils.py", line 30, in <module>
    NON_JWST = [filt for filt in FILTERS if not filt.startswith('NIRISS') and not filt.startswith('NIRCam') and not filt.startswith('NIRSpec') and not filt.startswith('MIRI')]
  File "/Users/bourque/Desktop/repositories/exoctk/exoctk/utils.py", line 30, in <listcomp>
    NON_JWST = [filt for filt in FILTERS if not filt.startswith('NIRISS') and not filt.startswith('NIRCam') and not filt.startswith('NIRSpec') and not filt.startswith('MIRI')]
AttributeError: 'Row' object has no attribute 'startswith'

hover2pi · 2021-04-27T17:53:36Z

@bourque can you check if it's running svo_filters v0.4.1? That's the only thing that can be breaking it. Thanks!

bourque · 2021-04-27T18:35:59Z

@hover2pi Ah yes, good point. I updated svo_filters and that appears to solve that problem, but a new one arises:

>>> import exoctk
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/bourque/Desktop/repositories/exoctk/exoctk/__init__.py", line 10, in <module>
    from . import modelgrid
  File "/Users/bourque/Desktop/repositories/exoctk/exoctk/modelgrid.py", line 24, in <module>
    from . import utils
  File "/Users/bourque/Desktop/repositories/exoctk/exoctk/utils.py", line 19, in <module>
    from svo_filters import svo
  File "/Users/bourque/miniconda3/envs/exoctk-3.7/lib/python3.7/site-packages/svo_filters/__init__.py", line 1, in <module>
    from .svo import Filter
  File "/Users/bourque/miniconda3/envs/exoctk-3.7/lib/python3.7/site-packages/svo_filters/svo.py", line 17, in <module>
    from astroquery.svo_fps import SvoFps
ModuleNotFoundError: No module named 'astroquery.svo_fps'

Which version of astroquery are you using? The one that comes with the exoctk-3.7 environment doesn't appear to be working with this.

hover2pi · 2021-04-27T19:14:37Z

Oh yeah. It requires git+https://github.com/astropy/astroquery.git@ccc96185beeff86f3a12a31a00a801afcebe1dbe, which is astroquery v0.4.2.dev####. I'll add that to the env files.

hover2pi · 2021-04-27T19:17:43Z

Ok, that should do it @bourque . Sorry to make you look at this so many times.

bourque · 2021-04-27T19:41:47Z

@hover2pi No worries! I pulled in your changes and did some spot checking with some of the data packages. Its working like a charm! I think this is good to merge now.

Data download for individual tools

Added support for partial downloading of exoctk data and checks when …

811c130

…running each tool

hover2pi requested a review from bourque April 15, 2021 03:15

bourque assigned hover2pi Apr 21, 2021

bourque added 1: HIGH PRIORITY Category: Enhancement Tool: all labels Apr 21, 2021

bourque requested changes Apr 22, 2021

View reviewed changes

Bumped svo_filters version and removed commened code

d6cc589

hover2pi added 2 commits April 27, 2021 15:16

Updated env files for correct astroquery version

d7082e2

Updated pip requirements

c854a6a

bourque self-requested a review April 27, 2021 19:40

bourque approved these changes Apr 27, 2021

View reviewed changes

hover2pi merged commit d5fb4bf into develop Apr 27, 2021

hover2pi deleted the data branch April 27, 2021 20:00

bourque mentioned this pull request Apr 27, 2021

Check for existence of appropriate EXOCTK_DATA directories when necessary #386

Closed

hover2pi added a commit that referenced this pull request Dec 2, 2022

Merge pull request #488 from ExoCTK/data

e86f3fa

Data download for individual tools

hover2pi added a commit that referenced this pull request Jul 16, 2024

Merge pull request #488 from ExoCTK/data

5512441

Data download for individual tools

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Data download for individual tools #488

Data download for individual tools #488

hover2pi commented Apr 14, 2021

pep8speaks commented Apr 14, 2021 •

edited

Loading

hover2pi commented Apr 14, 2021

bourque left a comment

bourque Apr 22, 2021

bourque Apr 22, 2021

bourque Apr 22, 2021

hover2pi commented Apr 27, 2021

bourque commented Apr 27, 2021

hover2pi commented Apr 27, 2021

bourque commented Apr 27, 2021

hover2pi commented Apr 27, 2021

hover2pi commented Apr 27, 2021

bourque commented Apr 27, 2021

Data download for individual tools #488

Data download for individual tools #488

Conversation

hover2pi commented Apr 14, 2021

pep8speaks commented Apr 14, 2021 • edited Loading

Comment last updated at 2021-04-27 19:17:24 UTC

hover2pi commented Apr 14, 2021

bourque left a comment

Choose a reason for hiding this comment

bourque Apr 22, 2021

Choose a reason for hiding this comment

bourque Apr 22, 2021

Choose a reason for hiding this comment

bourque Apr 22, 2021

Choose a reason for hiding this comment

hover2pi commented Apr 27, 2021

bourque commented Apr 27, 2021

hover2pi commented Apr 27, 2021

bourque commented Apr 27, 2021

hover2pi commented Apr 27, 2021

hover2pi commented Apr 27, 2021

bourque commented Apr 27, 2021

pep8speaks commented Apr 14, 2021 •

edited

Loading