Only check if results are cloud hosted when there are results #839

jhkennedy · 2024-10-03T02:19:55Z

Currently, if you search for a collection that doesn't exist because you have the wrong name for it, earthaccess will bonk pretty hard with an error message that doesn't describe the problem:

>>> import earthaccess
>>> earthaccess.login()
>>> results = earthaccess.search_data(
...     short_name='OPERA_L3_DSWX-HLS_V1_1.0',
...     bounding_box=(-95.19, 30.59, -94.99, 30.79),
...     temporal=('2024-04-30', '2024-05-31'),
... )
Traceback (most recent call last):
  File ".../mambaforge/envs/earthaccess/lib/python3.12/site-packages/IPython/core/interactiveshell.py", line 3577, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-2-7ebcf77bff69>", line 5, in <module>
    results = earthaccess.search_data(
              ^^^^^^^^^^^^^^^^^^^^^^^^
  File ".../Documents/Code/contributing/earthaccess/earthaccess/api.py", line 131, in search_data
    return query.get_all()
           ^^^^^^^^^^^^^^^
  File ".../mambaforge/envs/earthaccess/lib/python3.12/site-packages/cmr/queries.py", line 127, in get_all
    return self.get(self.hits())
           ^^^^^^^^^^^^^^^^^^^^^
  File ".../earthaccess/earthaccess/search.py", line 472, in get
    cloud = self._is_cloud_hosted(response[0])
                                  ~~~~~~~~^^^
IndexError: list index out of range

I would expect this instead:

>>> import earthaccess
>>> earthaccess.login()
>>> results = earthaccess.search_data(
...     short_name='OPERA_L3_DSWX-HLS_V1_1.0',
...     bounding_box=(-95.19, 30.59, -94.99, 30.79),
...     temporal=('2024-04-30', '2024-05-31'),
... )
>>> print(len(results))
0

This PR:

only checks if the data is cloud-hosted where there are more than no results returned, allowing earthaccess to return an empty list.
adds a header to the changelog since I can't ever remember which changelog styling we use off the top of my head ("Not Keep a Changelog... the other one...")
Add forgotten links for the v0.11.0 release to the Changelog

Background:

I was helping a user on Earthdata Forum troubleshoot why their STAC CMR queries weren't working and gave them a version of their script using earthaccess:
https://forum.earthdata.nasa.gov/viewtopic.php?t=5953&sid=578d74d50b040bc37ad8aa188610a90a

When I initially tried to convert the script, I had the short name wrong and encountered the error message above, which I found rather incomprehensible.

I can see having the wrong short name and getting no results as a common situation users could encounter.

As an aside, I had initially naively used the CMR collection name as the short name. What I didn't realize is that In CMR, short_name is not required to be unique, but short_name + version is, so CMR STAC uses short_name + version for the collection ID to provide a unique, human-readable collection ID.

This is the search I was trying to perform:

>>> import earthaccess
>>> earthaccess.login()
>>> results = earthaccess.search_data(
...     short_name='OPERA_L3_DSWX-HLS_V1',
...     bounding_box=(-95.19, 30.59, -94.99, 30.79),
...     temporal=('2024-04-30', '2024-05-31'),
... )
>>> print(len(results))
58

Pull Request (PR) draft checklist - click to expand

Please review our
contributing documentation
before getting started.
~~Ensure an issue exists representing the problem being solved in this PR.~~
Populate a descriptive title. For example, instead of "Updated README.md", use a
title such as "Add testing details to the contributor section of the README".
Example PRs: #763
Populate the body of the pull request with:
- A clear description of the change you are proposing.
- Links to any issues resolved by this PR with text in the PR description, for
  example, closes #1. See
  GitHub docs - Linking a pull request to an issue.
Update CHANGELOG.md with details about your change in a section titled
## Unreleased. If such a section does not exist, please create one. Follow
Common Changelog for your additions.
Example PRs: #763
Update the documentation and/or the README.md with details of changes to the
earthaccess interface, if any. Consider new environment variables, function names,
decorators, etc.

Click the "Ready for review" button at the bottom of the "Conversation" tab in GitHub
once these requirements are fulfilled. Don't worry if you see any test failures in
GitHub at this point!

Pull Request (PR) merge checklist - click to expand

Please do your best to complete these requirements! If you need help with any of these
requirements, you can ping the @nsidc/earthaccess-support team in a comment and we
will help you out!

Add unit tests for any new features.
Apply formatting and linting autofixes. You can add a GitHub comment in this Pull
Request containing "pre-commit.ci autofix" to automate this.
Ensure all automated PR checks (seen at the bottom of the "conversation" tab) pass.
Get at least one approving review.

📚 Documentation preview 📚: https://earthaccess--839.org.readthedocs.build/en/839/

codecov-commenter · 2024-10-03T02:21:00Z

⚠️ Please install the to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 72.35%. Comparing base (2f49c08) to head (34f4102).
Report is 62 commits behind head on main.

❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #839      +/-   ##
==========================================
- Coverage   73.88%   72.35%   -1.54%     
==========================================
  Files          31       31              
  Lines        2003     1993      -10     
==========================================
- Hits         1480     1442      -38     
- Misses        523      551      +28

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

chuckwondo

This fixes #526, so I've suggested an update to the changelog, plus always assigning a value to cloud.

CHANGELOG.md

earthaccess/search.py

chuckwondo · 2024-10-04T03:17:43Z

@jhkennedy, how about a unit test as well?

jhkennedy · 2024-10-04T04:08:40Z

@chuckwondo yes, that would be a good idea, wouldn't it! 🤦🤣

It'll take me a little but I'll circle back around to that

jhkennedy · 2024-10-04T13:30:41Z

This fixes #526

🤔 now, how did I miss that issue?! From the conversation there, there's also a refactor we should do.

I prefer to merge this to get the quick fix out the door and circle back to the refactor later. I think that means closing #526 and opening a new issue for it, but leaving #526 open would also work.

earthaccess/search.py

mfisher87 · 2024-10-04T13:57:57Z

CHANGELOG.md

+All notable changes to this project will be documented in this file.
+
+The format is based on [Common Changelog](https://common-changelog.org/)
+and this project uses [Semantic Versioning](https://semver.org/spec/v2.0.0.html).


💯 💯 💯

mfisher87 · 2024-10-04T13:58:40Z

I prefer to merge this to get the quick fix out the door and circle back to the refactor later. I think that means closing #526 and opening a new issue for it

💯

chuckwondo · 2024-10-06T13:41:40Z

This fixes #526

🤔 now, how did I miss that issue?! From the conversation there, there's also a refactor we should do.

I prefer to merge this to get the quick fix out the door and circle back to the refactor later. I think that means closing #526 and opening a new issue for it, but leaving #526 open would also work.

Good point. I forgot about the refactoring bit. I think closing #526 with this PR makes sense, and opening a new issue for the refactoring bit would be my preference (and @mfisher87's preference too, if I'm interpreting his previous comment correctly).

chuckwondo

LGTM! Thanks for adding the ref to Common Changelog that was missing.

chuckwondo

Oops! I was too hasty in approving. How about a unit test?

jhkennedy · 2024-10-07T17:23:33Z

Yep, unit test is on my todo list, but I likely won't get to it this week. Any maintainer should feel free to push a test to this branch if they want. Otherwise, I'll get to it next week.

github-actions · 2024-10-16T23:06:38Z

👈 Launch a binder notebook on this branch for commit 8389aa0

I will automatically update this comment whenever this PR is modified

👈 Launch a binder notebook on this branch for commit 34f4102

Co-authored-by: Chuck Daniels <cjdaniels4@gmail.com>

jhkennedy · 2024-10-16T23:23:54Z

Alright, I added a unit test and confirmed the test fails on the main branch but works on this branch.

jhkennedy · 2024-10-16T23:47:22Z

I also opened the follow-on refactor issue here: #844

@chuckwondo @mfisher87 @betolink feel, free to edit/expand/etc.

chuckwondo

Thanks Joe!

CHANGELOG.md

github-actions · 2024-10-28T14:20:44Z

User Sherwin-14 does not have permission to run integration tests. A maintainer must perform a security review of the code changes in this pull request and re-run the failed integration tests jobs, if the code is deemed safe.

github-actions · 2024-10-28T17:22:41Z

User Sherwin-14 does not have permission to run integration tests. A maintainer must perform a security review of the code changes in this pull request and re-run the failed integration tests jobs, if the code is deemed safe.

github-actions · 2024-10-28T17:23:35Z

User Sherwin-14 does not have permission to run integration tests. A maintainer must perform a security review of the code changes in this pull request and re-run the failed integration tests jobs, if the code is deemed safe.

github-actions · 2024-10-28T17:25:18Z

User Sherwin-14 does not have permission to run integration tests. A maintainer must perform a security review of the code changes in this pull request and re-run the failed integration tests jobs, if the code is deemed safe.

github-actions · 2024-10-29T14:57:27Z

User Sherwin-14 does not have permission to run integration tests. A maintainer must perform a security review of the code changes in this pull request and re-run the failed integration tests jobs, if the code is deemed safe.

github-actions · 2024-10-31T18:57:42Z

User ebolch does not have permission to run integration tests. A maintainer must perform a security review of the code changes in this pull request and re-run the failed integration tests jobs, if the code is deemed safe.

github-actions · 2024-11-04T06:49:28Z

User Sherwin-14 does not have permission to run integration tests. A maintainer must perform a security review of the code changes in this pull request and re-run the failed integration tests jobs, if the code is deemed safe.

jhkennedy requested review from chuckwondo, betolink and mfisher87 October 3, 2024 02:19

chuckwondo requested changes Oct 4, 2024

View reviewed changes

CHANGELOG.md Outdated Show resolved Hide resolved

earthaccess/search.py Outdated Show resolved Hide resolved

jhkennedy commented Oct 4, 2024

View reviewed changes

earthaccess/search.py Outdated Show resolved Hide resolved

mfisher87 reviewed Oct 4, 2024

View reviewed changes

chuckwondo approved these changes Oct 6, 2024

View reviewed changes

chuckwondo requested changes Oct 6, 2024

View reviewed changes

jhkennedy linked an issue Oct 7, 2024 that may be closed by this pull request

DataGranules.get raises IndexError when Granules found = zero #526

Closed

jhkennedy and others added 6 commits October 16, 2024 15:07

Only check if results are cloud hosted when there are results

240bdc9

Update Changelog

c494978

typo

d5c0346

Apply suggestions from code review

544b4d0

Co-authored-by: Chuck Daniels <cjdaniels4@gmail.com>

Update earthaccess/search.py

b8c470c

Add unit test for no results

34f4102

jhkennedy force-pushed the dont-bonk branch from 8389aa0 to 34f4102 Compare October 16, 2024 23:14

jhkennedy requested review from mfisher87 and chuckwondo October 16, 2024 23:17

jhkennedy mentioned this pull request Oct 16, 2024

Refactor how we determine if granules are cloud-hosted #844

Open

3 tasks

chuckwondo approved these changes Oct 17, 2024

View reviewed changes

CHANGELOG.md Show resolved Hide resolved

jhkennedy merged commit 9784e4c into nsidc:main Oct 17, 2024
13 checks passed

jhkennedy deleted the dont-bonk branch October 17, 2024 16:45

mfisher87 mentioned this pull request Nov 4, 2024

[BUG] Permission check workflow always posts to the same PR #863

Open

1 task

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Only check if results are cloud hosted when there are results #839

Only check if results are cloud hosted when there are results #839

jhkennedy commented Oct 3, 2024 •

edited

Loading

codecov-commenter commented Oct 3, 2024 •

edited

Loading

chuckwondo left a comment

chuckwondo commented Oct 4, 2024

jhkennedy commented Oct 4, 2024

jhkennedy commented Oct 4, 2024

mfisher87 Oct 4, 2024

mfisher87 commented Oct 4, 2024

chuckwondo commented Oct 6, 2024

chuckwondo left a comment

chuckwondo left a comment

jhkennedy commented Oct 7, 2024

github-actions bot commented Oct 16, 2024 •

edited

Loading

jhkennedy commented Oct 16, 2024

jhkennedy commented Oct 16, 2024 •

edited by chuckwondo

Loading

chuckwondo left a comment

github-actions bot commented Oct 28, 2024

github-actions bot commented Oct 28, 2024

github-actions bot commented Oct 28, 2024

github-actions bot commented Oct 28, 2024

github-actions bot commented Oct 29, 2024

github-actions bot commented Oct 31, 2024

github-actions bot commented Nov 4, 2024

Only check if results are cloud hosted when there are results #839

Only check if results are cloud hosted when there are results #839

Conversation

jhkennedy commented Oct 3, 2024 • edited Loading

This PR:

Background:

codecov-commenter commented Oct 3, 2024 • edited Loading

Codecov Report

chuckwondo left a comment

Choose a reason for hiding this comment

chuckwondo commented Oct 4, 2024

jhkennedy commented Oct 4, 2024

jhkennedy commented Oct 4, 2024

mfisher87 Oct 4, 2024

Choose a reason for hiding this comment

mfisher87 commented Oct 4, 2024

chuckwondo commented Oct 6, 2024

chuckwondo left a comment

Choose a reason for hiding this comment

chuckwondo left a comment

Choose a reason for hiding this comment

jhkennedy commented Oct 7, 2024

github-actions bot commented Oct 16, 2024 • edited Loading

jhkennedy commented Oct 16, 2024

jhkennedy commented Oct 16, 2024 • edited by chuckwondo Loading

chuckwondo left a comment

Choose a reason for hiding this comment

github-actions bot commented Oct 28, 2024

github-actions bot commented Oct 28, 2024

github-actions bot commented Oct 28, 2024

github-actions bot commented Oct 28, 2024

github-actions bot commented Oct 29, 2024

github-actions bot commented Oct 31, 2024

github-actions bot commented Nov 4, 2024

jhkennedy commented Oct 3, 2024 •

edited

Loading

codecov-commenter commented Oct 3, 2024 •

edited

Loading

github-actions bot commented Oct 16, 2024 •

edited

Loading

jhkennedy commented Oct 16, 2024 •

edited by chuckwondo

Loading