Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

TST: Reorganizing tests and moving to pytest-style #103

Merged

Conversation

greglucas
Copy link
Collaborator

Change Summary

Overview

This is a major reorganization and refactor of the test suite to try and align more closely with pytest-style tests, using fixtures instead of setup/teardown methods.

The tests have been reorganized into folders that more closely align to names that are being tested as well.

NOTE: The infrastructure tests will still be failing. The lambda_endpoints test_indexer and test_queries rely on the actual server running, so I did not update those tests.

New Dependencies

N/A

New Files

N/A

Deleted Files

Several unused test data files

Updated Files

All test files other than the infrastructure portion (that will be handled in a separate PR).

  • Moved the tests into "opensearch" and "lambda_endpoints" to try and delineate what is being tested where a little more clearly.
  • Made use of fixtures instead of setup/teardown methods on the unittest classes
  • Changed test assertions from test_true to test_expected to more clearly explain that it is an expected value (some of these were set to False which confused me when looking at the assertions.

Testing

See above for updated files

@greglucas
Copy link
Collaborator Author

OK, I decided to touch the test_sds_data_manager in a minimal way for now just to get tests actually running on CI. I put a pytest.skip() within that file so that we ignore it for now, but I or someone else can remove that in a follow-up PR if we need to.

@codecov
Copy link

codecov bot commented Jul 21, 2023

Codecov Report

Patch coverage: 84.89% and project coverage change: -18.35% ⚠️

Comparison is base (34124f0) 86.49% compared to head (3a1b08e) 68.14%.

Additional details and impacted files
@@               Coverage Diff                @@
##           development     #103       +/-   ##
================================================
- Coverage        86.49%   68.14%   -18.35%     
================================================
  Files               21       23        +2     
  Lines              955      945       -10     
================================================
- Hits               826      644      -182     
- Misses             129      301      +172     
Files Changed Coverage Δ
tests/lambda_endpoints/test_indexer.py 32.14% <0.00%> (ø)
sds_data_manager/lambda_code/SDSCode/indexer.py 28.35% <14.28%> (-1.81%) ⬇️
sds_data_manager/lambda_code/SDSCode/queries.py 45.16% <20.00%> (-9.01%) ⬇️
tests/infrastructure/test_data_manager_stack.py 3.90% <30.00%> (-96.10%) ⬇️
sds_data_manager/stacks/sds_data_manager_stack.py 32.50% <32.50%> (ø)
tests/lambda_endpoints/test_queries.py 42.22% <40.00%> (ø)
sds_data_manager/stacks/opensearch_stack.py 53.33% <53.33%> (ø)
tests/infrastructure/conftest.py 70.58% <70.00%> (-29.42%) ⬇️
..._manager/lambda_code/SDSCode/download_query_api.py 92.10% <100.00%> (+0.21%) ⬆️
tests/lambda_endpoints/conftest.py 100.00% <100.00%> (ø)
... and 7 more

... and 1 file with indirect coverage changes

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

class TestQueries(unittest.TestCase):
def setUp(self):
# Opensearch client Params
os.environ[
"OS_DOMAIN"
] = "search-sds-metadata-uum2vnbdbqbnh7qnbde6t74xim.us-west-2.es.amazonaws.com"
] = "search-sds-metadata-uum2vnbdbqbnh7qnbde6t74xim.us-east-1.es.amazonaws.com"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You could consider doing this. You would still have to hard-code the DomainName though so maybe not worth the change.

    session = boto3.Session()

    # get the opensearch client
    opensearch_client = session.client('opensearch')

    # describe the OpenSearch domain and get its endpoint
    domain_description = opensearch_client.describe_domain(DomainName='sdsmetadatadomain-tlcs-dev')
    os.environ["OS_DOMAIN"] = domain_description['DomainStatus']['Endpoint']

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That is a really good idea @laspsandoval! Right now this test doesn't even pass I don't think because it requires the opensearch stack to be setup and running already, so we are testing against deployed resources. @sdhoyt is that right, or is there a different way to test this?

Also, I just found out about vcrpy that may be an interesting thing to investigate for these remote resources and testing against them without requiring them to be available 24/7.
https://vcrpy.readthedocs.io/en/latest/
with a pytest plugin that we could make use of:
https://pytest-vcr.readthedocs.io/en/latest/
For another time though...

@laspsandoval
Copy link
Contributor

looks good to me. I'll submit my pr on change for test_data_manager_stack.py

assert result["Buckets"][0]["Name"] == BUCKET_NAME

# upload a file
local_filepath = f"tests/test-data/{TEST_FILE}"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Make this change and the test will pass:
local_filepath = os.path.join(os.path.dirname(file), '..', 'test-data', TEST_FILE)

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This test should be passing already? But, I agree that in general we should move towards Pathlib for path joining and our friends using Windows ;) I've made a similar change, but with pathlib instead.

@laspsandoval
Copy link
Contributor

Made updates to infrastructure tests. Note that I left some comments in to explain the changes. I will remove those comments before we merge.

@greglucas
Copy link
Collaborator Author

Made updates to infrastructure tests. Note that I left some comments in to explain the changes. I will remove those comments before we merge.

The tests you added don't pass... Can we move that to a separate branch that you rebase off of this to keep them separate PRs? I think that is actually the more complicated piece you're tackling, and I'd like to keep incrementally making updates rather than intimidating people with large PRs to review.

@greglucas
Copy link
Collaborator Author

@laspsandoval , I just removed your commit and rebased and added the additional change you suggested. Let me know if this messed up your workflow at all and I'm happy to try and help out on that front.

@maxinelasp I also added the [--no-update] arg to pre-commit here so I didn't have to update the poetry.lock file (copying over from the imap_processing repository, IMAP-Science-Operations-Center/imap_processing#41). I'm happy to move that out to a separate PR if you would prefer too.


@pytest.fixture()
def s3_client(_aws_credentials):
with mock_s3():
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does this mean all the tests in lambda_endpoints are using a mock for all the S3 calls?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, any tests that have def test_name(s3_client): in them would use this fixture and therefore get the mock. I also added another "autouse" fixture that relies on this one so that all of the lambda_endpoints get this during the setup step without needing to think about it. It is perhaps hiding a bit too much, but with the number of tests we had defined it seemed nicer to do this and make sure we were getting mocks all the time rather than a potential mix in the future.

Copy link
Contributor

@maxinelasp maxinelasp left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think leaving the poetry change here is fine, it's not really a major difference. It does look like poetry.lock has some problems merging, I assume you can just keep the changes from this branch and that will be fine.

It looks like the code coverage bot is adding warnings to the tests that the lines aren't tested - is that expected behavior? Seems odd to me.

Overall it looks good!

@greglucas
Copy link
Collaborator Author

It looks like the code coverage bot is adding warnings to the tests that the lines aren't tested - is that expected behavior?

We will need to make some updates to make codecov happy later, but it is a bit of a misnomer for now since we are skipping entire chunks of tests in this and I'm purposefully just doing it incrementally. @laspsandoval will be updating some more after this to hopefully get us a bit further along but I suspect we still won't be at 100% and then we can start digging more into it.

In general though, I think codecov warning about not hitting tests is actually important and telling us that our tests aren't running :) We should have 100% coverage of tests, otherwise, we aren't testing what we think we're testing!



@pytest.fixture(autouse=True)
def setup_s3(s3_client):
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure how much we should care about this for test functions, but it looks like some docstrings are missing here (as well as the test_object_exists_with_s3_uri function)

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yep, good catch here! I think if something isn't immediately obvious we should have a docstring or comment letting us know what it is. I've also added a few others.

Specifically, though, I'd say that fixtures definitely should have some comments since they are meant for re-use, tests I don't mind as much if they don't have docstrings as long as they are named well and obvious about what they do. So, up to the developer on those.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd also like to see a specific note on whether the tests are using mocked clients or real ones - unless it's obvious. So I think the existing comment is good!

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
This is a major reorganization and refactor of the test suite to
try and align more closely with pytest-style tests, using fixtures
instead of setup/teardown methods.

The tests have been reorganized into folders that more closely
align to names that are being tested as well.
@bourque bourque self-requested a review July 25, 2023 16:11
@greglucas greglucas merged commit 9ddefd4 into IMAP-Science-Operations-Center:development Jul 26, 2023
@greglucas greglucas deleted the test-reorg branch July 26, 2023 15:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
testing Testing related updates
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants