Skip to content

Commit

Permalink
No test left behind
Browse files Browse the repository at this point in the history
Part of #26088 and #17235

Ideas for better catching behavior changes in the product code. Specifically:
- Detect when a negative case stops failing
- Detect when a negative case starts failing in a different way

Fundamental approach: don't skip tests.

In NorthwindAggregateOperatorsQueryTests, we had:
- Negative cases that were no longer failing
- Negative cases that were skipped for all providers, but worked on some. For example:
  - Failed on relational, but passed on in-memory
  - Failed on relational, but passed on Cosmos
  - Failed on SQL Server, but passed on SQLite
- Negative cases that failed in different ways on different providers

Specifics:
- If a test throws, catch the exception
  - Were feasible, also validate the exception message or error number
- Always call base where possible, rather than repeating the query in an overriden test
- Add a standard comment where we have a bug or enhancement tracking the issue. For example:
  - `// Contains over subquery. Issue #17246.`
- Always have an `AssertSql` call in Cosmos and SQL Server tests
  - Where we expect a provider-specific class to verify SQL, then add a test that checks all test methods are overriden.
  • Loading branch information
ajcvickers committed Oct 17, 2021
1 parent bb46c5e commit 431bcb0
Show file tree
Hide file tree
Showing 7 changed files with 1,528 additions and 644 deletions.
Loading

0 comments on commit 431bcb0

Please sign in to comment.