Skip to content

Commit

Permalink
fail when no tests were run and --fail-on-empty was set
Browse files Browse the repository at this point in the history
  • Loading branch information
grosser authored and onsi committed May 21, 2024
1 parent fcf1fd7 commit d80eebe
Show file tree
Hide file tree
Showing 5 changed files with 23 additions and 8 deletions.
2 changes: 1 addition & 1 deletion CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,4 +10,4 @@ Your contributions to Ginkgo are essential for its long-term maintenance and imp
- Vet your changes via `go vet ./...`
- Update the documentation. Ginkgo uses `godoc` comments and documentation in `docs/index.md`. You can run `bundle exec jekyll serve` in the `docs` directory to preview your changes.

Thanks for supporting Ginkgo!
Thanks for supporting Ginkgo!
18 changes: 12 additions & 6 deletions docs/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -2697,6 +2697,12 @@ These mechanisms can all be used in concert. They combine with the following ru
- Programmatic filters always apply and result in a non-zero exit code. Any additional CLI filters only apply to the subset of specs selected by the programmatic filters.
- When multiple CLI filters (`--label-filter`, `--focus-file/--skip-file`, `--focus/--skip`) are provided they are all ANDed together. The spec must satisfy the label filter query **and** any location-based filters **and** any description based filters.

#### Avoiding filtering out all tests

Especially for CI it is useful to fail when all tests were filtered out by accident (either via skip or typo in label filter).

`ginkgo --fail-on-empty --label-filter mytypo ./...` will fail since no test was run.

### Repeating Spec Runs and Managing Flaky Specs

Ginkgo wants to help you write reliable, deterministic, tests. Flaky specs - i.e. specs that fail _sometimes_ in non-deterministic or difficult to reason about ways - can be incredibly frustrating to debug and can erode faith in the value of a spec suite.
Expand Down Expand Up @@ -3098,8 +3104,8 @@ SynchronizedBeforeSuite(func(ctx SpecContext) []byte {
```
are all valid interruptible signatures. Of course you can specify `context.Context` instead and can mix-and-match interruptibility between the two functions.

**Reporting** nodes `ReportAfterEach`, `ReportBeforeEach`, `ReportBeforeSuite` `ReportAfterSuite` can be made interruptible,
to do this you need to provide it a node function which accepts both `SpecContext` and `SpecReport` for `*Each` nodes and `Report` for `*Suite` nodes.
**Reporting** nodes `ReportAfterEach`, `ReportBeforeEach`, `ReportBeforeSuite` `ReportAfterSuite` can be made interruptible,
to do this you need to provide it a node function which accepts both `SpecContext` and `SpecReport` for `*Each` nodes and `Report` for `*Suite` nodes.

As for **Container** nodes, since these run during the Tree Construction Phase they cannot be made interruptible and so do not accept functions that expect a context. And since the `By` annotation is simply syntactic sugar enabling more detailed spec documentation, any callbacks passed to `By` cannot be independently marked as interruptible (you should, instead, use the `context` passed into the node that you're calling `By` from).

Expand Down Expand Up @@ -3502,7 +3508,7 @@ Ginkgo's reporting infrastructure provides an alternative solution for this use

Ginkgo provides four reporting-focused nodes `ReportAfterEach`, `ReportBeforeEach` `ReportBeforeSuite`, and `ReportAfterSuite`.

`ReportAfterEach` behaves similarly to a standard `AfterEach` node and can be declared anywhere an `AfterEach` node can be declared.
`ReportAfterEach` behaves similarly to a standard `AfterEach` node and can be declared anywhere an `AfterEach` node can be declared.
`ReportAfterEach` can take either a closure that accepts a single [`SpecReport`](https://pkg.go.dev/github.com/onsi/ginkgo/v2/types#SpecReport) argument or both `SpecContext` and `SpecReport`
For example, we could implement a top-level ReportAfterEach that emits information about every spec to a remote server:

Expand All @@ -3511,7 +3517,7 @@ ReportAfterEach(func(report SpecReport) {
customFormat := fmt.Sprintf("%s | %s", report.State, report.FullText())
client.SendReport(customFormat)
})
// interruptible ReportAfterEach node
// interruptible ReportAfterEach node
ReportAfterEach(func(ctx SpecContext, report SpecReport) {
customFormat := fmt.Sprintf("%s | %s", report.State, report.FullText())
client.SendReport(customFormat)
Expand All @@ -3522,7 +3528,7 @@ ReportAfterEach(func(ctx SpecContext, report SpecReport) {

In addition, `ReportAfterEach` closures are called after a spec completes. i.e. _after_ all `AfterEach` closures have run. This gives them access to the complete final state of the spec. Note that if a failure occurs in a `ReportAfterEach` your the spec will be marked as failed. Subsequent `ReportAfterEach` closures will see the failed state, but not the closure in which the failure occurred.

`ReportAfterEach` is useful if you need to stream or emit up-to-date information about the suite as it runs. Ginkgo also provides `ReportBeforeEach` which is called before the test runs and
`ReportAfterEach` is useful if you need to stream or emit up-to-date information about the suite as it runs. Ginkgo also provides `ReportBeforeEach` which is called before the test runs and
receives a preliminary `types.SpecReport` ( or both `SpecContext` and `types.SpecReport` for interruptible behaviour) - the state of this report will indicate whether the test will be skipped or is marked pending.

You should be aware that when running in parallel, each parallel process will be running specs and their `ReportAfterEach`es. This means that multiple `ReportAfterEach` blocks can be running concurrently on independent processes. Given that, code like this won't work:
Expand All @@ -3542,7 +3548,7 @@ ReportAfterEach(func(report SpecReport) {
you'll end up with multiple processes writing to the same file and the output will be a mess. There is a better approach for this usecase...

#### Reporting Nodes - ReportBeforeSuite and ReportAfterSuite
`ReportBeforeSuite` and `ReportAfterSuite` nodes behave similarly to `BeforeSuite` and `AfterSuite` and can be placed at the top-level of your suite (typically in the suite bootstrap file).
`ReportBeforeSuite` and `ReportAfterSuite` nodes behave similarly to `BeforeSuite` and `AfterSuite` and can be placed at the top-level of your suite (typically in the suite bootstrap file).
`ReportBeforeSuite` node take a closure that accepts either [`Report`]((https://pkg.go.dev/github.com/onsi/ginkgo/v2/types#Report)) or, both `SpecContext` and `Report` converting the node to an interruptible node.

```go
Expand Down
7 changes: 6 additions & 1 deletion internal/suite.go
Original file line number Diff line number Diff line change
Expand Up @@ -489,10 +489,15 @@ func (suite *Suite) runSpecs(description string, suiteLabels Labels, suitePath s
newGroup(suite).run(specs.AtIndices(groupedSpecIndices[groupedSpecIdx]))
}

if specs.HasAnySpecsMarkedPending() && suite.config.FailOnPending {
if suite.config.FailOnPending && specs.HasAnySpecsMarkedPending() {
suite.report.SpecialSuiteFailureReasons = append(suite.report.SpecialSuiteFailureReasons, "Detected pending specs and --fail-on-pending is set")
suite.report.SuiteSucceeded = false
}

if suite.config.FailOnEmpty && specs.CountWithoutSkip() == 0 {
suite.report.SpecialSuiteFailureReasons = append(suite.report.SpecialSuiteFailureReasons, "Detected no specs ran and --fail-on-empty is set")
suite.report.SuiteSucceeded = false
}
}

if ranBeforeSuite {
Expand Down
1 change: 1 addition & 0 deletions reporters/junit_report.go
Original file line number Diff line number Diff line change
Expand Up @@ -177,6 +177,7 @@ func GenerateJUnitReportWithConfig(report types.Report, dst string, config Junit
{"FocusFiles", strings.Join(report.SuiteConfig.FocusFiles, ";")},
{"SkipFiles", strings.Join(report.SuiteConfig.SkipFiles, ";")},
{"FailOnPending", fmt.Sprintf("%t", report.SuiteConfig.FailOnPending)},
{"FailOnEmpty", fmt.Sprintf("%t", report.SuiteConfig.FailOnEmpty)},
{"FailFast", fmt.Sprintf("%t", report.SuiteConfig.FailFast)},
{"FlakeAttempts", fmt.Sprintf("%d", report.SuiteConfig.FlakeAttempts)},
{"DryRun", fmt.Sprintf("%t", report.SuiteConfig.DryRun)},
Expand Down
3 changes: 3 additions & 0 deletions types/config.go
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,7 @@ type SuiteConfig struct {
SkipFiles []string
LabelFilter string
FailOnPending bool
FailOnEmpty bool
FailFast bool
FlakeAttempts int
MustPassRepeatedly int
Expand Down Expand Up @@ -275,6 +276,8 @@ var SuiteConfigFlags = GinkgoFlags{
Usage: "If set, ginkgo will stop running a test suite after a failure occurs."},
{KeyPath: "S.FlakeAttempts", Name: "flake-attempts", SectionKey: "failure", UsageDefaultValue: "0 - failed tests are not retried", DeprecatedName: "flakeAttempts", DeprecatedDocLink: "changed-command-line-flags",
Usage: "Make up to this many attempts to run each spec. If any of the attempts succeed, the suite will not be failed."},
{KeyPath: "S.FailOnEmpty", Name: "fail-on-empty", SectionKey: "failure",
Usage: "If set, ginkgo will mark the test suite as failed if no specs are run."},

{KeyPath: "S.DryRun", Name: "dry-run", SectionKey: "debug", DeprecatedName: "dryRun", DeprecatedDocLink: "changed-command-line-flags",
Usage: "If set, ginkgo will walk the test hierarchy without actually running anything. Best paired with -v."},
Expand Down

0 comments on commit d80eebe

Please sign in to comment.