dotnet test argument errors buried in noise #52297
Description
Is your feature request related to a problem? Please describe.
If you run dotnet test in MTP mode with an invalid argument, whilst you do get a clear error message at the top, you then get hundreds of lines (int he case of xunit over 700 lines) after it, which does a great job of hiding the error
For example, if you run dotnet test --show-live-output true for an xunit test app, you'll get the following output:
artifacts\bin\SharedAppHost\debug\SharedAppHost.dll (net10.0) Zero tests ran
Exit code: 5
Standard output: xUnit.net v3 Microsoft.Testing.Platform v2 Runner v3.2.1+a9cfb80929 (64-bit .NET 10.0.1)
Option '--show-live-output' has invalid arguments: Invalid value 'true' (must be one of: 'on', 'off')
Usage SharedAppHost.exe [option providers] [extension option providers]
Execute a .NET Test Application.
Options:
--config-file
Specifies a testconfig.json file.
...<SNIP> - 700 more lines </SNIP>...
Describe the solution you'd like
Whilst the error message as a whole is clear, it is followed by hundreds of lines of noise, and it can take quite a lot of scrolling to find the original error. And in a longer console session, you can easily scroll right past it).
I can think of three ideas that would help make these errors easier to spot:
- Repeat the actual error at the very end of the
dotnet test --helpoutput so it shows as the last line(s) of output. - When an error relates to a known argument (in this case
--show-live-output), print just the help output for that argument - i.e. something like:Option '--show-live-output' has invalid arguments: Invalid value 'true' (must be one of: 'on', 'off') --show-live-output Determine whether to show test output (from ITestOutputHelper) live during test execution. on - turn on live reporting of test output off - turn off live reporting of test output [default] - Don't print the output of
dotnet test --helpat all, and just print a singe line error along the lines of `Run "dotnet test --help" for more details on arguments"
Additional context
For full context, the full output from dotnet test --show-live-output true. Taken from https://github.com/afscrome/aspire-otel-testing.
Full Output
``` ➜ dotnet test --show-live-output true artifacts\bin\SharedAppHost\debug\SharedAppHost.dll (net10.0) Zero tests ran Exit code: 5 Standard output: xUnit.net v3 Microsoft.Testing.Platform v2 Runner v3.2.1+a9cfb80929 (64-bit .NET 10.0.1)Option '--show-live-output' has invalid arguments: Invalid value 'true' (must be one of: 'on', 'off')
Usage SharedAppHost.exe [option providers] [extension option providers]
Execute a .NET Test Application.
Options:
--config-file
Specifies a testconfig.json file.
--debug
Allows to pause execution in order to attach to the process for debug purposes.
--diagnostic
Enable the diagnostic logging. The default log level is 'Trace'.
The file will be written in the output directory with the name log_[yyMMddHHmmssfff].diag
--diagnostic-file-prefix
Prefix for the log file name that will replace '[log]_.'
--diagnostic-output-directory
Output directory of the diagnostic logging.
If not specified the file will be generated inside the default 'TestResults' directory.
--diagnostic-synchronous-write
Force the built-in file logger to write the log synchronously.
Useful for scenario where you don't want to lose any log (i.e. in case of crash).
Note that this is slowing down the test execution.
--diagnostic-verbosity
Define the level of the verbosity for the --diagnostic.
The available values are 'Trace', 'Debug', 'Information', 'Warning', 'Error', and 'Critical'.
--exit-on-process-exit
Exit the test process if dependent process exits. PID must be provided.
--filter-uid
Provides a list of test node UIDs to filter by.
--help
Show the command line help.
--ignore-exit-code
Do not report non successful exit value for specific exit codes
(e.g. '--ignore-exit-code 8;9' ignore exit code 8 and 9 and will return 0 in these case)
--info
Display .NET test application information.
--list-tests
List available tests.
--minimum-expected-tests
Specifies the minimum number of tests that are expected to run.
--results-directory
The directory where the test results are going to be placed.
If the specified directory doesn't exist, it's created.
The default is TestResults in the directory that contains the test application.
--timeout
A global test execution timeout.
Takes one argument as string in the format <value>[h|m|s] where 'value' is float.
Extension options:
--assert-equivalent-max-depth
Set the maximum recursive depth when comparing objects with Assert.Equivalent. Default value is 50.
(integer) - maximum depth to compare; exceeding this fails the assertion
--auto-reporters
Change whether reporters can be auto-enabled.
on - allow reporters to be auto-enabled by the environment [default]
off - do not allow reporters to be auto-enabled by the environment
--coverage
Collect the code coverage using dotnet-coverage tool.
--coverage-output
The output of the generated code coverage report.
If only filename is specified then the coverage report will be generated in the '--results-directory' directory.
--coverage-output-format
Output file format. Supported values: 'coverage', 'xml' and 'cobertura'
--coverage-settings
The path to the code coverage XML settings file.
--culture
Run tests under the given culture.
default - run with the default operating system culture [default]
invariant - run with the invariant culture
(string) - run with the given culture (i.e., 'en-US')
--explicit
Change the way explicit tests are handled.
on - run both explicit and non-explicit tests
off - run only non-explicit tests [default]
only - run only explicit tests
--fail-skips
Change the way skipped tests are handled.
on - treat skipped tests as failed
off - treat skipped tests as skipped [default]
--fail-warns
Change the way passing tests with warnings are handled.
on - treat passing tests with warnings as failed
off - treat passing tests with warnings as passed [default]
--filter-class
Run all methods in a given test class. Pass one or more fully qualified type names (i.e.,
'MyNamespace.MyClass' or 'MyNamespace.MyClass+InnerClass'). Wildcard '*' is supported at
the beginning and/or end of each filter.
Note: Specifying more than one is an OR operation.
This is categorized as a simple filter. You cannot use both simple filters and query filters.
--filter-method
Run a given test method. Pass one or more fully qualified method names (i.e.,
'MyNamespace.MyClass.MyTestMethod'). Wildcard '*' is supported at the beginning and/or end
of each filter.
Note: Specifying more than one is an OR operation.
This is categorized as a simple filter. You cannot use both simple filters and query filters.
--filter-namespace
Run all methods in the given namespace. Pass one or more namespaces (i.e., 'MyNamespace' or
'MyNamespace.MySubNamespace'). Wildcard '*' is supported at the beginning and/or end of
each filter.
Note: Specifying more than one is an OR operation.
This is categorized as a simple filter. You cannot use both simple filters and query filters.
--filter-not-class
Do not run any methods in the given test class. Pass one or more fully qualified type names
(i.e., 'MyNamespace.MyClass', or 'MyNamespace.MyClass+InnerClass'). Wildcard '*' is supported at
the beginning and/or end of each filter.
Note: Specifying more than one is an AND operation.
This is categorized as a simple filter. You cannot use both simple filters and query filters.
--filter-not-method
Do not run a given test method. Pass one or more fully qualified method names (i.e.,
'MyNamespace.MyClass.MyTestMethod'). Wildcard '*' is supported at the beginning and/or end
of each filter.
Note: Specifying more than one is an AND operation.
This is categorized as a simple filter. You cannot use both simple filters and query filters.
--filter-not-namespace
Do not run any methods in the given namespace. Pass one or more namespaces (i.e., 'MyNamespace'
or 'MyNamespace.MySubNamespace'). Wildcard '*' is supported at the beginning and/or end of
each filter.
Note: Specifying more than one is an AND operation.
This is categorized as a simple filter. You cannot use both simple filters and query filters.
--filter-not-trait
Do not run any methods with a given trait value. Pass one or more name/value pairs (i.e.,
'name=value'). Wildcard '*' is supported at the beginning and/or end of the trait name
and/or value.
Note: Specifying more than one is an AND operation.
This is categorized as a simple filter. You cannot use both simple filters and query filters.
--filter-query
Filter based on the filter query language. Pass one or more filter queries (in the
'/assemblyName/namespace/type/method[trait=value]' format. For more information, see
https://xunit.net/docs/query-filter-language
Note: Specifying more than one is an OR operation.
This is categorized as a query filter. You cannot use both query filters and simple filters.
--filter-trait
Run all methods with a given trait value. Pass one or more name/value pairs (i.e.,
'name=value'). Wildcard '*' is supported at the beginning and/or end of the trait name
and/or value.
Note: Specifying more than one is an OR operation.
This is categorized as a simple filter. You cannot use both simple filters and query filters.
--hangdump
Generate a dump file if the test process hangs
--hangdump-filename
Specify the name of the dump file
--hangdump-timeout
Specify the timeout after which the dump will be generated.
The timeout value is specified in one of the following formats:
1.5h, 1.5hour, 1.5hours,
90m, 90min, 90minute, 90minutes,
5400s, 5400sec, 5400second, 5400seconds.
Default is 30m.
--hangdump-type
Specify the type of the dump.
Valid values are 'Mini', 'Heap', 'Triage' (only available in .NET 6+) or 'Full'.
Default type is 'Full'
--long-running
Enable long running (hung) test detection.
(integer) - number of seconds a test runs to be considered 'long running'
--max-threads
Set maximum thread count for collection parallelization.
default - run with default (1 thread per CPU thread)
unlimited - run with unbounded thread count
(integer) - use exactly this many threads (e.g., '2' = 2 threads)
(float)x - use a multiple of CPU threads (e.g., '2.0x' = 2.0 * the number of CPU threads)
--method-display
Set default test display name.
classAndMethod - use a fully qualified name [default]
method - use just the method name
--method-display-options
Alters the default test display name.
none - apply no alterations [default]
all - apply all alterations
Or one or more of:
replacePeriodWithComma - replace periods in names with commas
replaceUnderscoreWithSpace - replace underscores in names with spaces
useOperatorMonikers - replace operator names with operators
'lt' becomes '<'
'le' becomes '<='
'eq' becomes '='
'ne' becomes '!='
'gt' becomes '>'
'ge' becomes '>='
useEscapeSequences - replace ASCII and Unicode escape sequences
X + 2 hex digits (i.e., 'X2C' becomes ',')
U + 4 hex digits (i.e., 'U0192' becomes 'ƒ')
--no-ansi
Disable outputting ANSI escape characters to screen.
--no-progress
Disable reporting progress to screen.
--output
Output verbosity when reporting tests.
Valid values are 'Normal', 'Detailed'. Default is 'Normal'.
--parallel
Change test parallelization.
none - turn off parallelization
collections - parallelize by collections [default]
--parallel-algorithm
Change the parallelization algorithm.
conservative - start the minimum number of tests [default]
aggressive - start as many tests as possible
--pre-enumerate-theories
Change theory pre-enumeration during discovery.
on - turns on theory pre-enumeration [default]
off - turns off theory pre-enumeration
--print-max-enumerable-length
Set the maximum number of values to show when printing a collection. Default value is 5.
0 - always print the full collection
(integer) - maximum values to print, followed by an ellipsis
--print-max-object-depth
Set the maximum recursive depth when printing object values. Default value is 3.
0 - print objects at all depths
(integer) - maximum depth to print, followed by an ellipsis
Warning: Setting '0' or a very large value can cause stack overflows that may crash the test process
--print-max-object-member-count
Set the maximum number of fields and properties to show when printing an object. Default value is 5.
0 - always print the full collection
(integer) - maximum members to print, followed by an ellipsis
--print-max-string-length
Set the maximum length when printing a string. Default value is 50.
0 - always print the full collection
(integer) - maximum string length to print, followed by an ellipsis
--report-ctrf
Enable generating CTRF (JSON) report
--report-ctrf-filename
The name of the generated CTRF report
--report-junit
Enable generating JUnit (XML) report
--report-junit-filename
The name of the generated JUnit report
--report-nunit
Enable generating NUnit (v2.5 XML) report
--report-nunit-filename
The name of the generated NUnit report
--report-trx
Enable generating TRX report
--report-trx-filename
The name of the generated TRX report
--report-xunit
Enable generating xUnit.net (v2+ XML) report
--report-xunit-filename
The name of the generated xUnit.net report
--report-xunit-html
Enable generating xUnit.net HTML report
--report-xunit-html-filename
The name of the generated xUnit.net HTML report
--report-xunit-trx
Enable generating xUnit.net TRX report
--report-xunit-trx-filename
The name of the generated xUnit.net TRX report
--seed
Set the randomization seed.
(integer) - use this as the randomization seed
--show-live-output
Determine whether to show test output (from ITestOutputHelper) live during test execution.
on - turn on live reporting of test output
off - turn off live reporting of test output [default]
--stop-on-fail
Stop running tests after the first test failure.
on - stop running tests after the first test failure
off - run all tests regardless of failures [default]
--xunit-config-filename
Sets the configuration file. By default, this is 'xunit.runner.json' in the bin directory
alongside the compiled output.
--xunit-diagnostics
Determine whether to show diagnostic messages.
on - display diagnostic messages
off - hide diagnostic messages [default]
--xunit-info
Show xUnit.net headers and information
--xunit-internal-diagnostics
Determine whether to show internal diagnostic messages.
on - display internal diagnostic messages
off - hide internal diagnostic messages [default]
artifacts\bin\AppHostPerTest\debug\AppHostPerTest.dll (net10.0) Zero tests ran
Exit code: 5
Standard output: xUnit.net v3 Microsoft.Testing.Platform v2 Runner v3.2.1+a9cfb80929 (64-bit .NET 10.0.1)
Option '--show-live-output' has invalid arguments: Invalid value 'true' (must be one of: 'on', 'off')
Usage AppHostPerTest.exe [option providers] [extension option providers]
Execute a .NET Test Application.
Options:
--config-file
Specifies a testconfig.json file.
--debug
Allows to pause execution in order to attach to the process for debug purposes.
--diagnostic
Enable the diagnostic logging. The default log level is 'Trace'.
The file will be written in the output directory with the name log_[yyMMddHHmmssfff].diag
--diagnostic-file-prefix
Prefix for the log file name that will replace '[log]_.'
--diagnostic-output-directory
Output directory of the diagnostic logging.
If not specified the file will be generated inside the default 'TestResults' directory.
--diagnostic-synchronous-write
Force the built-in file logger to write the log synchronously.
Useful for scenario where you don't want to lose any log (i.e. in case of crash).
Note that this is slowing down the test execution.
--diagnostic-verbosity
Define the level of the verbosity for the --diagnostic.
The available values are 'Trace', 'Debug', 'Information', 'Warning', 'Error', and 'Critical'.
--exit-on-process-exit
Exit the test process if dependent process exits. PID must be provided.
--filter-uid
Provides a list of test node UIDs to filter by.
--help
Show the command line help.
--ignore-exit-code
Do not report non successful exit value for specific exit codes
(e.g. '--ignore-exit-code 8;9' ignore exit code 8 and 9 and will return 0 in these case)
--info
Display .NET test application information.
--list-tests
List available tests.
--minimum-expected-tests
Specifies the minimum number of tests that are expected to run.
--results-directory
The directory where the test results are going to be placed.
If the specified directory doesn't exist, it's created.
The default is TestResults in the directory that contains the test application.
--timeout
A global test execution timeout.
Takes one argument as string in the format <value>[h|m|s] where 'value' is float.
Extension options:
--assert-equivalent-max-depth
Set the maximum recursive depth when comparing objects with Assert.Equivalent. Default value is 50.
(integer) - maximum depth to compare; exceeding this fails the assertion
--auto-reporters
Change whether reporters can be auto-enabled.
on - allow reporters to be auto-enabled by the environment [default]
off - do not allow reporters to be auto-enabled by the environment
--coverage
Collect the code coverage using dotnet-coverage tool.
--coverage-output
The output of the generated code coverage report.
If only filename is specified then the coverage report will be generated in the '--results-directory' directory.
--coverage-output-format
Output file format. Supported values: 'coverage', 'xml' and 'cobertura'
--coverage-settings
The path to the code coverage XML settings file.
--culture
Run tests under the given culture.
default - run with the default operating system culture [default]
invariant - run with the invariant culture
(string) - run with the given culture (i.e., 'en-US')
--explicit
Change the way explicit tests are handled.
on - run both explicit and non-explicit tests
off - run only non-explicit tests [default]
only - run only explicit tests
--fail-skips
Change the way skipped tests are handled.
on - treat skipped tests as failed
off - treat skipped tests as skipped [default]
--fail-warns
Change the way passing tests with warnings are handled.
on - treat passing tests with warnings as failed
off - treat passing tests with warnings as passed [default]
--filter-class
Run all methods in a given test class. Pass one or more fully qualified type names (i.e.,
'MyNamespace.MyClass' or 'MyNamespace.MyClass+InnerClass'). Wildcard '*' is supported at
the beginning and/or end of each filter.
Note: Specifying more than one is an OR operation.
This is categorized as a simple filter. You cannot use both simple filters and query filters.
--filter-method
Run a given test method. Pass one or more fully qualified method names (i.e.,
'MyNamespace.MyClass.MyTestMethod'). Wildcard '*' is supported at the beginning and/or end
of each filter.
Note: Specifying more than one is an OR operation.
This is categorized as a simple filter. You cannot use both simple filters and query filters.
--filter-namespace
Run all methods in the given namespace. Pass one or more namespaces (i.e., 'MyNamespace' or
'MyNamespace.MySubNamespace'). Wildcard '*' is supported at the beginning and/or end of
each filter.
Note: Specifying more than one is an OR operation.
This is categorized as a simple filter. You cannot use both simple filters and query filters.
--filter-not-class
Do not run any methods in the given test class. Pass one or more fully qualified type names
(i.e., 'MyNamespace.MyClass', or 'MyNamespace.MyClass+InnerClass'). Wildcard '*' is supported at
the beginning and/or end of each filter.
Note: Specifying more than one is an AND operation.
This is categorized as a simple filter. You cannot use both simple filters and query filters.
--filter-not-method
Do not run a given test method. Pass one or more fully qualified method names (i.e.,
'MyNamespace.MyClass.MyTestMethod'). Wildcard '*' is supported at the beginning and/or end
of each filter.
Note: Specifying more than one is an AND operation.
This is categorized as a simple filter. You cannot use both simple filters and query filters.
--filter-not-namespace
Do not run any methods in the given namespace. Pass one or more namespaces (i.e., 'MyNamespace'
or 'MyNamespace.MySubNamespace'). Wildcard '*' is supported at the beginning and/or end of
each filter.
Note: Specifying more than one is an AND operation.
This is categorized as a simple filter. You cannot use both simple filters and query filters.
--filter-not-trait
Do not run any methods with a given trait value. Pass one or more name/value pairs (i.e.,
'name=value'). Wildcard '*' is supported at the beginning and/or end of the trait name
and/or value.
Note: Specifying more than one is an AND operation.
This is categorized as a simple filter. You cannot use both simple filters and query filters.
--filter-query
Filter based on the filter query language. Pass one or more filter queries (in the
'/assemblyName/namespace/type/method[trait=value]' format. For more information, see
https://xunit.net/docs/query-filter-language
Note: Specifying more than one is an OR operation.
This is categorized as a query filter. You cannot use both query filters and simple filters.
--filter-trait
Run all methods with a given trait value. Pass one or more name/value pairs (i.e.,
'name=value'). Wildcard '*' is supported at the beginning and/or end of the trait name
and/or value.
Note: Specifying more than one is an OR operation.
This is categorized as a simple filter. You cannot use both simple filters and query filters.
--hangdump
Generate a dump file if the test process hangs
--hangdump-filename
Specify the name of the dump file
--hangdump-timeout
Specify the timeout after which the dump will be generated.
The timeout value is specified in one of the following formats:
1.5h, 1.5hour, 1.5hours,
90m, 90min, 90minute, 90minutes,
5400s, 5400sec, 5400second, 5400seconds.
Default is 30m.
--hangdump-type
Specify the type of the dump.
Valid values are 'Mini', 'Heap', 'Triage' (only available in .NET 6+) or 'Full'.
Default type is 'Full'
--long-running
Enable long running (hung) test detection.
(integer) - number of seconds a test runs to be considered 'long running'
--max-threads
Set maximum thread count for collection parallelization.
default - run with default (1 thread per CPU thread)
unlimited - run with unbounded thread count
(integer) - use exactly this many threads (e.g., '2' = 2 threads)
(float)x - use a multiple of CPU threads (e.g., '2.0x' = 2.0 * the number of CPU threads)
--method-display
Set default test display name.
classAndMethod - use a fully qualified name [default]
method - use just the method name
--method-display-options
Alters the default test display name.
none - apply no alterations [default]
all - apply all alterations
Or one or more of:
replacePeriodWithComma - replace periods in names with commas
replaceUnderscoreWithSpace - replace underscores in names with spaces
useOperatorMonikers - replace operator names with operators
'lt' becomes '<'
'le' becomes '<='
'eq' becomes '='
'ne' becomes '!='
'gt' becomes '>'
'ge' becomes '>='
useEscapeSequences - replace ASCII and Unicode escape sequences
X + 2 hex digits (i.e., 'X2C' becomes ',')
U + 4 hex digits (i.e., 'U0192' becomes 'ƒ')
--no-ansi
Disable outputting ANSI escape characters to screen.
--no-progress
Disable reporting progress to screen.
--output
Output verbosity when reporting tests.
Valid values are 'Normal', 'Detailed'. Default is 'Normal'.
--parallel
Change test parallelization.
none - turn off parallelization
collections - parallelize by collections [default]
--parallel-algorithm
Change the parallelization algorithm.
conservative - start the minimum number of tests [default]
aggressive - start as many tests as possible
--pre-enumerate-theories
Change theory pre-enumeration during discovery.
on - turns on theory pre-enumeration [default]
off - turns off theory pre-enumeration
--print-max-enumerable-length
Set the maximum number of values to show when printing a collection. Default value is 5.
0 - always print the full collection
(integer) - maximum values to print, followed by an ellipsis
--print-max-object-depth
Set the maximum recursive depth when printing object values. Default value is 3.
0 - print objects at all depths
(integer) - maximum depth to print, followed by an ellipsis
Warning: Setting '0' or a very large value can cause stack overflows that may crash the test process
--print-max-object-member-count
Set the maximum number of fields and properties to show when printing an object. Default value is 5.
0 - always print the full collection
(integer) - maximum members to print, followed by an ellipsis
--print-max-string-length
Set the maximum length when printing a string. Default value is 50.
0 - always print the full collection
(integer) - maximum string length to print, followed by an ellipsis
--report-ctrf
Enable generating CTRF (JSON) report
--report-ctrf-filename
The name of the generated CTRF report
--report-junit
Enable generating JUnit (XML) report
--report-junit-filename
The name of the generated JUnit report
--report-nunit
Enable generating NUnit (v2.5 XML) report
--report-nunit-filename
The name of the generated NUnit report
--report-trx
Enable generating TRX report
--report-trx-filename
The name of the generated TRX report
--report-xunit
Enable generating xUnit.net (v2+ XML) report
--report-xunit-filename
The name of the generated xUnit.net report
--report-xunit-html
Enable generating xUnit.net HTML report
--report-xunit-html-filename
The name of the generated xUnit.net HTML report
--report-xunit-trx
Enable generating xUnit.net TRX report
--report-xunit-trx-filename
The name of the generated xUnit.net TRX report
--seed
Set the randomization seed.
(integer) - use this as the randomization seed
--show-live-output
Determine whether to show test output (from ITestOutputHelper) live during test execution.
on - turn on live reporting of test output
off - turn off live reporting of test output [default]
--stop-on-fail
Stop running tests after the first test failure.
on - stop running tests after the first test failure
off - run all tests regardless of failures [default]
--xunit-config-filename
Sets the configuration file. By default, this is 'xunit.runner.json' in the bin directory
alongside the compiled output.
--xunit-diagnostics
Determine whether to show diagnostic messages.
on - display diagnostic messages
off - hide diagnostic messages [default]
--xunit-info
Show xUnit.net headers and information
--xunit-internal-diagnostics
Determine whether to show internal diagnostic messages.
on - display internal diagnostic messages
off - hide internal diagnostic messages [default]
Test run summary: Zero tests ran
error: 2
total: 0
failed: 0
succeeded: 0
skipped: 0
duration: 158ms
Test run completed with non-success exit code: 1 (see: https://aka.ms/testingplatform/exitcodes)
</details>