[GITHUB-2897] Not exception but warning if some (not all) of the give… #2900

wenijinew · 2023-04-25T10:09:08Z

…n test names are not found in suite files.

Change-Id: Ica210f6c05b395f1d0a6718479f71b645ebd39cc

Fixes #2897 .

Did you remember to?

Add test case(s)
Update CHANGES.txt
Auto applied styling via ./gradlew autostyleApply

We encourage pull requests that:

Add new features to TestNG (or)
Fix bugs in TestNG

If your pull request involves fixing SonarQube issues then we would suggest that you please discuss this with the
TestNG-dev before you spend time working on it.

Note: For more information on contribution guidelines please make sure you refer our Contributing section for detailed set of steps.

…n test names are not found in suite files. Change-Id: Ica210f6c05b395f1d0a6718479f71b645ebd39cc Signed-off-by: Bruce Wen <wenijinew@gmail.com>

…n test names are not found in suite files. 1. Add new boolean option '-ignoreMissedTestNames' to work with the option '-testnames'. 2. When -testnames is given, and '-ignoreMissedTestNames true' is also given, then in case any missed test names not found in the suite, only warning message will be printed, TestNG will continue to run other test names which are existing in the suite. 3. Users who are going to use the new option '-ignoreMissedTestNames' should be aware of that the logging level should be properly configured to make sure the warning message is visible in output or console, rather than missed the notification of the missed test names, if any. Change-Id: Ica210f6c05b395f1d0a6718479f71b645ebd39cc Signed-off-by: Bruce Wen <wenijinew@gmail.com>

…n test names are not found in suite files. Change-Id: Ica210f6c05b395f1d0a6718479f71b645ebd39cc Signed-off-by: Bruce Wen <wenijinew@gmail.com>

test names are not found in suite files. Change-Id: Ic1835ef6768b8594f54217b3eb8dfd282901aa29

(not all) of the given test names are not found in suite files. 1. Add new boolean option '-ignoreMissedTestNames' to work with the option '-testnames'. 2. When -testnames is given, and '-ignoreMissedTestNames true' is also given, then in case any missed test names not found in the suite, only warning message will be printed, TestNG will continue to run other test names which are existing in the suite. 3. Users who are going to use the new option '-ignoreMissedTestNames' should be aware of that the logging level should be properly configured to make sure the warning message is visible in output or console, rather than missed the notification of the missed test names, if any. Change-Id: I259ae3feef5bebd3a58afe634b7c96a8d6e9ea01

wenijinew

Fixed typos and import issues

testng-core/src/main/java/org/testng/JarFileUtils.java

testng-core/src/main/java/org/testng/CommandLineArgs.java

testng-core/src/test/java/org/testng/xml/internal/TestNamesMatcherTest.java

(not all) of the given test names are not found in suite files. 1. Add new boolean option '-ignoreMissedTestNames' to work with the option '-testnames'. Add new supported system property 'testng.ignore.missed.testnames' for the same purpose. 2. When -testnames is given, and '-ignoreMissedTestNames true' is also given, then in case any missed test names not found in the suite, only warning message will be printed, TestNG will continue to run other test names which are existing in the suite. 3. Users who are going to use the new option '-ignoreMissedTestNames' should be aware of that the logging level should be properly configured to make sure the warning message is visible in output or console, rather than missed the notification of the missed test names, if any. Change-Id: I259ae3feef5bebd3a58afe634b7c96a8d6e9ea01

wenijinew

to fix issues about import and boolean values

testng-core/src/test/java/org/testng/xml/internal/TestNamesMatcherTest.java

testng-core/src/main/java/org/testng/xml/internal/TestNamesMatcher.java

(not all) of the given test names are not found in suite files. 1. Add new boolean option '-ignoreMissedTestNames' to work with the option '-testnames'. Add new supported system property 'testng.ignore.missed.testnames' for the same purpose. 2. When -testnames is given, and '-ignoreMissedTestNames true' is also given, then in case any missed test names not found in the suite, only warning message will be printed, TestNG will continue to run other test names which are existing in the suite. 3. Users who are going to use the new option '-ignoreMissedTestNames' should be aware of that the logging level should be properly configured to make sure the warning message is visible in output or console, rather than missed the notification of the missed test names, if any. Change-Id: I259ae3feef5bebd3a58afe634b7c96a8d6e9ea01

(not all) of the given test names are not found in suite files. 1. Add new boolean option '-ignoreMissedTestNames' to work with the option '-testnames'. Add new supported system property 'testng.ignore.missed.testnames' for the same purpose. 2. When -testnames is given, and '-ignoreMissedTestNames true' is also given, then in case any missed test names not found in the suite, only warning message will be printed, TestNG will continue to run other test names which are existing in the suite. 3. Users who are going to use the new option '-ignoreMissedTestNames' should be aware of that the logging level should be properly configured to make sure the warning message is visible in output or console, rather than missed the notification of the missed test names, if any. Change-Id: Iffbf3803c706ea4a9c7dd0930cec9c666962c16a Signed-off-by: Bruce Wen <wenijinew@gmail.com>

testng-core/src/main/java/org/testng/JarFileUtils.java

testng-core-api/src/main/java/org/testng/internal/RuntimeBehavior.java

testng-core/src/main/java/org/testng/xml/internal/TestNamesMatcher.java

CHANGES.txt

krmahadevan · 2023-04-27T03:23:19Z

@wenijinew - I see this warning This branch cannot be rebased due to conflicts on github on your PR. Can you please help fix this ?

Also please help add a test that vets out the below behaviour.

User has tests t1, t2,t3 in their suite.
User specifies the tests to run as t4, t5.
User has enabled the newly introduced configuration parameter that would basically ignore missing tests.

What would the behaviour be? Should TestNG return success, even though it didn't run any tests or should TestNG fail ?

(not all) of the given test names are not found in suite files. 1. Add new boolean option '-ignoreMissedTestNames' to work with the option '-testnames'. Add new supported system property 'testng.ignore.missed.testnames' for the same purpose. 2. When -testnames is given, and '-ignoreMissedTestNames true' is also given, then in case any missed test names not found in the suite, only warning message will be printed, TestNG will continue to run other test names which are existing in the suite. 3. Users who are going to use the new option '-ignoreMissedTestNames' should be aware of that the logging level should be properly configured to make sure the warning message is visible in output or console, rather than missed the notification of the missed test names, if any. Change-Id: Iffbf3803c706ea4a9c7dd0930cec9c666962c16a Signed-off-by: Bruce Wen <wenijinew@gmail.com>

Change-Id: I6796e900610fa6df169a6a63368af938cec9f204

(not all) of the given test names are not found in suite files. 1. Add new boolean option '-ignoreMissedTestNames' to work with the option '-testnames'. Add new supported system property 'testng.ignore.missed.testnames' for the same purpose. 2. When -testnames is given, and '-ignoreMissedTestNames true' is also given, then in case any missed test names not found in the suite, only warning message will be printed, TestNG will continue to run other test names which are existing in the suite. 3. Users who are going to use the new option '-ignoreMissedTestNames' should be aware of that the logging level should be properly configured to make sure the warning message is visible in output or console, rather than missed the notification of the missed test names, if any. Change-Id: I3b26df325b3e1576a8153c50c555a4dcadab820e

(not all) of the given test names are not found in suite files. 1. Add new boolean option '-ignoreMissedTestNames' to work with the option '-testnames'. Add new supported system property 'testng.ignore.missed.testnames' for the same purpose. 2. When -testnames is given, and '-ignoreMissedTestNames true' is also given, then in case any missed test names not found in the suite, only warning message will be printed, TestNG will continue to run other test names which are existing in the suite. 3. Users who are going to use the new option '-ignoreMissedTestNames' should be aware of that the logging level should be properly configured to make sure the warning message is visible in output or console, rather than missed the notification of the missed test names, if any. 4. Fixed legacy unit test issue for ensuring no exception thrown when given test names are from different levels of test suites. Change-Id: I614d74afabf8d9c8af28f0104188d68abafc1aa1

Change-Id: I166d1337bbb3dc7d511c9f2f8f9f85c4884cc851

Change-Id: I8c80c0fcd680c79e30b2405f10d1e0394d8bc3df

(not all) of the given test names are not found in suite files. 1. Add new boolean option '-ignoreMissedTestNames' to work with the option '-testnames'. Add new supported system property 'testng.ignore.missed.testnames' for the same purpose. 2. When -testnames is given, and '-ignoreMissedTestNames true' is also given, then in case any missed test names not found in the suite, only warning message will be printed, TestNG will continue to run other test names which are existing in the suite. 3. Users who are going to use the new option '-ignoreMissedTestNames' should be aware of that the logging level should be properly configured to make sure the warning message is visible in output or console, rather than missed the notification of the missed test names, if any. 4. Fixed legacy unit test issue for ensuring no exception thrown when given test names are from different levels of test suites. Change-Id: Icc02e7f9384a7213861d46dfb9aed246f4bde692

(not all) of the given test names are not found in suite files. 1. Add new boolean option '-ignoreMissedTestNames' to work with the option '-testnames'. Add new supported system property 'testng.ignore.missed.testnames' for the same purpose. 2. When -testnames is given, and '-ignoreMissedTestNames true' is also given, then in case any missed test names not found in the suite, only warning message will be printed, TestNG will continue to run other test names which are existing in the suite. 3. Users who are going to use the new option '-ignoreMissedTestNames' should be aware of that the logging level should be properly configured to make sure the warning message is visible in output or console, rather than missed the notification of the missed test names, if any. 4. Fixed legacy unit test issue for ensuring no exception thrown when given test names are from different levels of test suites. Change-Id: I03b554abe0a28a6717644b4388fd5e4d5f77b2bd

wenijinew · 2023-04-27T08:07:15Z

Exception will be thrown if all given test names are invalid. Please help check the latest code change and new test cases. Thanks.

One special case: given test names are empty. -testnames "". I've also made change for that.

if (testNames != null && !testNames().isEmpty())

Instead of throwing an strange exception like "The test(s) <[]> cannot be found", the whole suite will be run. Hope you agree on it.
https://groups.google.com/g/testng-users/c/bzuyXNtECFw

wenijinew · 2023-04-27T08:20:35Z

@wenijinew - I see this warning This branch cannot be rebased due to conflicts on github on your PR. Can you please help fix this ?

Also please help add a test that vets out the below behaviour.

User has tests t1, t2,t3 in their suite.

User specifies the tests to run as t4, t5.

User has enabled the newly introduced configuration parameter that would basically ignore missing tests.

What would the behaviour be? Should TestNG return success, even though it didn't run any tests or should TestNG fail ?

I suppose the conflicts are gone now.

wenijinew · 2023-04-27T08:31:59Z

Exception will be thrown if all given test names are invalid. Please help check the latest code change and new test cases. Thanks.

One special case: given test names are empty. -testnames "". I've also made change for that.

if (testNames != null && !testNames().isEmpty())

Instead of throwing an strange exception like "The test(s) <[]> cannot be found", the whole suite will be run. Hope you agree on it. https://groups.google.com/g/testng-users/c/bzuyXNtECFw

In that case, maybe more special input need to consider: test names are not an empty list but each element in the list is blank. :(

(not all) of the given test names are not found in suite files. 1. Add new boolean option '-ignoreMissedTestNames' to work with the option '-testnames'. Add new supported system property 'testng.ignore.missed.testnames' for the same purpose. 2. When -testnames is given, and '-ignoreMissedTestNames true' is also given, then in case any missed test names not found in the suite, only warning message will be printed, TestNG will continue to run other test names which are existing in the suite. 3. Users who are going to use the new option '-ignoreMissedTestNames' should be aware of that the logging level should be properly configured to make sure the warning message is visible in output or console, rather than missed the notification of the missed test names, if any. 4. Fixed legacy unit test issue for ensuring no exception thrown when given test names are from different levels of test suites. Change-Id: I03b554abe0a28a6717644b4388fd5e4d5f77b2bd

Change-Id: Ia4b599031512fcf99cfd24d646490d6ac7b583ed

(not all) of the given test names are not found in suite files. 1. Add new boolean option '-ignoreMissedTestNames' to work with the option '-testnames'. Add new supported system property 'testng.ignore.missed.testnames' for the same purpose. 2. When -testnames is given, and '-ignoreMissedTestNames true' is also given, then in case any missed test names not found in the suite, only warning message will be printed, TestNG will continue to run other test names which are existing in the suite. 3. Users who are going to use the new option '-ignoreMissedTestNames' should be aware of that the logging level should be properly configured to make sure the warning message is visible in output or console, rather than missed the notification of the missed test names, if any. 4. Fixed legacy unit test issue for ensuring no exception thrown when given test names are from different levels of test suites. Change-Id: I03b554abe0a28a6717644b4388fd5e4d5f77b2bd

(not all) of the given test names are not found in suite files. 1. Add new boolean option '-ignoreMissedTestNames' to work with the option '-testnames'. Add new supported system property 'testng.ignore.missed.testnames' for the same purpose. 2. When -testnames is given, and '-ignoreMissedTestNames true' is also given, then in case any missed test names not found in the suite, only warning message will be printed, TestNG will continue to run other test names which are existing in the suite. 3. Users who are going to use the new option '-ignoreMissedTestNames' should be aware of that the logging level should be properly configured to make sure the warning message is visible in output or console, rather than missed the notification of the missed test names, if any. 4. Fixed legacy unit test issue for ensuring no exception thrown when given test names are from different levels of test suites. Change-Id: I3a1e10c5443a9f5705ebe9d2fd4061157361389d

Change-Id: Id167abd2eae744b23d3c820e47129f618cae010b

…structors, not to instance method Change-Id: I75bb215fed1ba19938580391a7980439d937883b

…eMissedTestNames clear. Change-Id: Id831021470cd013ce55d5b4a245200b8ffee1897

testng-core/src/main/java/org/testng/JarFileUtils.java

testng-core/src/test/java/org/testng/JarFileUtilsTest.java

wenijinew · 2023-04-29T09:16:54Z

Also please help add a test that vets out the below behaviour.

User has tests t1, t2,t3 in their suite.

User specifies the tests to run as t4, t5.

User has enabled the newly introduced configuration parameter that would basically ignore missing tests.

What would the behaviour be? Should TestNG return success, even though it didn't run any tests or should TestNG fail ?

TestNamesMatcherTest#testHaveExceptionWhenIgnoreMissedTestNamesEnabledWithAllInvalidTestNames added for this and TestNGException is expected.

wenijinew · 2023-04-29T09:18:41Z

One special case: given test names are empty. -testnames "". I've also made change for that.

if (testNames != null && !testNames().isEmpty())

Instead of throwing an strange exception like "The test(s) <[]> cannot be found", the whole suite will be run. Hope you agree on it. https://groups.google.com/g/testng-users/c/bzuyXNtECFw

Any null or empty list, or any list with all elements are null or empty or blank are treated as invalid input, and exception will be thrown from TestNamesMatcher:

throw new TestNGException("Please provide a valid list of names to check.")

Added list wtih empty element in dataProvider = "getData"
Added test TestNamesMatcherTest#testCloneIfContainsTestsWithNamesMatchingAnyNegativeConditionWithIgnoreMissedTestNamesEnabled

(not all) of the given test names are not found in suite files. 1. Add new boolean option '-ignoreMissedTestNames' to work with the option '-testnames'. Add new supported system property 'testng.ignore.missed.testnames' for the same purpose. 2. When -testnames is given, and '-ignoreMissedTestNames true' is also given, then in case any missed test names not found in the suite, only warning message will be printed, TestNG will continue to run other test names which are existing in the suite. 3. Users who are going to use the new option '-ignoreMissedTestNames' should be aware of that the logging level should be properly configured to make sure the warning message is visible in output or console, rather than missed the notification of the missed test names, if any. 4. Fixed legacy unit test issue for ensuring no exception thrown when given test names are from different levels of test suites. 5. Abnormal case: when value of testNames is null, empty, or a list with all elements are null or empty or blank, exception will be thrown: TestNGException("Please provide a valid list of names to check."). Change-Id: I004dd3f500246856fc603d418fb834c96e8f6922

wenijinew · 2023-05-03T06:59:46Z

@krmahadevan

Could you please help review the latest commit? Thanks!

krmahadevan · 2023-05-03T14:09:20Z

@krmahadevan

Could you please help review the latest commit? Thanks!

@wenijinew - Please let me know once you have removed the JVM argument part in this PR. We are trying to stick to just setting all dynamic behaviours via command line arguments (which can be set both via a build tool as well as from the command line). Once done, please ping back. I can wrap up the review for this PR.

Change-Id: Ie278c1e68ca0ef6179f0689983c9f5f73e5eeb71

Change-Id: I063900208fd7ad28149dd53a3084783c640ab421

(not all) of the given test names are not found in suite files. 1. Add new boolean option '-ignoreMissedTestNames' to work with the option '-testnames'. 2. When -testnames is given, and '-ignoreMissedTestNames true' is also given, then in case any missed test names not found in the suite, only warning message will be printed, TestNG will continue to run other test names which are existing in the suite. 3. Users who are going to use the new option '-ignoreMissedTestNames' should be aware of that the logging level should be properly configured to make sure the warning message is visible in output or console, rather than missed the notification of the missed test names, if any. 4. Fixed legacy unit test issue for ensuring no exception thrown when given test names are from different levels of test suites. 5. Abnormal case: when value of testNames is null, empty, or a list with all elements are null or empty or blank, exception will be thrown: TestNGException("Please provide a valid list of names to check."). Change-Id: I37f762cdda4f91f9d43cc69cc03f0135ea282aaa

wenijinew · 2023-05-05T09:23:21Z

@krmahadevan

As requested, the VM argument has been removed in the latest commit. Please help review. Thanks.

krmahadevan · 2023-05-05T10:38:42Z

@juherr - Please help take a quick look. We would need to squash and merge this PR, because there are a lot of commits in this PR.

wenijinew added 6 commits April 25, 2023 12:05

[GITHUB-2897] Not exception but warning if some (not all) of the give…

86b4fb9

…n test names are not found in suite files. Change-Id: Ica210f6c05b395f1d0a6718479f71b645ebd39cc Signed-off-by: Bruce Wen <wenijinew@gmail.com>

[GITHUB-2897] Not exception but warning if some (not all) of the give…

ca68ec6

…n test names are not found in suite files. Change-Id: Ica210f6c05b395f1d0a6718479f71b645ebd39cc Signed-off-by: Bruce Wen <wenijinew@gmail.com>

[GITHUB-2897] Not exception but warning if some (not all) of the given

0894435

test names are not found in suite files. Change-Id: Ic1835ef6768b8594f54217b3eb8dfd282901aa29

wenijinew marked this pull request as ready for review April 26, 2023 11:38

wenijinew requested review from juherr and krmahadevan as code owners April 26, 2023 11:38

wenijinew added 3 commits April 26, 2023 13:53

wenijinew commented Apr 26, 2023

View reviewed changes

wenijinew added 2 commits April 26, 2023 15:54

wenijinew commented Apr 26, 2023

View reviewed changes

testng-core/src/main/java/org/testng/JarFileUtils.java Show resolved Hide resolved

krmahadevan reviewed Apr 27, 2023

View reviewed changes

testng-core-api/src/main/java/org/testng/internal/RuntimeBehavior.java Outdated Show resolved Hide resolved

krmahadevan reviewed Apr 27, 2023

View reviewed changes

testng-core/src/main/java/org/testng/xml/internal/TestNamesMatcher.java Outdated Show resolved Hide resolved

krmahadevan reviewed Apr 27, 2023

View reviewed changes

CHANGES.txt Show resolved Hide resolved

wenijinew added 8 commits April 27, 2023 07:28

Merge branch '2897' of https://github.com/wenijinew/testng into 2897

82278a5

Change-Id: I6796e900610fa6df169a6a63368af938cec9f204

A

19f2761

Change-Id: I166d1337bbb3dc7d511c9f2f8f9f85c4884cc851

Merge branch '2897' of https://github.com/wenijinew/testng into 2897

050ef65

Change-Id: I8c80c0fcd680c79e30b2405f10d1e0394d8bc3df

wenijinew requested a review from krmahadevan April 27, 2023 08:01

wenijinew added 9 commits April 27, 2023 15:17

Merge branch '2897' of https://github.com/wenijinew/testng into 2897

31c7f8c

Change-Id: Ia4b599031512fcf99cfd24d646490d6ac7b583ed

Merge branch 'master' into 2897

d20f21a

Backward compatible when empty or blank test names are given

80fd124

Change-Id: Id167abd2eae744b23d3c820e47129f618cae010b

The option ignoredMissedTestNames should be passed from TestNG or con…

2a12d6d

…structors, not to instance method Change-Id: I75bb215fed1ba19938580391a7980439d937883b

Optimize test cases, Fix legacy code smell, Make the meaning of ignor…

153a46e

…eMissedTestNames clear. Change-Id: Id831021470cd013ce55d5b4a245200b8ffee1897

wenijinew commented Apr 29, 2023

View reviewed changes

testng-core/src/main/java/org/testng/JarFileUtils.java Show resolved Hide resolved

testng-core/src/test/java/org/testng/JarFileUtilsTest.java Show resolved Hide resolved

wenijinew added 3 commits May 5, 2023 10:18

Remove VM argument support

1b421f3

Change-Id: Ie278c1e68ca0ef6179f0689983c9f5f73e5eeb71

Merge branch 'master' of https://github.com/wenijinew/testng into 2897

4b0e483

Change-Id: I063900208fd7ad28149dd53a3084783c640ab421

krmahadevan approved these changes May 5, 2023

View reviewed changes

juherr approved these changes May 5, 2023

View reviewed changes

krmahadevan merged commit 4c7d903 into testng-team:master May 6, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[GITHUB-2897] Not exception but warning if some (not all) of the give… #2900

[GITHUB-2897] Not exception but warning if some (not all) of the give… #2900

wenijinew commented Apr 25, 2023 •

edited

Loading

wenijinew left a comment

wenijinew left a comment

krmahadevan commented Apr 27, 2023

wenijinew commented Apr 27, 2023 •

edited

Loading

wenijinew commented Apr 27, 2023

wenijinew commented Apr 27, 2023

wenijinew commented Apr 29, 2023

wenijinew commented Apr 29, 2023 •

edited

Loading

wenijinew commented May 3, 2023

krmahadevan commented May 3, 2023

wenijinew commented May 5, 2023

krmahadevan commented May 5, 2023

[GITHUB-2897] Not exception but warning if some (not all) of the give… #2900

[GITHUB-2897] Not exception but warning if some (not all) of the give… #2900

Conversation

wenijinew commented Apr 25, 2023 • edited Loading

Did you remember to?

wenijinew left a comment

Choose a reason for hiding this comment

wenijinew left a comment

Choose a reason for hiding this comment

krmahadevan commented Apr 27, 2023

wenijinew commented Apr 27, 2023 • edited Loading

wenijinew commented Apr 27, 2023

wenijinew commented Apr 27, 2023

wenijinew commented Apr 29, 2023

wenijinew commented Apr 29, 2023 • edited Loading

wenijinew commented May 3, 2023

krmahadevan commented May 3, 2023

wenijinew commented May 5, 2023

krmahadevan commented May 5, 2023

wenijinew commented Apr 25, 2023 •

edited

Loading

wenijinew commented Apr 27, 2023 •

edited

Loading

wenijinew commented Apr 29, 2023 •

edited

Loading