Update test framework docs #4378
Conversation
Signed-off-by: Mats Wichmann <mats@linux.com>
testing/framework/test-framework.rst
Outdated
| behavior. In general, we should try to move those things to end-to-end | ||
| scripts as we find them.) | ||
| scripts as we find them. |
There was a problem hiding this comment.
Not sure I agree with the assertion that testing of fixed APIs should be moved to system tests.
If unit tests can adequately test API, they should be used as they are considerably faster and simpler to debug.
testing/framework/test-framework.rst
Outdated
| @@ -165,6 +189,13 @@ into a directory to debug after a test has gone wrong. | |||
| For a way around this, check out the ``PRESERVE`` environment variable. | |||
| It can be seen in action in `How to convert old tests to use fixures`_ below. | |||
|
|
|||
| By the way, there's nothing magical about ``runtest.py``, the intent | |||
There was a problem hiding this comment.
Not sure there's any benefit to encouraging anyone to submit PR with tests not using the current test infrastructure.
Just would lead to more maintenance burden.. (mainly for me)
There was a problem hiding this comment.
Hmmm. I meant if people didn't like our runner they could write their own, not that they'd send it to us. Maybe I was opening unwanted doors here... just wanted to make clear that unlike many test frameworks, ours rise only to the level of a single test and the separation was apparently, once, intentional.
There was a problem hiding this comment.
Yeah.. Let people know they can do this, and chance are someone who doesn't really understand it will do it and then burn a lot of support cycles..
If they're knowledgable enough to be able to do this correctly, they'll figure it out... others have.
I'd remove it lest we add another entry to pandora's box..
testing/framework/test-framework.rst
Outdated
|
|
||
| Test infrastructure | ||
| =================== | ||
|
|
||
| The main test API is defined in the ``TestSCons`` class. ``TestSCons`` | ||
| The main e2e test API is defined in the ``TestSCons`` class. ``TestSCons`` |
There was a problem hiding this comment.
I don't mind using "system test" as the term, but this doc, and I think many elsewhere, have tended to use end to end, and I'm being nice and going with the existing convention? ...
There was a problem hiding this comment.
sure "end to end" vs e2e works..
|
I didn't add that (at least not in this edit)... but we can certainly take it out. |
which comment? |
|
Hmmm, thought I'd clicked Quote Reply.
that one. |
No longer suggest tests should move from unittests to e2e. Signed-off-by: Mats Wichmann <mats@linux.com>
Signed-off-by: Mats Wichmann <mats@linux.com>
This is a change to docs only, but these don't appear in the regular documentation. It's duplicated somewhere on the website (main or wiki) but pulling it there will be a separate operation, so for now this only touches the file in
testing/framework.Contributor Checklist:
CHANGES.txt(and read theREADME.rst)