Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Clarify status and message in test runner spec #175

Merged

Commits on Jul 7, 2021

  1. Clarify top-level fail in test runner spec

    Consider the situation of:
    - The user submits a solution in a compiled language
    - The user's solution compiles without error
    - The first test passes, but the next test "produces an error" (for
      example, it raises an exception).
    
    In this case, the correct top-level status is `fail`, with one
    test-level status of `pass`, and one test-level status of `error`. But
    the previous wording could imply that a top-level status of `fail`
    is only correct if at least one test has the status `fail`.
    ee7 committed Jul 7, 2021
    Configuration menu
    Copy the full SHA
    2af18a6 View commit details
    Browse the repository at this point in the history
  2. Clarify per-test message in test runner spec

    The previous wording could imply that a per-test `message` property
    can only be present when the test `status` is `fail`.
    ee7 committed Jul 7, 2021
    Configuration menu
    Copy the full SHA
    8b5174a View commit details
    Browse the repository at this point in the history
  3. Clarify top-level error in test runner spec

    To me, the previous wording implied that a top-level `status` of `error`
    is valid when at least one test "errors". But actually, if at least one
    test was "executed", the top-level status should be `pass` or `fail`.
    ee7 committed Jul 7, 2021
    Configuration menu
    Copy the full SHA
    920131c View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    718393f View commit details
    Browse the repository at this point in the history

Commits on Jul 22, 2021

  1. Further clarify test-level message

    Co-authored-by: Erik Schierboom <erik_schierboom@hotmail.com>
    ee7 and ErikSchierboom authored Jul 22, 2021
    Configuration menu
    Copy the full SHA
    5daab45 View commit details
    Browse the repository at this point in the history
  2. Describe top-level error more consistently

    Co-authored-by: Erik Schierboom <erik_schierboom@hotmail.com>
    ee7 and ErikSchierboom authored Jul 22, 2021
    Configuration menu
    Copy the full SHA
    e8cb9c5 View commit details
    Browse the repository at this point in the history
  3. Update building/tooling/test-runners/interface.md

    Co-authored-by: ee7 <45465154+ee7@users.noreply.github.com>
    ErikSchierboom and ee7 authored Jul 22, 2021
    Configuration menu
    Copy the full SHA
    60e3eda View commit details
    Browse the repository at this point in the history