TEST_SUITE

Tim Test Suite
==============

(c) Copyright 2013 by Neels Janosch Hofmeyr <neels@hofmeyr.de> (GPLv3)

This file is part of Tim project timer.

Tim project timer is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.

Tim project timer is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

You should have received a copy of the GNU General Public License
along with Tim project timer.  If not, see <http://www.gnu.org/licenses/>.


Test Paths
==========

  tim/tests            test suite parent folder
  tim/tests/scripts/*  all test scripts
  tim/tests/test       the executable script that runs the test scripts


Testing
=======

  cd tim/tests
  ./test         # run all available tests
  ./test basic   # run only the test script 'scripts/basic'
  ls scripts/    # list all available test names


Writing Tests
=============

A test script is not a shell script. It is a plain text file that has shell
commands alternated with expected output lines. However, since the expected
output lines start with a '#', it is possible to run a test script in a shell.

A test script contains:

tim new foo
  shell script lines, usually tim commands. 'tim' at a start of a line is
  replaced with the path to the real tim script, i.e. '.../src/tim'. Any shell
  commands are available.

#O output
#E error
  expected-output lines. These start either with '#O ' or '#E ' (with a
  space). #O matches stdout and #E matches stderr. Each shell script that
  generates output must be followed by one or more expected-output lines.

-tim new foo
  unchecked shell script commands. If you start a line with a dash, the shell
  script command after the dash is run, but the output is ignored. The command
  may only produce stdout output. Stderr output will cause the test to fail.

# usual comment
  comment lines. These start with a '#' and are ignored but kept in the file.

FOO
  directives. These are keywords telling the test suite to handle the output
  in certain ways. See below.


  empty lines. Any blank line is ignored and kept in the file.

It makes most sense to simply write down the shell script lines to be tested
and then let the test suite fill in the actual output of the test run (see
Maintaining Tests below). Change the expected output as needed, later.


Example
=======

tim new foo
#O added foo
tim new invalid,name
#E *** Invalid name: 'invalid,name'
tim log
#O ...


Output Matching
===============

Example      What it matches
---------------------------
#O output    stdout = "output"
#E errmsg    stderr = "errmsg"; Any '#O ' matching is also available for '#E '.
#O ...       any number of output lines, up to the end of the output.
#O ^reg.*ex  '#O ^' starts a regular expression to match. The caret '^' is
             mandatory and is also part of the regular expression itself. A '$'
             is inherently appended, to match the end of the line.
#O a DATE b  the keyword 'DATE' matches a date of the format
             'YYYY-MM-DD hh:mm'.


Keyword Directives
==================

A keyword directive must stand alone on a line.

WARP TO YYYY-MM-DD
WARP TO YYYY-MM-DD hh:mm
  artificially set the current time to the given date (and time). Without a
  time, 0:00 is used. 'tim' will internally use this date-time whenever it
  asks for the system's current time (except for the timing daemon, which
  would break if the time stayed fixed). Of course, other programs and the
  system clock are entirely unaffected by this option.
WARP TO PRESENT
  remove any previous time warps.

MANGLE DATES
  switch on date replacement with the DATE keyword. Only useful so that when
  updating a test (-u option), the expected output is kept with 'DATE'
  placeholders in it.  Note: With the 'WARP TO' keyword, date mangling becomes
  pretty much obsolete.
VERBATIM DATES
  return to the default of keeping dates as-are in the program output.


Maintaining Tests
=================

The -u option runs a test, records the actual real life output generated by
the shell script lines and writes that output back into the test script,
ideally so that the test will pass the current output.

Any expected output that is already written down in the test script is only
changed when it doesn't match the actual output. This way, regular expressions
and other special matching are kept (as long as they still match).