Release 1.18

• Add result done that tools can output if the standard results true/false/unknown
  are not applicable (for example because no property was checked),
  and the run completed successfully.
• In container mode, --keep-system-config is no longer necessary if overlayfs
  is not used for /etc, and thus it is is no longer automatically implied in such cases.
• Benchmark definitions support a new attribute displayName with a human-readable name
  that will be shown in tables.
• A new variable ${taskdef_name} can now be used in places where variable substitution is supported.
• Table-generator supports % as unit for numerical values.
• Some improvements for score handling outside of SV-COMP (i.e., if scores are not calculated by BenchExec).
• New tool-info modules for Test-Comp'19
• Several small bug fixes and improvements

Release 1.17

• Tasks can now be defined in a YAML-based format, cf. the documentation.
  This supports tasks with several input files, and allows providing metadata such as expected verdicts
  in a structured format instead of encoded in the file name.
  The format will be extended to handle more information in the future.
• The wall-time limit can now be specified separately from the CPU-time limit for benchexec as command-line parameter or in the benchmark definition.
• Support for SV-COMP'19 property memcleanup.
• In containers, properly handle /run/systemd/resolve, which is necessary for DNS resolution on systems with systemd-resolved.
• Avoid warnings for mountpoints below inaccessible directories in containers.
• Improvements for handling NaN and Inf values in table-generator.
• Log output of BenchExec will now have colors if coloredlogs is installed.
• New tool-info modules and updates for SV-COMP'19.

Release 1.16

• Support for energy measurements if cpu-energy-meter is installed.
• Several small bug fixes and improvements

Release 1.14

• Updated tool-info modules for all participants of SV-COMP'18.
• Extended support for variable replacements in table-definitions of table-generator.

Release 1.13

• For Debian/Ubuntu, the .deb package is now the recommended way of installation,
  because it automatically configures cgroups as necessary.
• BenchExec now automatically attempts to use the sub-cgroup system.slice/benchexec-cgroup.service
  if it does not have access to the current cgroup.
  This means that if you followed our installation instructions
  for systems with systemd, there is no need anymore to manually
  put your shell into the correct cgroup.
• Several smaller bug fixes for table-generator:
  #249, #250, #259, #260, #271, #272
• For users of the Python API of RunExecutor,
  different file names can now be specified for stdout and stderr of the tool.
• Some new tool-info modules and updates for SV-COMP'18.

Release 1.12

• Fix execution of runs specified with <withoutfile> tags
  in the benchmark definition: the name of the run was missing
  from the command-line in BenchExec 1.11.

Release 1.11

• table-generator can now be given result XML files as arguments in addition to a table-definition XML file (with parameter -x).
  In this case, it will use the column definitions from the latter for tables with the separately given results.
• The directory contrib of the repository now contains a script statistics-tex.py, which can export summary data for benchmark results (e.g., number of solved tasks, average CPU time, etc.) to LaTeX.
• The dummy tools true and false, which could be used for testing a BenchExec installation, are replaced with a more generic dummy tool called dummy.
• A few minor bug fixes and performance optimizations.

A new paper about BenchExec called Reliable Benchmarking: Requirements and Solutions is now available.

Please note that support for Python 3.2 and 3.3 is deprecated.
Furthermore, the support for "sudo mode" (parameter --user/--users) is also deprecated.
All deprecated features will be removed in BenchExec 2.0.

Release 1.10

This release brings several smaller and medium-sized features:

• Tool-info modules for all participants of SV-COMP'17,
  and support for results of the category correct-unconfirmed,
  which is used by SV-COMP if witness validation was not successful.
  To conform with SV-COMP's definitions, violations of the SV-COMP reachability property unreach-call
  will now be reported as false(unreach-call) instead of false(reach).
• Measurement of block I/O if the blkio cgroup controller is available
  (experimental, please read the documentation!).
• Measurement of the energy used by the CPU for a run,
  if the tool cpu-energy-meter is installed on the system
  (experimental, please read the documentation!).
• Ability to limit the disk space a tool can occupy in container mode.
• Various minor improvements to make container mode more robust.
• The feature for executing benchmarks under different user accounts with sudo
  is now marked as deprecated and may be removed in the future,
  consider using the container mode instead for isolating runs
  (cf. issue #215).
• table-generator is now more flexible:
  • Builtin support for certain unit conversions,
    such that the scale factor does not always need to be explicitly specified.
    Furthermore, unit conversions now work even if the values already have a unit.
  • Column titles can be manually specified with the displayTitle attribute`.
  • What columns are relevant for the "diff" table can be configured.

Please also note that we are considering dropping the support for Python 3.2
and maybe 3.3 in BenchExec 2.0 (to be released in a few weeks).
If this is a problem for you, please tell us in issue #207.

Release 1.9

The main feature of this release is the addition of a container mode that allows to isolate runs from each other and from the host, for example preventing filesystem and network accesses. It also allows to collect and store all files created by the tool in a run. The container mode is still in beta and disabled by default for now, it will be enabled by default in BenchExec 2.0. Please try it out and tell us your experiences!

Further changes:

• table-generator now supports HTTP(S) URLs to be given for result XML files
  to allow generating tables for results without needing to download them first.
  The HTML tables will contain correct links to the log files.
• New SV-COMP property deadlock supported by benchexec.
• The parameters --rundefinition and --tasks of benchexec now support wildcards.
• Rounding of very small and very large values in table-generator has been fixed.
• The default font for HTML tables has changed,
  it is now a font that supports correctly aligned digits.

Release 1.8

• benchexec now compresses results by default: XML result files
  are compressed with BZip2, and log files are stored within a ZIP archive.
  This can reduce the necessary disk space significantly
  (typically these logs compress very well),
  and for large benchmark sets it reduces the number of necessary files,
  which can make dealing with the results much faster.
  The previous behavior can be restored with the parameter --no-compress-results.
• table-generator now supports benchmark results where the log files
  are stored in a ZIP file instead of a regular directory.
  All features continue to work with compressed results,
  including extraction of values from log files and viewing log files from HTML tables
  (cf. table-generator documentation for more details).
  Compressed and uncompressed results are handled transparently and can be mixed,
  and using results that were manually compressed or decompressed
  is also supported.