diff --git a/check/README.md b/check/README.md index 797e2037..a9a36c65 100644 --- a/check/README.md +++ b/check/README.md @@ -7,9 +7,9 @@ check the results. ## Prerequisites The test runner script is written in [Ruby](https://www.ruby-lang.org/) -and requires Ruby 2.0 or later. The script uses the so-called `test/unit` -library. In some Linux distributions the library is installed together with -Ruby, while some distributions may have the library as an optional package, +and requires Ruby 2.0 or later. The script uses the library commonly referred to as +`test/unit`. In some Linux distributions, it is installed together with +Ruby, while some distributions may have it as an optional package, or one may need to manually install [test-unit](http://test-unit.github.io/test-unit/en/) via the `gem` command. @@ -37,7 +37,7 @@ Alternatively, one can run the test runner script directly: ./check.rb ``` -By default, this tests `form` found in `$PATH`. +By default, this runs tests with the `form` executable found in `$PATH`. To test another executable, specify its path as a command-line argument: ```bash @@ -50,7 +50,7 @@ option). By default, all test cases in all FORM files (`*.frm`) found in the `check` directory (not in subdirectories) are used. To select test cases or FORM files -to be run, specify their names as command-line arguments. For example: +to run, specify their names as command-line arguments. For example: ```bash ./check.rb Issue8 @@ -64,7 +64,7 @@ For more advanced options, refer to the help message using the `--help` option. ### Where to add test cases? -Currently, the standard test set (run by default) consists of 4 files: +Currently, the standard test set (run by default) includes: - `examples.frm`: Examples provided in the manual. - `features.frm`: Test cases for newly added features. @@ -73,11 +73,11 @@ Currently, the standard test set (run by default) consists of 4 files: Each test case in these files should finish in a short time: the timeout is set to 10 seconds. Bigger tests that take more time are put in subdirectories -(e.g., `forcer`) and should be specified by command-line options when the test +(e.g., `extra`) and should be specified by command-line options when the test suite is invoked: ```bash -./check.rb -C forcer # The Forcer library must be available in FORMPATH. +./check.rb -C extra # Extra library files must be available in FORMPATH. ``` ### Structure of a test case @@ -171,8 +171,31 @@ assert result("F") =~ expr("1 + 2*x + x^2") In this example, `#require unix?` ensures that the test runs only on Unix, where `#pipe` is expected to work. +### Available environment variables + +The following environment variables are accessible in FORM test cases via preprocessor variables. + +- `FORM` + Path to the currently used FORM executable, possibly with additional command-line options. + Example: `/home/form-dev/form/build/sources/tvorm -w4` +- `TESTFILE` + Path to the FORM test file. + Example: `/home/form-dev/form/check/examples.frm` +- `TESTFILEDIR` + Path to the directory containing the FORM test file. + This path is prepended to the `FORMPATH` environment variable. + Example: `/home/form-dev/form/check` +- `TESTCASE` + Name of the current test case. + Example: `Var_Symbols_1` +- `TESTTMPDIR` + Path to the temporary directory used as the current working directory. + Example: `/tmp/form_check_20251121-426943-de5rmj/Test_Var_Symbols_1_20251121-426943-cczq26` + ### Available methods +The following methods are available in Ruby test programs. + #### Execution configuration - `timeout → integer or float` @@ -274,6 +297,8 @@ The following methods assume the default format for statistics: ### Available instructions +`check.rb` recognises the following instructions. + - `#require ` Ensures that the test is executed only if the specified `` is met. - `#pend_if ` diff --git a/check/check.rb b/check/check.rb index 99928880..000f886a 100755 --- a/check/check.rb +++ b/check/check.rb @@ -435,9 +435,15 @@ def do_test(&block) prepare @stdout = "" @stderr = "" + old_formpath = ENV.fetch("FORMPATH", nil) begin + ENV["FORM"] = FormTest.cfg.form_cmd + ENV["FORMPATH"] = add_path(old_formpath, File.dirname(info.full_filename)) + ENV["TESTFILE"] = info.full_filename + ENV["TESTFILEDIR"] = File.dirname(info.full_filename) + ENV["TESTCASE"] = info.classname + ENV["TESTTMPDIR"] = @tmpdir nfiles.times do |i| - ENV["FORM"] = FormTest.cfg.form_cmd @filename = "#{i + 1}.frm" execute("#{ulimits}#{FormTest.cfg.form_cmd} #{@filename}") if !finished? @@ -502,6 +508,8 @@ def do_test(&block) $stderr.puts end info.status = "OK" + ensure + ENV["FORMPATH"] = old_formpath end break end @@ -666,7 +674,7 @@ def result(exprname, index = -1) # The number of terms in the given expression. # Must be in the default statistics format. def nterms(exprname, index = -1) - matches = @stdout.scan(/^[ \t]+#{exprname}\s*Terms in output\s*=\s*(\d+)\s*Bytes used\s*=\s*\d+/m) + matches = @stdout.scan(/^[ \t]+#{exprname}\s*Terms in output\s*=\s*(\d+).*?Bytes used\s*=\s*\d+/m) return matches[index].first.to_i if !matches.empty? && !matches[index].nil? -1 @@ -675,7 +683,7 @@ def nterms(exprname, index = -1) # The size in byte. # Must be in the default statistics format. def bytesize(exprname, index = -1) - matches = @stdout.scan(/^[ \t]+#{exprname}\s*Terms in output\s*=\s*\d+\s*Bytes used\s*=\s*(\d+)/m) + matches = @stdout.scan(/^[ \t]+#{exprname}\s*Terms in output\s*=\s*\d+.*?Bytes used\s*=\s*(\d+)/m) return matches[index].first.to_i if !matches.empty? && !matches[index].nil? -1 @@ -841,6 +849,7 @@ def expr(str) class TestInfo def initialize @classname = nil + @full_filename = nil @where = nil # where the test is defined @foldname = nil # fold name of the test @enabled = nil # enabled or not @@ -851,7 +860,7 @@ def initialize @times = nil # elapsed time (array) end - attr_accessor :classname, :where, :foldname, :enabled, :sources, :time_dilation, + attr_accessor :classname, :full_filename, :where, :foldname, :enabled, :sources, :time_dilation, :status, :times # Return the description of the test. @@ -888,8 +897,24 @@ def classes_info_list end # Convert a .frm file to a .rb file and load it. - def make_ruby_file(filename) + def make_ruby_file(filename, fold_markers = nil) + # Handle fold markers. + if fold_markers.nil? + fold_open_pattern = /^\*..#\[/ + fold_open_with_name_pattern = /^\*..#\[\s*([^:]*)/ + fold_close_pattern = /^\*..#\]/ + fold_close_with_name_pattern = /^\*..#\]\s*([^:]*)/ + else + fold_open_marker = Regexp.escape(fold_markers[:open]) + fold_close_marker = Regexp.escape(fold_markers[:close]) + fold_open_pattern = /^#{fold_open_marker}/ + fold_open_with_name_pattern = /^#{fold_open_marker}\s*([^:]*)/ + fold_close_pattern = /^#{fold_close_marker}/ + fold_close_with_name_pattern = /^#{fold_close_marker}\s*([^:]*)/ + end + # Check existing files. + full_filename = File.expand_path(filename) inname = File.basename(filename) outname = "#{File.basename(filename, '.frm')}.rb" if @files.include?(outname) @@ -921,7 +946,7 @@ def make_ruby_file(filename) lineno += 1 if level == 0 case line - when /^\*..#\[\s*([^:]*)/ + when fold_open_with_name_pattern # fold open: start a class fold = $1.strip if fold.empty? @@ -932,6 +957,7 @@ def make_ruby_file(filename) @classes.push(classname) @classes_info["Test_#{classname}"] = info info.classname = classname + info.full_filename = full_filename info.where = "#{inname}:#{lineno}" info.foldname = fold info.enabled = test_enabled?(classname) @@ -952,14 +978,14 @@ def make_ruby_file(filename) else line = "class Test_#{classname} < Test::Unit::TestCase; include FormTest" end - when /^\*..#\]/ + when fold_close_pattern # unexpected fold close fatal("unexpected fold close", inname, lineno) else # as commentary line = "" end - elsif heredoc.nil? && line =~ /^\*..#\]\s*([^:]*)/ && level == 1 + elsif heredoc.nil? && line =~ fold_close_with_name_pattern && level == 1 # fold close: end of the class fold = $1.strip foldname = info.foldname @@ -1076,10 +1102,10 @@ def make_ruby_file(filename) line = "" else if heredoc.nil? - if line =~ /^\*..#\[/ + if line =~ fold_open_pattern # fold open level += 1 - elsif line =~ /^\*..#\]\s*([^:]*)/ + elsif line =~ fold_close_with_name_pattern # fold close level -= 1 elsif line =~ /< len - str[0..len - 1] + str[0..(len - 1)] elsif str.length < len str + " " * (len - str.length) else