ReFrame 4.7.0

Release Notes

New features

• Make directory prefix for topology files configurable (#3317)

• Allow partition/environment extras to be used also as feature constraints in valid_systems and valid_prog_environs (#3316)

• Show the last N lines in test failures (#3291)

• New configuration parameter for specifying commands to install reframe remotely for topology detection (#3281)

• Add a check_fail_phase log formatting specifier (#3292)

• Allow users define custom log handlers and attach them to the framework (#3274)

• Add new --retries-threshold option (#3262)

• Introduce permanent result storage and support performance results comparisons between runs (#3227)

  • Add a configuration option to enable/disable results storage (#3295)
  • Support session annotations with a new --session-extras option (#3266)
  • Expose SQLite connection timeout as configuration parameter (#3253)
  • Disable results storage by default (#3313)
  • Update --table-format arguments (#3307)
  • Use builtin json package for decoding testcase data (#3296)
  • Various enhancements to the results DB feature (#3283)
  • Allow setting the file mode of the SQLite database (#3260)
  • Change default values for --performance-report and --list-stored-sessions options (#3258)
  • Fix query error when aggregation groups and presentation columns are disjoint (#3294)
  • Fix DST skew in CMPSPEC period calculations (#3288)
  • Optimize testcase queries over time period (#3256)
  • Ignore records with unset performance when grouping test cases (#3247)

• Add a perflog entry also when sanity fails (#3189)

• Skip by default flexible tests when not enough nodes found (#3173)

For the new feature of permanent results storage and performance analytics, please have a look at the tutorial and the manpage for details.

Enhancements

• Report the undefined parameters if an instantiation of an abstract test or fixture is attempted (#3293)
• Warn instead of failing when a filter expression is invalid for test (#3310)
• Issue a more compact message when JSON schema validation fails (#3314)
• Do not issue warning when skipping tests during the init phase (#3286)
• Reorganize the structure of the manpage (#3267)
• Improve performance of module backend when detecting module conflicts (#3263)
• Improve naming of automatically generated tests (#3265)

Bug fixes

• Add job options from the command-line to processor auto-detection remote job (#3318)
• Handle None cpu model name returned from archspec (#3199)
• Constraint the lxml version on aarch64 (#3193)

Test library

• Fix compilation of gpu burn benchmark with hip (#3232)
• Added MetalWalls scalability test for mw (#3233)

Other

• Enable Python 3.13 in CI + expand Github test workflows (#3309)
• Use local copies of get-pip.py for bootstrapping (#3287)
• Run scheduler unittests with GitHub Actions (#3268)
• Do not upload to Codecov through a GH action (#3188)

ReFrame 4.6.4

Release Notes

Enhancements

• Print test file in the failure info (#3282)

Bug fixes

• Allow RESERVED nodes for Slurm "avail" state (#3290)
• Let Slurm decide partition in flexible node allocation if no default partition is defined (#3280)

ReFrame 4.6.3

Release Notes

Enhancements

• Add information on how to cite ReFrame (#3259)
• Print selected system in [ReFrame Setup] information (#3255)
• Update disclaimer on the support of EOL Python versions (#3238)

Bug fixes

• Fix typo in the --distribute docs (#3239)

ReFrame 4.6.2

Release Notes

Enhancements

• Update description of the configuration in the manpage (#3213)
• Improve description of valid_systems and valid_prog_environs in the tutorial and other minor tutorial fixes (#3208)

Bug fixes

• Fix report schema to properly accept all test results (#3223)
• Fix logging crash when timestamp=True for the file log handler (#3215)

Other

• Fix Tmod32 container due to Centos 7 EOL (#3229)
• Update get-pip.py url for Python 3.7 (#3222)

ReFrame 4.6.1

Release Notes

Enhancements

• Fix broken cross links in the documentation (#3196)
• Display Python imports in the first tutorial example (#3195)
• Use long options in places in the documentation (#3197)
• Add EUM24 slides (#3191)

Bug fixes

• Do not use CLI defaults for options that are backed by a configuration option (#3206)
• Make version hash suffix calculation more robust (#3204)
• Fix perflog header formatting when perflog_compat is used (#3201)
• Fix logging crash when log format is %(check_#ALL)s (#3202)
• Retrieve the test job's nodelist when using the squeue backend (#3184)

Other

• Bump version of requests package (#3205)

ReFrame 4.6.0

Release Notes

New features

• Add support for associating scheduler resources with an environment (#3152)
• Do not require a sanity function for compile-only tests (#3157)
• Include hardware platform info in ProcessorInfo (#3151)
• Make processor model available through ProcessorInfo (#3150)
• Improve state selection criteria in the --distribute and the --flex-alloc-nodes option (#3140)
• Add a model property to the devices configuration (#3138)
• Enable default value type checking and implicit conversions at the variable declaration level (#3115)
• Introduce launcher modifiers replacing LauncherWrapper (#3114)
• Enable multiple inheritance of test variables (#3113)
• Add CPU vendor and model name to topology files (#3107)
• Support relative imports in tests (#3110)
• Include scheduler and launcher as default partition extras (#3108)
• Add nvcc as an environment configuration parameter (#3098)

Enhancements

• Revamp the documentation (#3095)
• Update Job.nodelist description (#3155)

Bug fixes

• Treat OR Slurm constraints in flexible node allocation (#3158)
• Hide disabled hooks from --describe option's output (#3153)

Test library

• Added QuantumESPRESSO scalability test for pw.x (#3134)

Other

• Update versions of GitHub Actions in CI (#3154)
• Stricter install requirements in setup.cfg (#3135)

ReFrame 4.5.2

Release Notes

Enhancements

• Fix spelling mistake in the documentation (#3133)
• Fix missing JQuery dependency in documentation's conf.py (#3121)
• Fix broken link to older versions of the documentation (#3117)

Bug fixes

• Fix logging crash when the check_#ALL placeholder is used (#3142)
• Do not fail if squeue cannot retrieve the reason that a job is pending (#3123)

ReFrame 4.5.1

Release Notes

Bug fixes

• Better version check for Slurm in the srun backend (#3109)
• Do not ignore the use_login_shell configuration option when generation the auto-detection script (#3094)
• Detect missing curl in bootstrap.sh (#3096)
• Take into account the resolve_module_conflicts option from the configuration (#3093)

Other

• Update the copyright year (#3097)

ReFrame 4.5.0

Release Notes

New features

n/a

Enhancements

• Expose job submit time as a loggable property (#3084)
• Allow always_last to be defined in multiple hooks and add a note about users' workaround code for enforcing hook execution order (#3083)
• Make variables and parameters loggable by default (#3082)
• Document the limitations of the test generation options (#3080)
• Issue an error when invoking an invalid mode (#3070)
• Skip modules system sanity checking when module conflict resolution is off (#3054)
• Update doc dependencies (#3026)

Bug fixes

• Fix crash when --distribute option is applied to a run-only test (#3081)

Test library

• Add fs mount options check to hpctstlib (#2958)

Other

• Use again latest Flux in CI (#3043)
• Make bootstrap.sh aware of multiple architectures and fetch always a fresh pip (#3041)
• Redefine CI base images (#3042)
• Enable Python 3.12 in CI (#3033)
• Validate docs everywhere (#3020)

ReFrame 4.4.2

Release Notes

Enhancements

• Remove use of deprecated %Z directive in strftime() (#3069)
• Properly quote reframe command in output (#3071)
• Remove outdated text in configuration docs (#3068)
• Fix typo in docs about sanity patterns debugging (#3051)
• Move webinars and tutorials under the same section (#3047)

Bug fixes

• Do not pass the stdout/stderr options to srun when using the srunalloc launcher (#3058)
• Fix behaviour of set_var_default() when setting a variable after test initialisation (#3050)