Any value is accepted. No conversion is done.
+ DataType Standard Should Be 1 + ... boolean + ...Strings TRUE,
+ DataType Standard Should Be 6
+ ... Literal
+ ...
Only specified values are accepted. + +Standard with generics + DataType Standard Should Be 2 + ... dictionary + ...
Strings must be Python Strings must be Python Defines the geolocation.
- ...latitude Latitude between -90 and 90.longitude Longitude between -180 and 180.accuracy Optional Non-negative accuracy value. Defaults to 0.Example usage: {'latitude': 59.95, 'longitude': 30.31667}
Defines the geolocation.
+ ${MODEL}[typedocs][7][type] TypedDict + ${MODEL}[typedocs][7][name] GeoLocation + ${MODEL}[typedocs][7][doc]Defines the geolocation.
...latitude Latitude between -90 and 90.longitude Longitude between -180 and 180.This is some Doc
- ...This has was defined by assigning to __doc__.
- ${MODEL}[typedocs][0][type] Enum - ${MODEL}[typedocs][0][name] AssertionOperator - ${MODEL}[typedocs][0][doc]This is some Doc
+ ${MODEL}[typedocs][1][type] Enum + ${MODEL}[typedocs][1][name] AssertionOperator + ${MODEL}[typedocs][1][doc]This is some Doc
...This has was defined by assigning to __doc__.
Enum Members [Template] NONE ${exp_list} Evaluate [{"name": "equal","value": "=="},{"name": "==","value": "=="},{"name": "<","value": "<"},{"name": ">","value": ">"},{"name": "<=","value": "<="},{"name": ">=","value": ">="}] - FOR ${cur} ${exp} IN ZIP ${MODEL}[dataTypes][enums][0][members] ${exp_list} - Dictionaries Should Be Equal ${cur} ${exp} - END - FOR ${cur} ${exp} IN ZIP ${MODEL}[typedocs][0][members] ${exp_list} + FOR ${cur} ${exp} IN ZIP ${MODEL}[typedocs][1][members] ${exp_list} Dictionaries Should Be Equal ${cur} ${exp} END Custom types - ${MODEL}[typedocs][2][type] Custom - ${MODEL}[typedocs][2][name] CustomType - ${MODEL}[typedocs][2][doc]Converter method doc is used when defined.
${MODEL}[typedocs][3][type] Custom - ${MODEL}[typedocs][3][name] CustomType2 - ${MODEL}[typedocs][3][doc]Class doc is used when converter method has no doc.
+ ${MODEL}[typedocs][3][name] CustomType + ${MODEL}[typedocs][3][doc]Converter method doc is used when defined.
+ ${MODEL}[typedocs][4][type] Custom + ${MODEL}[typedocs][4][name] CustomType2 + ${MODEL}[typedocs][4][doc]Class doc is used when converter method has no doc.
Standard types - ${MODEL}[typedocs][1][type] Standard - ${MODEL}[typedocs][1][name] boolean - ${MODEL}[typedocs][1][doc]Strings TRUE, YES, start=True
+ ${MODEL}[typedocs][0][type] Standard
+ ${MODEL}[typedocs][0][name] Any
+ ${MODEL}[typedocs][0][doc]
Any value is accepted. No conversion is done.
+ ${MODEL}[typedocs][2][type] Standard + ${MODEL}[typedocs][2][name] boolean + ${MODEL}[typedocs][2][doc]Strings TRUE, YES, start=True
+ ${MODEL}[typedocs][10][name] Literal
+ ${MODEL}[typedocs][10][doc]
Only specified values are accepted. start=True + +Standard types with generics + ${MODEL}[typedocs][5][type] Standard + ${MODEL}[typedocs][5][name] dictionary + ${MODEL}[typedocs][5][doc]
Strings must be Python Strings must be Python Defines the geolocation.
- ...latitude Latitude between -90 and 90.longitude Longitude between -180 and 180.accuracy Optional Non-negative accuracy value. Defaults to 0.Example usage: {'latitude': 59.95, 'longitude': 30.31667}
Defines the geolocation.
+ ${MODEL}[typedocs][7][type] TypedDict + ${MODEL}[typedocs][7][name] GeoLocation + ${MODEL}[typedocs][7][doc]Defines the geolocation.
...latitude Latitude between -90 and 90.longitude Longitude between -180 and 180.This is some Doc
- ...This has was defined by assigning to __doc__.
- ${MODEL}[typedocs][0][type] Enum - ${MODEL}[typedocs][0][name] AssertionOperator - ${MODEL}[typedocs][0][doc]This is some Doc
+ ${MODEL}[typedocs][1][type] Enum + ${MODEL}[typedocs][1][name] AssertionOperator + ${MODEL}[typedocs][1][doc]This is some Doc
...This has was defined by assigning to __doc__.
Enum Members [Template] NONE ${exp_list} Evaluate [{"name": "equal","value": "=="},{"name": "==","value": "=="},{"name": "<","value": "<"},{"name": ">","value": ">"},{"name": "<=","value": "<="},{"name": ">=","value": ">="}] - FOR ${cur} ${exp} IN ZIP ${MODEL}[dataTypes][enums][0][members] ${exp_list} - Dictionaries Should Be Equal ${cur} ${exp} - END - FOR ${cur} ${exp} IN ZIP ${MODEL}[typedocs][0][members] ${exp_list} + FOR ${cur} ${exp} IN ZIP ${MODEL}[typedocs][1][members] ${exp_list} Dictionaries Should Be Equal ${cur} ${exp} END Custom types - ${MODEL}[typedocs][2][type] Custom - ${MODEL}[typedocs][2][name] CustomType - ${MODEL}[typedocs][2][doc]Converter method doc is used when defined.
- ${MODEL}[typedocs][3][type] Custom - ${MODEL}[typedocs][3][name] CustomType2 - ${MODEL}[typedocs][3][doc]Class doc is used when converter method has no doc.
+ ${MODEL}[typedocs][3][type] Custom + ${MODEL}[typedocs][3][name] CustomType + ${MODEL}[typedocs][3][doc]Converter method doc is used when defined.
+ ${MODEL}[typedocs][4][type] Custom + ${MODEL}[typedocs][4][name] CustomType2 + ${MODEL}[typedocs][4][doc]Class doc is used when converter method has no doc.
Standard types - ${MODEL}[typedocs][1][type] Standard - ${MODEL}[typedocs][1][name] boolean - ${MODEL}[typedocs][1][doc]Strings TRUE, YES, start=True
+ ${MODEL}[typedocs][0][type] Standard
+ ${MODEL}[typedocs][0][name] Any
+ ${MODEL}[typedocs][0][doc]
Any value is accepted. No conversion is done.
+ ${MODEL}[typedocs][2][type] Standard + ${MODEL}[typedocs][2][name] boolean + ${MODEL}[typedocs][2][doc]Strings TRUE, YES, start=True
+ ${MODEL}[typedocs][10][name] Literal
+ ${MODEL}[typedocs][10][doc]
Only specified values are accepted. start=True + +Standard types with generics + ${MODEL}[typedocs][5][type] Standard + ${MODEL}[typedocs][5][name] dictionary + ${MODEL}[typedocs][5][doc]
Strings must be Python Strings must be Python ${HTML DOC}
HTML --format jSoN --specdocformat hTML DocFormat.py${HTML DOC}
HTML --format jSoN DocFormat.py${HTML DOC}
HTML --docfor RoBoT -f JSON -s HTML DocFormatHtml.py @@ -68,6 +70,7 @@ Format from XML spec Format from JSON RAW spec [Template] NONE + [Tags] require-jsonschema Test Format In JSON ${RAW DOC} ROBOT -F Robot -s RAW lib=DocFormat.py Copy File ${OUTJSON} ${OUTBASE}-2.json Test Format In JSON${HTML DOC}
HTML lib=${OUTBASE}-2.json @@ -80,6 +83,7 @@ Format from LIBSPEC spec Format from JSON spec [Template] NONE + [Tags] require-jsonschema Test Format In JSON${HTML DOC}
HTML -F Robot lib=DocFormat.py Copy File ${OUTJSON} ${OUTBASE}-2.json Test Format In JSON${HTML DOC}
HTML lib=${OUTBASE}-2.json diff --git a/atest/robot/libdoc/dynamic_library.robot b/atest/robot/libdoc/dynamic_library.robot index 6539af01d7e..a3adf492b29 100644 --- a/atest/robot/libdoc/dynamic_library.robot +++ b/atest/robot/libdoc/dynamic_library.robot @@ -90,6 +90,9 @@ Keyword tags from documentation Keyword types Keyword Arguments Should Be 18 integer: int no type boolean: bool = True +Return type + Return Type Should Be 18 int + No keyword source info Keyword Name Should Be 0 0 Keyword Should Not Have Source 0 diff --git a/atest/robot/libdoc/html_output.robot b/atest/robot/libdoc/html_output.robot index f42a4b150b8..d259c49bc7d 100644 --- a/atest/robot/libdoc/html_output.robot +++ b/atest/robot/libdoc/html_output.robot @@ -15,7 +15,7 @@ Version Generated [Template] Should Match Regexp - ${MODEL}[generated] \\d{4}-\\d{2}-\\d{2} \\d{2}:\\d{2}:\\d{2} + ${MODEL}[generated] \\d{4}-\\d{2}-\\d{2}T\\d{2}:\\d{2}:\\d{2}[+-]\\d{2}:\\d{2} Scope ${MODEL}[scope] GLOBAL @@ -33,7 +33,7 @@ Keyword Arguments [Template] Verify Argument Models ${MODEL}[keywords][0][args] ${MODEL}[keywords][1][args] a1=d *a2 - ${MODEL}[keywords][6][args] arg=hyv\\xe4 + ${MODEL}[keywords][6][args] arg=hyvä ${MODEL}[keywords][9][args] arg=hyvä ${MODEL}[keywords][10][args] a=1 b=True c=(1, 2, None) ${MODEL}[keywords][11][args] arg=\\ robot \\ escapers\\n\\t\\r \\ \\ @@ -41,8 +41,18 @@ Keyword Arguments Embedded Arguments [Template] NONE - Should Be Equal ${MODEL}[keywords][13][name] Takes \${embedded} \${args} - Should Be Empty ${MODEL}[keywords][13][args] + Should Be Equal ${MODEL}[keywords][13][name] Takes \${embedded} \${args} + Should Be Empty ${MODEL}[keywords][13][args] + +Embedded and Normal Arguments + [Template] NONE + Should Be Equal ${MODEL}[keywords][14][name] Takes \${embedded} and normal args + Verify Argument Models ${MODEL}[keywords][14][args] mandatory optional=None + +Embedded and Positional-only Arguments + [Template] NONE + Should Be Equal ${MODEL}[keywords][15][name] Takes \${embedded} and positional-only args + Verify Argument Models ${MODEL}[keywords][15][args] mandatory / optional=None Keyword Documentation ${MODEL}[keywords][1][doc] @@ -105,6 +115,13 @@ User keyword documentation formatting ... ... +Private keyword should be excluded + [Setup] Run Libdoc And Parse Model From HTML ${TESTDATADIR}/resource.robot + [Template] None + FOR ${keyword} IN @{MODEL}[keywords] + Should Not Be Equal ${keyword}[name] Private + END + *** Keywords *** Verify Argument Models [Arguments] ${arg_models} @{expected_reprs} diff --git a/atest/robot/libdoc/invalid_library_keywords.robot b/atest/robot/libdoc/invalid_library_keywords.robot index 4f212480158..f923179c717 100644 --- a/atest/robot/libdoc/invalid_library_keywords.robot +++ b/atest/robot/libdoc/invalid_library_keywords.robot @@ -20,7 +20,7 @@ Invalid embedded arguments Keyword Count Should Be 3 Stdout should contain adding keyword error ... Invalid embedded \${args} - ... Embedded argument count does not match number of accepted arguments. + ... Keyword must accept at least as many positional arguments as it has embedded arguments. *** Keywords *** Stdout should contain adding keyword error diff --git a/atest/robot/libdoc/invalid_usage.robot b/atest/robot/libdoc/invalid_usage.robot index 24f1c0b4ec7..51b7daada69 100644 --- a/atest/robot/libdoc/invalid_usage.robot +++ b/atest/robot/libdoc/invalid_usage.robot @@ -2,7 +2,6 @@ Resource libdoc_resource.robot Test Setup Remove File ${OUT HTML} Test Template Run libdoc and verify error -Test Teardown Should Not Exist ${OUT HTML} *** Test Cases *** No arguments @@ -22,6 +21,7 @@ Invalid format --format XML:XXX BuiltIn ${OUT HTML} Format must be 'HTML', 'XML', 'JSON' or 'LIBSPEC', got 'XML:XXX'. --format XML:HTML BuiltIn ${OUT HTML} Format must be 'HTML', 'XML', 'JSON' or 'LIBSPEC', got 'XML:HTML'. BuiltIn out.ext Format must be 'HTML', 'XML', 'JSON' or 'LIBSPEC', got 'EXT'. + BuiltIn BuiltIn Format must be 'HTML', 'XML', 'JSON' or 'LIBSPEC', got ''. Invalid specdocformat -s XXX BuiltIn ${OUT HTML} Spec doc format must be 'RAW' or 'HTML', got 'XXX'. @@ -36,6 +36,10 @@ Invalid doc format Invalid doc format in library ${TESTDATADIR}/DocFormatInvalid.py ${OUT HTML} Invalid documentation format 'INVALID'. +Invalid theme + --theme bad String ${OUT XML} Theme must be 'DARK', 'LIGHT' or 'NONE', got 'BAD'. + --theme light --format xml String ${OUT XML} The --theme option is only applicable with HTML outputs. + Non-existing library NonExistingLib ${OUT HTML} Importing library 'NonExistingLib' failed: * @@ -53,11 +57,17 @@ Non-XML spec [Teardown] Remove File ${OUT XML} Invalid resource - ${CURDIR}/invalid_usage.robot ${OUT HTML} - ... ? ERROR ? Error in file '*' on line 3: Setting 'Test Setup' is not allowed in resource file. - ... ? ERROR ? Error in file '*' on line 4: Setting 'Test Template' is not allowed in resource file. - ... ? ERROR ? Error in file '*' on line 5: Setting 'Test Teardown' is not allowed in resource file. - ... Error in file '*[/\\]invalid_usage.robot' on line 7: Resource file with 'Test Cases' section is invalid. + ${TESTDATADIR}/invalid_resource.resource ${OUT HTML} + ... ? ERROR ? Error in file '*[/\\]invalid_resource.resource' on line 2: Setting 'Metadata' is not allowed in resource file. + ... ? ERROR ? Error in file '*[/\\]invalid_resource.resource' on line 3: Setting 'Test Setup' is not allowed in resource file. + ... Error in file '*[/\\]invalid_resource.resource' on line 5: Resource file with 'Test Cases' section is invalid. + +Invalid resource with '.robot' extension + ${TESTDATADIR}/invalid_resource.robot ${OUT HTML} + ... ? ERROR ? Error in file '*[/\\]invalid_resource.robot' on line 2: Setting 'Metadata' is not allowed in resource file. + ... ? ERROR ? Error in file '*[/\\]invalid_resource.robot' on line 3: Setting 'Test Setup' is not allowed in resource file. + ... ${OUT HTML} + ... fatal=False Invalid output file [Setup] Run Keywords @@ -70,10 +80,16 @@ Invalid output file ... Remove Directory ${OUT HTML} AND ... Remove Directory ${OUT XML} -invalid Spec File version - ${TESTDATADIR}/OldSpec.xml ${OUT XML} Invalid spec file version 'None'. Supported versions are 3 and 4. +Invalid Spec File version + ${TESTDATADIR}/OldSpec.xml ${OUT XML} Invalid spec file version 'None'. Supported versions are 3, 4, 5, and 6. *** Keywords *** Run libdoc and verify error - [Arguments] ${args} @{error} - Run libdoc and verify output ${args} @{error} ${USAGE TIP[1:]} + [Arguments] ${args} @{error} ${fatal}=True + IF ${fatal} + Run Libdoc And Verify Output ${args} @{error} ${USAGE TIP[1:]} + File Should Not Exist ${OUT HTML} + ELSE + Run Libdoc And Verify Output ${args} @{error} + File Should Exist ${OUT HTML} + END diff --git a/atest/robot/libdoc/invalid_user_keywords.robot b/atest/robot/libdoc/invalid_user_keywords.robot index 530cec1840c..9dcec3a8cf7 100644 --- a/atest/robot/libdoc/invalid_user_keywords.robot +++ b/atest/robot/libdoc/invalid_user_keywords.robot @@ -6,16 +6,16 @@ Resource libdoc_resource.robot Invalid arg spec Keyword Name Should Be 0 Invalid arg spec Keyword Doc Should Be 0 *Creating keyword failed:* Invalid argument specification: Only last argument can be kwargs. - Stdout should contain error Invalid arg spec 2 + Stdout should contain error Invalid arg spec 3 ... Invalid argument specification: Only last argument can be kwargs. -Dublicate name - Keyword Name Should Be 3 Same twice +Duplicate name + Keyword Name Should Be 3 Same Twice Keyword Doc Should Be 3 *Creating keyword failed:* Keyword with same name defined multiple times. - Stdout should contain error Same twice 8 + Stdout should contain error Same twice 10 ... Keyword with same name defined multiple times -Dublicate name with embedded arguments +Duplicate name with embedded arguments Keyword Name Should Be 1 same \${embedded match} Keyword Doc Should Be 1 ${EMPTY} Keyword Name Should Be 2 Same \${embedded} diff --git a/atest/robot/libdoc/json_output.robot b/atest/robot/libdoc/json_output.robot index 78305a45851..deec2eb1cf4 100644 --- a/atest/robot/libdoc/json_output.robot +++ b/atest/robot/libdoc/json_output.robot @@ -2,6 +2,7 @@ Resource libdoc_resource.robot Suite Setup Run Libdoc And Parse Model From JSON ${TESTDATADIR}/module.py Test Template Should Be Equal Multiline +Test Tags require-jsonschema *** Test Cases *** Name @@ -15,7 +16,7 @@ Version Generated [Template] Should Match Regexp - ${MODEL}[generated] \\d{4}-\\d{2}-\\d{2} \\d{2}:\\d{2}:\\d{2} + ${MODEL}[generated] \\d{4}-\\d{2}-\\d{2}T\\d{2}:\\d{2}:\\d{2}[+-]\\d{2}:\\d{2} Scope ${MODEL}[scope] GLOBAL @@ -33,7 +34,7 @@ Keyword Arguments [Template] Verify Argument Models ${MODEL}[keywords][0][args] ${MODEL}[keywords][1][args] a1=d *a2 - ${MODEL}[keywords][6][args] arg=hyv\\xe4 + ${MODEL}[keywords][6][args] arg=hyvä ${MODEL}[keywords][9][args] arg=hyvä ${MODEL}[keywords][10][args] a=1 b=True c=(1, 2, None) ${MODEL}[keywords][11][args] arg=\\ robot \\ escapers\\n\\t\\r \\ \\ @@ -41,8 +42,18 @@ Keyword Arguments Embedded Arguments [Template] NONE - Should Be Equal ${MODEL}[keywords][13][name] Takes \${embedded} \${args} - Should Be Empty ${MODEL}[keywords][13][args] + Should Be Equal ${MODEL}[keywords][13][name] Takes \${embedded} \${args} + Should Be Empty ${MODEL}[keywords][13][args] + +Embedded and Normal Arguments + [Template] NONE + Should Be Equal ${MODEL}[keywords][14][name] Takes \${embedded} and normal args + Verify Argument Models ${MODEL}[keywords][14][args] mandatory optional=None + +Embedded and Positional-only Arguments + [Template] NONE + Should Be Equal ${MODEL}[keywords][15][name] Takes \${embedded} and positional-only args + Verify Argument Models ${MODEL}[keywords][15][args] mandatory / optional=None Keyword Documentation ${MODEL}[keywords][1][doc] @@ -105,6 +116,20 @@ User keyword documentation formatting ... ... +Private user keyword should be included + [Setup] Run Libdoc And Parse Model From JSON ${TESTDATADIR}/resource.robot + ${MODEL}[keywords][-1][name] Private + ${MODEL}[keywords][-1][tags] ['robot:private'] + ${MODEL}[keywords][-1][private] True + ${MODEL['keywords'][0].get('private')} None + +Deprecation + [Setup] Run Libdoc And Parse Model From JSON ${TESTDATADIR}/Deprecation.py + ${MODEL}[keywords][0][deprecated] True + ${MODEL}[keywords][1][deprecated] True + ${MODEL['keywords'][2].get('deprecated')} None + ${MODEL['keywords'][3].get('deprecated')} None + *** Keywords *** Verify Argument Models [Arguments] ${arg_models} @{expected_reprs} diff --git a/atest/robot/libdoc/libdoc_resource.robot b/atest/robot/libdoc/libdoc_resource.robot index 351b5e793d3..8f08ec03f6a 100644 --- a/atest/robot/libdoc/libdoc_resource.robot +++ b/atest/robot/libdoc/libdoc_resource.robot @@ -31,8 +31,8 @@ Run Libdoc And Parse Output Run Libdoc And Verify Output [Arguments] ${args} @{expected} + VAR ${expected} @{expected} separator=\n ${output}= Run Libdoc ${args} - ${expected}= Catenate SEPARATOR=\n @{expected} Should Match ${output} ${expected}\n Run Libdoc And Parse Model From HTML @@ -80,7 +80,7 @@ Type Should Be Element Attribute Should Be ${LIBDOC} type ${type} Scope Should Be - [Arguments] ${scope} ${old}=${{ {'GLOBAL': 'global', 'SUITE': 'test suite', 'TEST': 'test case'}[$scope] }} + [Arguments] ${scope} Element Attribute Should Be ${LIBDOC} scope ${scope} Source Should Be @@ -93,10 +93,16 @@ Lineno Should Be Element Attribute Should Be ${LIBDOC} lineno ${lineno} Generated Should Be Defined - Element Attribute Should Match ${LIBDOC} generated ????-??-??T??:??:??Z + # For example, '1970-01-01T00:00:01+00:00'. + Element Attribute Should Match ${LIBDOC} generated ????-??-??T??:??:?????:?? + +Generated Should Be + [Arguments] ${generated} + Generated Should Be Defined + Element Attribute Should Be ${LIBDOC} generated ${generated} Spec version should be correct - Element Attribute Should Be ${LIBDOC} specversion 4 + Element Attribute Should Be ${LIBDOC} specversion 6 Should Have No Init ${inits} = Get Elements ${LIBDOC} xpath=inits/init @@ -132,12 +138,22 @@ Verify Arguments Structure [Arguments] ${index} ${xpath} ${expected} ${kws}= Get Elements ${LIBDOC} xpath=${xpath} ${arg_elems}= Get Elements ${kws}[${index}] xpath=arguments/arg - FOR ${arg_elem} ${exp_repr} IN ZIP ${arg_elems} ${expected} + FOR ${arg_elem} ${exp_repr} IN ZIP ${arg_elems} ${expected} mode=STRICT + IF $INTERPRETER.version_info >= (3, 11) + ${exp_repr} = Replace String ${exp_repr} | None = None = None + END ${kind}= Get Element Attribute ${arg_elem} kind ${required}= Get Element Attribute ${arg_elem} required ${repr}= Get Element Attribute ${arg_elem} repr ${name}= Get Element Optional Text ${arg_elem} name - ${type}= Get Elements Texts ${arg_elem} type + ${types}= Get Elements ${arg_elem} type + IF not $types + ${type}= Set Variable ${None} + ELSE IF len($types) == 1 + ${type}= Get Type ${types}[0] + ELSE + Fail Cannot have more than oneContent flattened.\\x3c/b>\\x3c/i>\\x3c/p>
+${FLATTENED} Content flattened.
${ERROR} [ ERROR ] Invalid value for option '--flattenkeywords': Expected 'FOR', 'WHILE', 'ITERATION', 'TAG: Doc of keyword 3\\x3c/p>\\n${FLAT HTML}
- Should Contain ${LOG} *${FLAT HTML}
- Should Contain ${LOG} * Logs the given message with the given level.\\x3c/p>\\n${FLAT HTML}
+ Should Contain ${LOG} "*Content flattened."
Flatten controls in keyword
${tc} = Check Test Case ${TEST NAME}
- Length Should Be ${tc.body[0].body.filter(messages=False)} 0
- Length Should Be ${tc.body[0].body.filter(messages=True)} 23
- Length Should Be ${tc.body[0].body} 23
+ Check Counts ${tc[0]} 23
@{expected} = Create List
... Outside IF Inside IF 1 Nested IF
... 3 2 1 BANG!
... FOR: 0 1 FOR: 1 1 FOR: 2 1
... WHILE: 2 1 \${i} = 1 WHILE: 1 1 \${i} = 0
- FOR ${msg} ${exp} IN ZIP ${tc.body[0].body} ${expected}
- Check Log Message ${msg} ${exp}
+ ... AssertionError 1 finally
+ FOR ${msg} ${exp} IN ZIP ${tc[0].body} ${expected}
+ Check Log Message ${msg} ${exp} level=IGNORE
END
- Check log message ${tc.body[0].body[20]} AssertionError level=FAIL
- Check log message ${tc.body[0].body[21]} 1
- Check log message ${tc.body[0].body[22]} finally
-Flatten for loops
+Flatten FOR
Run Rebot --flatten For ${OUTFILE COPY}
- ${tc} = Check Test Case For loop
- Should Be Equal ${tc.kws[0].type} FOR
- Should Be Equal ${tc.kws[0].doc} ${FLAT TEXT}
- Length Should Be ${tc.kws[0].kws} 0
- Length Should Be ${tc.kws[0].msgs} 60
+ ${tc} = Check Test Case FOR loop
+ Should Be Equal ${tc[0].type} FOR
+ Should Be Equal ${tc[0].message} *HTML* ${FLATTENED}
+ Check Counts ${tc[0]} 60
FOR ${index} IN RANGE 10
- Check Log Message ${tc.kws[0].msgs[${index * 6 + 0}]} index: ${index}
- Check Log Message ${tc.kws[0].msgs[${index * 6 + 1}]} 3
- Check Log Message ${tc.kws[0].msgs[${index * 6 + 2}]} 2
- Check Log Message ${tc.kws[0].msgs[${index * 6 + 3}]} 1
- Check Log Message ${tc.kws[0].msgs[${index * 6 + 4}]} 2
- Check Log Message ${tc.kws[0].msgs[${index * 6 + 5}]} 1
+ Check Log Message ${tc[0, ${index * 6 + 0}]} index: ${index}
+ Check Log Message ${tc[0, ${index * 6 + 1}]} 3
+ Check Log Message ${tc[0, ${index * 6 + 2}]} 2
+ Check Log Message ${tc[0, ${index * 6 + 3}]} 1
+ Check Log Message ${tc[0, ${index * 6 + 4}]} 2
+ Check Log Message ${tc[0, ${index * 6 + 5}]} 1
END
-Flatten for loop iterations
+Flatten FOR iterations
Run Rebot --flatten ForItem ${OUTFILE COPY}
- ${tc} = Check Test Case For loop
- Should Be Equal ${tc.kws[0].type} FOR
- Should Be Empty ${tc.kws[0].doc}
- Length Should Be ${tc.kws[0].kws} 10
- Should Be Empty ${tc.kws[0].msgs}
+ ${tc} = Check Test Case FOR loop
+ Should Be Equal ${tc[0].type} FOR
+ Should Be Equal ${tc[0].message} ${EMPTY}
+ Check Counts ${tc[0]} 0 10
FOR ${index} IN RANGE 10
- Should Be Equal ${tc.kws[0].kws[${index}].type} ITERATION
- Should Be Equal ${tc.kws[0].kws[${index}].doc} ${FLAT TEXT}
- Should Be Empty ${tc.kws[0].kws[${index}].kws}
- Length Should Be ${tc.kws[0].kws[${index}].msgs} 6
- Check Log Message ${tc.kws[0].kws[${index}].msgs[0]} index: ${index}
- Check Log Message ${tc.kws[0].kws[${index}].msgs[1]} 3
- Check Log Message ${tc.kws[0].kws[${index}].msgs[2]} 2
- Check Log Message ${tc.kws[0].kws[${index}].msgs[3]} 1
- Check Log Message ${tc.kws[0].kws[${index}].msgs[4]} 2
- Check Log Message ${tc.kws[0].kws[${index}].msgs[5]} 1
+ Should Be Equal ${tc[0, ${index}].type} ITERATION
+ Should Be Equal ${tc[0, ${index}].message} *HTML* ${FLATTENED}
+ Check Counts ${tc[0, ${index}]} 6
+ Check Log Message ${tc[0, ${index}, 0]} index: ${index}
+ Check Log Message ${tc[0, ${index}, 1]} 3
+ Check Log Message ${tc[0, ${index}, 2]} 2
+ Check Log Message ${tc[0, ${index}, 3]} 1
+ Check Log Message ${tc[0, ${index}, 4]} 2
+ Check Log Message ${tc[0, ${index}, 5]} 1
END
-Flatten while loops
+Flatten WHILE
Run Rebot --flatten WHile ${OUTFILE COPY}
${tc} = Check Test Case WHILE loop
- Should Be Equal ${tc.body[1].type} WHILE
- Should Be Equal ${tc.body[1].doc} ${FLAT TEXT}
- Length Should Be ${tc.body[1].kws} 0
- Length Should Be ${tc.body[1].msgs} 70
+ Should Be Equal ${tc[1].type} WHILE
+ Should Be Equal ${tc[1].message} *HTML* ${FLATTENED}
+ Check Counts ${tc[1]} 70
FOR ${index} IN RANGE 10
- Check Log Message ${tc.body[1].msgs[${index * 7 + 0}]} index: ${index}
- Check Log Message ${tc.body[1].msgs[${index * 7 + 1}]} 3
- Check Log Message ${tc.body[1].msgs[${index * 7 + 2}]} 2
- Check Log Message ${tc.body[1].msgs[${index * 7 + 3}]} 1
- Check Log Message ${tc.body[1].msgs[${index * 7 + 4}]} 2
- Check Log Message ${tc.body[1].msgs[${index * 7 + 5}]} 1
+ Check Log Message ${tc[1, ${index * 7 + 0}]} index: ${index}
+ Check Log Message ${tc[1, ${index * 7 + 1}]} 3
+ Check Log Message ${tc[1, ${index * 7 + 2}]} 2
+ Check Log Message ${tc[1, ${index * 7 + 3}]} 1
+ Check Log Message ${tc[1, ${index * 7 + 4}]} 2
+ Check Log Message ${tc[1, ${index * 7 + 5}]} 1
${i}= Evaluate $index + 1
- Check Log Message ${tc.body[1].msgs[${index * 7 + 6}]} \${i} = ${i}
+ Check Log Message ${tc[1, ${index * 7 + 6}]} \${i} = ${i}
END
-Flatten while loop iterations
+Flatten WHILE iterations
Run Rebot --flatten iteration ${OUTFILE COPY}
${tc} = Check Test Case WHILE loop
- Should Be Equal ${tc.body[1].type} WHILE
- Should Be Empty ${tc.body[1].doc}
- Length Should Be ${tc.body[1].body} 10
- Should Be Empty ${tc.body[1].msgs}
+ Should Be Equal ${tc[1].type} WHILE
+ Should Be Equal ${tc[1].message} ${EMPTY}
+ Check Counts ${tc[1]} 0 10
FOR ${index} IN RANGE 10
- Should Be Equal ${tc.kws[1].kws[${index}].type} ITERATION
- Should Be Equal ${tc.kws[1].kws[${index}].doc} ${FLAT TEXT}
- Should Be Empty ${tc.kws[1].kws[${index}].kws}
- Length Should Be ${tc.kws[1].kws[${index}].msgs} 7
- Check Log Message ${tc.kws[1].kws[${index}].msgs[0]} index: ${index}
- Check Log Message ${tc.kws[1].kws[${index}].msgs[1]} 3
- Check Log Message ${tc.kws[1].kws[${index}].msgs[2]} 2
- Check Log Message ${tc.kws[1].kws[${index}].msgs[3]} 1
- Check Log Message ${tc.kws[1].kws[${index}].msgs[4]} 2
- Check Log Message ${tc.kws[1].kws[${index}].msgs[5]} 1
+ Should Be Equal ${tc[1, ${index}].type} ITERATION
+ Should Be Equal ${tc[1, ${index}].message} *HTML* ${FLATTENED}
+ Check Counts ${tc[1, ${index}]} 7
+ Check Log Message ${tc[1, ${index}, 0]} index: ${index}
+ Check Log Message ${tc[1, ${index}, 1]} 3
+ Check Log Message ${tc[1, ${index}, 2]} 2
+ Check Log Message ${tc[1, ${index}, 3]} 1
+ Check Log Message ${tc[1, ${index}, 4]} 2
+ Check Log Message ${tc[1, ${index}, 5]} 1
${i}= Evaluate $index + 1
- Check Log Message ${tc.kws[1].kws[${index}].msgs[6]} \${i} = ${i}
+ Check Log Message ${tc[1, ${index}, 6]} \${i} = ${i}
END
Invalid usage
@@ -173,3 +156,8 @@ Run And Rebot Flattened
Run Rebot ${FLATTEN} ${OUTFILE COPY}
${TC} = Check Test Case Flatten stuff
Set Suite Variable $TC
+
+Check Counts
+ [Arguments] ${item} ${messages} ${non_messages}=0
+ Length Should Be ${item.messages} ${messages}
+ Length Should Be ${item.non_messages} ${non_messages}
diff --git a/atest/robot/output/js_model.robot b/atest/robot/output/js_model.robot
index 9c63ece6380..bfdb0d6e3c2 100644
--- a/atest/robot/output/js_model.robot
+++ b/atest/robot/output/js_model.robot
@@ -44,4 +44,4 @@ Get JS model
${file} = Get File ${OUTDIR}/${type}.html
${strings} = Get Lines Matching Pattern ${file} window.output?"strings"?*
${settings} = Get Lines Matching Pattern ${file} window.settings =*
- [Return] ${strings} ${settings}
+ RETURN ${strings} ${settings}
diff --git a/atest/robot/output/json_output.robot b/atest/robot/output/json_output.robot
new file mode 100644
index 00000000000..d703bf2b8ec
--- /dev/null
+++ b/atest/robot/output/json_output.robot
@@ -0,0 +1,42 @@
+*** Settings ***
+Documentation JSON output is tested in detailed level using unit tests.
+Resource atest_resource.robot
+
+*** Variables ***
+${JSON} %{TEMPDIR}/output.json
+${XML} %{TEMPDIR}/output.xml
+
+*** Test Cases ***
+JSON output contains same suite information as XML output
+ Run Tests ${EMPTY} misc
+ Copy File ${OUTFILE} ${XML}
+ Run Tests Without Processing Output -o ${JSON} misc
+ Outputs Should Contain Same Data ${JSON} ${XML} ignore_timestamps=True
+
+JSON output structure
+ [Documentation] Full JSON schema validation would be good, but it's too slow with big output files.
+ ... The test after this one validates a smaller suite against a schema.
+ ${data} = Evaluate json.load(open($JSON, encoding='UTF-8'))
+ Lists Should Be Equal ${data} ${{['generator', 'generated', 'rpa', 'suite', 'statistics', 'errors']}}
+ Should Match ${data}[generator] Robot ?.* (* on *)
+ Should Match ${data}[generated] 20??-??-??T??:??:??.??????
+ Should Be Equal ${data}[rpa] ${False}
+ Should Be Equal ${data}[suite][name] Misc
+ Should Be Equal ${data}[suite][suites][1][name] Everything
+ Should Be Equal ${data}[statistics][total][skip] ${3}
+ Should Be Equal ${data}[statistics][tags][4][label] f1
+ Should Be Equal ${data}[statistics][suites][-1][id] s1-s17
+ Should Be Equal ${data}[errors][0][level] ERROR
+
+JSON output matches schema
+ [Tags] require-jsonschema
+ Run Tests Without Processing Output -o OUT.JSON misc/everything.robot
+ Validate JSON Output ${OUTDIR}/OUT.JSON
+
+Invalid JSON output file
+ ${path} = Normalize Path ${JSON}
+ Remove File ${path}
+ Create Directory ${path}
+ Run Tests Without Processing Output -o ${path} misc/pass_and_fail.robot
+ Stderr Should Match [[] ERROR ] Opening output file '${path}' failed: *${USAGE TIP}\n
+ [Teardown] Remove Directory ${JSON}
diff --git a/atest/robot/output/legacy_output.robot b/atest/robot/output/legacy_output.robot
new file mode 100644
index 00000000000..017cf0cc68a
--- /dev/null
+++ b/atest/robot/output/legacy_output.robot
@@ -0,0 +1,28 @@
+*** Settings ***
+Library LegacyOutputHelper.py
+Resource atest_resource.robot
+
+*** Test Cases ***
+Legacy output with Robot
+ Run Tests --legacyoutput output/legacy.robot validate output=False
+ Validate output
+
+Legacy output with Rebot
+ Run Tests ${EMPTY} output/legacy.robot
+ Copy Previous Outfile
+ Run Rebot --legacy-output ${OUTFILE COPY} validate output=False
+ Validate output
+
+Legacy output with Rebot when all times are not set
+ Run Rebot --legacy-output --test Passing ${OUTFILE COPY} validate output=False
+ Should Be Equal ${SUITE.start_time} ${None}
+ Should Be Equal ${SUITE.end_time} ${None}
+ Should Contain Tests ${SUITE} Passing
+
+*** Keywords ***
+Validate output
+ Should Contain Tests ${SUITE} Passing Failing Failing setup
+ ... Failing teardown Controls Embedded Warning
+ ${output} = Mask Changing Parts ${OUTFILE}
+ ${expected} = Mask Changing Parts ${DATADIR}/output/legacy.xml
+ Elements Should Be Equal ${output} ${expected} sort_children=True
diff --git a/atest/robot/output/listener_interface/body_items_v3.robot b/atest/robot/output/listener_interface/body_items_v3.robot
new file mode 100644
index 00000000000..fab0a6ee538
--- /dev/null
+++ b/atest/robot/output/listener_interface/body_items_v3.robot
@@ -0,0 +1,75 @@
+*** Settings ***
+Suite Setup Run Tests --listener ${DATADIR}/${MODIFIER} ${SOURCE}
+Resource atest_resource.robot
+
+*** Variables ***
+${SOURCE} output/listener_interface/body_items_v3/tests.robot
+${MODIFIER} output/listener_interface/body_items_v3/Modifier.py
+@{ALL TESTS} Library keyword User keyword Non-existing keyword
+... Empty keyword Duplicate keyword Invalid keyword
+... IF TRY FOR WHILE WHILE with modified limit VAR RETURN
+... Invalid syntax Run Keyword
+
+*** Test Cases ***
+Modify library keyword
+ Check Test Case Library keyword FAIL Expected state to be 'initial', but it was 'set by listener'.
+
+Modify user keyword
+ Check Test Case User keyword FAIL Failed by listener once!
+ Check Test Case Empty keyword PASS ${EMPTY}
+
+Modify invalid keyword
+ Check Test Case Non-existing keyword PASS ${EMPTY}
+ Check Test Case Duplicate keyword PASS ${EMPTY}
+ Check Test Case Invalid keyword PASS ${EMPTY}
+
+Modify keyword results
+ ${tc} = Get Test Case Invalid keyword
+ Check Keyword Data ${tc[0]} Invalid keyword
+ ... args=\${secret}
+ ... tags=end, fixed, start
+ ... doc=Results can be modified both in start and end!
+
+Modify FOR
+ ${tc} = Check Test Case FOR FAIL Listener failed me at 'b'!
+ Length Should Be ${tc[0].body} 2
+ Should Be Equal ${tc[0].assign}[0] secret
+ Should Be Equal ${tc[0, 0].assign}[\${x}] xxx
+ Should Be Equal ${tc[0, 1].assign}[\${x}] xxx
+
+Modify WHILE
+ ${tc} = Check Test Case WHILE FAIL Fail at iteration 10.
+ Length Should Be ${tc[0].body} 10
+
+Modify WHILE limit
+ ${tc} = Check Test Case WHILE with modified limit PASS ${EMPTY}
+ Length Should Be ${tc[1].body} 3
+ Check Log Message ${tc[1, 0, 0, 0]} \${x} = 1
+ Check Log Message ${tc[1, 1, 0, 0]} \${x} = 2
+ Check Log Message ${tc[1, 2]} Modified limit message.
+
+Modify IF
+ ${tc} = Check Test Case IF FAIL Executed!
+ Should Be Equal ${tc[0, 0].message} Secret message!
+ Should Be Equal ${tc[0, 1].message} Secret message!
+ Should Be Equal ${tc[0, 2].message} Executed!
+
+Modify TRY
+ ${tc} = Check Test Case TRY FAIL Not caught!
+ Length Should Be ${tc[0].body} 3
+
+Modify VAR
+ ${tc} = Check Test Case VAR FAIL value != VAR by listener
+ Should Be Equal ${tc[0].value}[0] secret
+ Should Be Equal ${tc[1].value}[0] secret
+
+Modify RETURN
+ ${tc} = Check Test Case RETURN FAIL RETURN by listener != value
+ Should Be Equal ${tc[0, 1].values}[0] secret
+
+Validate that all methods are called correctly
+ Run Tests --variable VALIDATE_EVENTS:True ${SOURCE}
+ Should contain tests ${SUITE} @{ALL TESTS}
+ Check Log Message ${SUITE.teardown.messages[0]} Listener StartEndBobyItemOnly is OK.
+ Check Log Message ${SUITE.teardown.messages[1]} Listener SeparateMethods is OK.
+ Check Log Message ${SUITE.teardown.messages[2]} Listener SeparateMethodsAlsoForKeywords is OK.
diff --git a/atest/robot/output/listener_interface/change_status.robot b/atest/robot/output/listener_interface/change_status.robot
new file mode 100644
index 00000000000..be97fe5db3f
--- /dev/null
+++ b/atest/robot/output/listener_interface/change_status.robot
@@ -0,0 +1,73 @@
+*** Settings ***
+Suite Setup Run Tests --listener ${DATADIR}/${MODIFIER} ${SOURCE}
+Resource atest_resource.robot
+
+*** Variables ***
+${SOURCE} output/listener_interface/body_items_v3/change_status.robot
+${MODIFIER} output/listener_interface/body_items_v3/ChangeStatus.py
+
+*** Test Cases ***
+Fail to pass
+ ${tc} = Check Test Case ${TEST NAME}
+ Check Keyword Data ${tc[0]} BuiltIn.Fail args=Pass me! status=PASS message=Failure hidden!
+ Check Log Message ${tc[0, 0]} Pass me! level=FAIL
+ Check Keyword Data ${tc[1]} BuiltIn.Log args=I'm run. status=PASS message=
+
+Pass to fail
+ ${tc} = Check Test Case ${TEST NAME}
+ Check Keyword Data ${tc[0]} BuiltIn.Log args=Fail me! status=FAIL message=Ooops!!
+ Check Log Message ${tc[0, 0]} Fail me! level=INFO
+ Check Keyword Data ${tc[1]} BuiltIn.Log args=I'm not run. status=NOT RUN message=
+
+Pass to fail without a message
+ ${tc} = Check Test Case ${TEST NAME}
+ Check Keyword Data ${tc[0]} BuiltIn.Log args=Silent fail! status=FAIL message=
+ Check Keyword Data ${tc[1]} BuiltIn.Log args=I'm not run. status=NOT RUN message=
+
+Skip to fail
+ ${tc} = Check Test Case ${TEST NAME}
+ Check Keyword Data ${tc[0]} BuiltIn.Skip args=Fail me! status=FAIL message=Failing!
+ Check Log Message ${tc[0, 0]} Fail me! level=SKIP
+ Check Keyword Data ${tc[1]} BuiltIn.Log args=I'm not run. status=NOT RUN message=
+
+Fail to skip
+ ${tc} = Check Test Case ${TEST NAME}
+ Check Keyword Data ${tc[0]} BuiltIn.Fail args=Skip me! status=SKIP message=Skipping!
+ Check Log Message ${tc[0, 0]} Skip me! level=FAIL
+ Check Keyword Data ${tc[1]} BuiltIn.Log args=I'm not run. status=NOT RUN message=
+
+Not run to fail
+ ${tc} = Check Test Case ${TEST NAME}
+ Check Keyword Data ${tc[0]} BuiltIn.Log args=Fail me! status=FAIL message=Ooops!!
+ Check Keyword Data ${tc[1]} BuiltIn.Log args=I'm not run. status=NOT RUN message=
+ Check Keyword Data ${tc[2]} BuiltIn.Log args=Fail me! status=FAIL message=Failing without running!
+ Check Keyword Data ${tc[3]} BuiltIn.Log args=I'm not run. status=NOT RUN message=
+
+Pass and fail to not run
+ ${tc} = Check Test Case ${TEST NAME}
+ Check Keyword Data ${tc[0]} BuiltIn.Log args=Mark not run! status=NOT RUN message=
+ Check Keyword Data ${tc[1]} BuiltIn.Fail args=Mark not run! status=NOT RUN message=Mark not run!
+ Check Keyword Data ${tc[2]} BuiltIn.Fail args=I fail! status=FAIL message=I fail!
+
+Only message
+ ${tc} = Check Test Case ${TEST NAME}
+ Check Keyword Data ${tc[0]} BuiltIn.Fail args=Change me! status=FAIL message=Changed!
+ Check Keyword Data ${tc[1]} Change message status=NOT RUN message=Changed!
+
+Control structures
+ ${tc} = Check Test Case ${TEST NAME}
+ Check Control Structure ${tc[0]} FOR
+ Check Control Structure ${tc[1]} WHILE
+ Check Control Structure ${tc[2]} IF/ELSE ROOT
+ Check Control Structure ${tc[3]} TRY/EXCEPT ROOT
+
+*** Keywords ***
+Check Control Structure
+ [Arguments] ${item} ${type}
+ VAR ${msg} Handled on ${type} level.
+ Should Be Equal ${item.type} ${type}
+ Should Be Equal ${item.status} PASS
+ Should Be Equal ${item.message} ${msg}
+ Should Be Equal ${item[0].status} FAIL
+ Should Be Equal ${item[0].message} ${msg}
+ Check Keyword Data ${item[0, 0]} BuiltIn.Fail args=${msg} status=FAIL message=${msg}
diff --git a/atest/robot/output/listener_interface/importing_listeners.robot b/atest/robot/output/listener_interface/importing_listeners.robot
index cd82a6b0416..f50534abe50 100644
--- a/atest/robot/output/listener_interface/importing_listeners.robot
+++ b/atest/robot/output/listener_interface/importing_listeners.robot
@@ -14,6 +14,15 @@ Python Class Listener From A Module With Different Name
Python Module Listener
module module_listener module_listener
+Listener Versions
+ [Template] NONE
+ Check Listener File listener-versions.txt
+ ... V2
+ ... V2AsNonInt
+ ... V3Implicit
+ ... V3Explicit
+ ... V3AsNonInt
+
Listener With Arguments
class listeners.WithArgs listeners 6
[Teardown] Check Listener File ${ARGS_FILE}
@@ -27,23 +36,33 @@ Listener With Argument Conversion
Listener With Path
class ${LISTENERS}${/}ListenAll.py ListenAll
- [Teardown] File Should Exist %{TEMPDIR}${/}${ALL_FILE2}
+ [Teardown] File Should Exist %{TEMPDIR}${/}${ALL_FILE2}
Listener With Wrong Number Of Arguments
[Template] Importing Listener Failed
0 listeners.WithArgs Listener 'WithArgs' expected 1 to 2 arguments, got 0.
1 listeners.WithArgs:1:2:3 Listener 'WithArgs' expected 1 to 2 arguments, got 3.
-
Non Existing Listener
[Template] Importing Listener Failed
2 NonExistingListener *${EMPTY TB}PYTHONPATH:* pattern=True
+Unsupported version
+ [Template] Taking Listener Into Use Failed
+ 3 unsupported_listeners.V1Listener Unsupported API version '1'.
+ 4 unsupported_listeners.V4Listener Unsupported API version '4'.
+ 5 unsupported_listeners.InvalidVersionListener Unsupported API version 'kekkonen'.
+
*** Keywords ***
Run Tests With Listeners
${listeners} = Catenate
... --listener ListenAll
... --listener listeners.ListenSome
... --listener module_listener
+ ... --listener listener_versions.V2
+ ... --listener listener_versions.V2AsNonInt
+ ... --listener listener_versions.V3Implicit
+ ... --listener listener_versions.V3Explicit
+ ... --listener listener_versions.V3AsNonInt
... --listener listeners.WithArgs:value
... --listener "listeners.WithArgs:a1:a;2"
... --listener "listeners.WithArgs;semi;colons:here"
@@ -53,11 +72,19 @@ Run Tests With Listeners
... --listener listeners.WithArgs
... --listener listeners.WithArgs:1:2:3
... --listener NonExistingListener
+ ... --listener unsupported_listeners.V1Listener
+ ... --listener unsupported_listeners.V4Listener
+ ... --listener unsupported_listeners.InvalidVersionListener
Run Tests ${listeners} misc/pass_and_fail.robot
Importing Listener Failed
+ [Arguments] ${index} ${name} ${error} ${pattern}=False
+ VAR ${error} Importing listener '${name.split(':')[0]}' failed: ${error}
+ Taking Listener Into Use Failed ${index} ${name} ${error} ${pattern}
+
+Taking Listener Into Use Failed
[Arguments] ${index} ${name} ${error} ${pattern}=False
Check Log Message
... ${ERRORS}[${index}]
- ... Taking listener '${name}' into use failed: Importing listener '${name.split(':')[0]}' failed: ${error}
+ ... Taking listener '${name}' into use failed: ${error}
... ERROR pattern=${pattern}
diff --git a/atest/robot/output/listener_interface/keyword_arguments_v3.robot b/atest/robot/output/listener_interface/keyword_arguments_v3.robot
new file mode 100644
index 00000000000..09b8b7d26b1
--- /dev/null
+++ b/atest/robot/output/listener_interface/keyword_arguments_v3.robot
@@ -0,0 +1,52 @@
+*** Settings ***
+Suite Setup Run Tests --listener ${DATADIR}/${MODIFIER} ${SOURCE}
+Resource atest_resource.robot
+
+*** Variables ***
+${SOURCE} output/listener_interface/body_items_v3/keyword_arguments.robot
+${MODIFIER} output/listener_interface/body_items_v3/ArgumentModifier.py
+
+*** Test Cases ***
+Library keyword arguments
+ ${tc} = Check Test Case ${TEST NAME}
+ Check Keyword Data ${tc[0]} Library.Library Keyword
+ ... args=\${STATE}, number=\${123}, obj=None, escape=c:\\\\temp\\\\new
+ Check Keyword Data ${tc[1]} Library.Library Keyword
+ ... args=new, 123, c:\\\\temp\\\\new, NONE
+ Check Keyword Data ${tc[2]} Library.Library Keyword
+ ... args=new, number=\${42}, escape=c:\\\\temp\\\\new, obj=Object(42)
+ Check Keyword Data ${tc[3]} Library.Library Keyword
+ ... args=number=1.0, escape=c:\\\\temp\\\\new, obj=Object(1), state=new
+
+User keyword arguments
+ ${tc} = Check Test Case ${TEST NAME}
+ Check Keyword Data ${tc[0]} User keyword
+ ... args=A, B, C, D
+ Check Keyword Data ${tc[1]} User keyword
+ ... args=A, B, d=D, c=\${{"c".upper()}}
+
+Invalid keyword arguments
+ ${tc} = Check Test Case Library keyword arguments
+ Check Keyword Data ${tc[4]} Non-existing
+ ... args=p, n=1 status=FAIL
+
+Too many arguments
+ ${tc} = Check Test Case ${TEST NAME}
+ Check Keyword Data ${tc[0]} Library.Library Keyword
+ ... args=a, b, c, d, e, f, g status=FAIL
+ Check Keyword Data ${tc[1]} User keyword
+ ... args=a, b, c, d, e, f, g status=FAIL
+ Check Keyword Data ${tc[2]} Library.Library Keyword
+ ... args=${{', '.join(str(i) for i in range(100))}} status=FAIL
+
+Conversion error
+ ${tc} = Check Test Case ${TEST NAME}
+ Check Keyword Data ${tc[0]} Library.Library Keyword
+ ... args=whatever, not a number status=FAIL
+ Check Keyword Data ${tc[1]} Library.Library Keyword
+ ... args=number=bad status=FAIL
+
+Positional after named
+ ${tc} = Check Test Case ${TEST NAME}
+ Check Keyword Data ${tc[0]} Library.Library Keyword
+ ... args=positional, number=-1, ooops status=FAIL
diff --git a/atest/robot/output/listener_interface/lineno_and_source.robot b/atest/robot/output/listener_interface/lineno_and_source.robot
index 2214c99a763..cc61cbe667d 100644
--- a/atest/robot/output/listener_interface/lineno_and_source.robot
+++ b/atest/robot/output/listener_interface/lineno_and_source.robot
@@ -12,206 +12,254 @@ ${RESOURCE FILE} ${LISTENER DIR}/lineno_and_source.resource
*** Test Cases ***
Keyword
START KEYWORD No Operation 6 NOT SET
- END KEYWORD No Operation 6 PASS
+ \END KEYWORD No Operation 6 PASS
User keyword
- START KEYWORD User Keyword 9 NOT SET
- START KEYWORD No Operation 85 NOT SET
- END KEYWORD No Operation 85 PASS
- START RETURN ${EMPTY} 86 NOT SET
- END RETURN ${EMPTY} 86 PASS
- END KEYWORD User Keyword 9 PASS
+ START KEYWORD User Keyword 9 NOT SET
+ START KEYWORD No Operation 101 NOT SET
+ \END KEYWORD No Operation 101 PASS
+ START RETURN ${EMPTY} 102 NOT SET
+ \END RETURN ${EMPTY} 102 PASS
+ \END KEYWORD User Keyword 9 PASS
User keyword in resource
START KEYWORD User Keyword In Resource 12 NOT SET
START KEYWORD No Operation 3 NOT SET source=${RESOURCE FILE}
- END KEYWORD No Operation 3 PASS source=${RESOURCE FILE}
- END KEYWORD User Keyword In Resource 12 PASS
+ \END KEYWORD No Operation 3 PASS source=${RESOURCE FILE}
+ \END KEYWORD User Keyword In Resource 12 PASS
Not run keyword
START KEYWORD Fail 16 NOT SET
- END KEYWORD Fail 16 FAIL
+ \END KEYWORD Fail 16 FAIL
START KEYWORD Fail 17 NOT RUN
- END KEYWORD Fail 17 NOT RUN
+ \END KEYWORD Fail 17 NOT RUN
START KEYWORD Non-existing 18 NOT RUN
- END KEYWORD Non-existing 18 NOT RUN
+ \END KEYWORD Non-existing 18 NOT RUN
FOR
- START FOR \${x} IN [ first | second ] 21 NOT SET
+ START FOR \${x} IN first second 21 NOT SET
START ITERATION \${x} = first 21 NOT SET
START KEYWORD No Operation 22 NOT SET
- END KEYWORD No Operation 22 PASS
- END ITERATION \${x} = first 21 PASS
+ \END KEYWORD No Operation 22 PASS
+ \END ITERATION \${x} = first 21 PASS
START ITERATION \${x} = second 21 NOT SET
START KEYWORD No Operation 22 NOT SET
- END KEYWORD No Operation 22 PASS
- END ITERATION \${x} = second 21 PASS
- END FOR \${x} IN [ first | second ] 21 PASS
+ \END KEYWORD No Operation 22 PASS
+ \END ITERATION \${x} = second 21 PASS
+ \END FOR \${x} IN first second 21 PASS
FOR in keyword
- START KEYWORD FOR In Keyword 26 NOT SET
- START FOR \${x} IN [ once ] 89 NOT SET
- START ITERATION \${x} = once 89 NOT SET
- START KEYWORD No Operation 90 NOT SET
- END KEYWORD No Operation 90 PASS
- END ITERATION \${x} = once 89 PASS
- END FOR \${x} IN [ once ] 89 PASS
- END KEYWORD FOR In Keyword 26 PASS
+ START KEYWORD FOR In Keyword 26 NOT SET
+ START FOR \${x} IN once 105 NOT SET
+ START ITERATION \${x} = once 105 NOT SET
+ START KEYWORD No Operation 106 NOT SET
+ \END KEYWORD No Operation 106 PASS
+ \END ITERATION \${x} = once 105 PASS
+ \END FOR \${x} IN once 105 PASS
+ \END KEYWORD FOR In Keyword 26 PASS
FOR in IF
START IF True 29 NOT SET
- START FOR \${x} | \${y} IN [ x | y ] 30 NOT SET
+ START FOR \${x} \${y} IN x y 30 NOT SET
START ITERATION \${x} = x, \${y} = y 30 NOT SET
START KEYWORD No Operation 31 NOT SET
- END KEYWORD No Operation 31 PASS
- END ITERATION \${x} = x, \${y} = y 30 PASS
- END FOR \${x} | \${y} IN [ x | y ] 30 PASS
- END IF True 29 PASS
+ \END KEYWORD No Operation 31 PASS
+ \END ITERATION \${x} = x, \${y} = y 30 PASS
+ \END FOR \${x} \${y} IN x y 30 PASS
+ \END IF True 29 PASS
FOR in resource
START KEYWORD FOR In Resource 36 NOT SET
- START FOR \${x} IN [ once ] 6 NOT SET source=${RESOURCE FILE}
+ START FOR \${x} IN once 6 NOT SET source=${RESOURCE FILE}
START ITERATION \${x} = once 6 NOT SET source=${RESOURCE FILE}
START KEYWORD Log 7 NOT SET source=${RESOURCE FILE}
- END KEYWORD Log 7 PASS source=${RESOURCE FILE}
- END ITERATION \${x} = once 6 PASS source=${RESOURCE FILE}
- END FOR \${x} IN [ once ] 6 PASS source=${RESOURCE FILE}
- END KEYWORD FOR In Resource 36 PASS
+ \END KEYWORD Log 7 PASS source=${RESOURCE FILE}
+ \END ITERATION \${x} = once 6 PASS source=${RESOURCE FILE}
+ \END FOR \${x} IN once 6 PASS source=${RESOURCE FILE}
+ \END KEYWORD FOR In Resource 36 PASS
IF
- START IF 1 > 2 39 NOT RUN
- START KEYWORD Fail 40 NOT RUN
- END KEYWORD Fail 40 NOT RUN
- END IF 1 > 2 39 NOT RUN
- START ELSE IF 1 < 2 41 NOT SET
- START KEYWORD No Operation 42 NOT SET
- END KEYWORD No Operation 42 PASS
- END ELSE IF 1 < 2 41 PASS
- START ELSE ${EMPTY} 43 NOT RUN
- START KEYWORD Fail 44 NOT RUN
- END KEYWORD Fail 44 NOT RUN
- END ELSE ${EMPTY} 43 NOT RUN
+ START IF 1 > 2 39 NOT RUN
+ START KEYWORD Fail 40 NOT RUN
+ \END KEYWORD Fail 40 NOT RUN
+ \END IF 1 > 2 39 NOT RUN
+ START ELSE IF 1 < 2 41 NOT SET
+ START KEYWORD No Operation 42 NOT SET
+ \END KEYWORD No Operation 42 PASS
+ \END ELSE IF 1 < 2 41 PASS
+ START ELSE \ 43 NOT RUN
+ START KEYWORD Fail 44 NOT RUN
+ \END KEYWORD Fail 44 NOT RUN
+ \END ELSE \ 43 NOT RUN
IF in keyword
- START KEYWORD IF In Keyword 48 NOT SET
- START IF True 94 NOT SET
- START KEYWORD No Operation 95 NOT SET
- END KEYWORD No Operation 95 PASS
- START RETURN ${EMPTY} 96 NOT SET
- END RETURN ${EMPTY} 96 PASS
- END IF True 94 PASS
- END KEYWORD IF In Keyword 48 PASS
+ START KEYWORD IF In Keyword 48 NOT SET
+ START IF True 110 NOT SET
+ START KEYWORD No Operation 111 NOT SET
+ \END KEYWORD No Operation 111 PASS
+ START RETURN ${EMPTY} 112 NOT SET
+ \END RETURN ${EMPTY} 112 PASS
+ \END IF True 110 PASS
+ \END KEYWORD IF In Keyword 48 PASS
IF in FOR
- START FOR \${x} IN [ 1 | 2 ] 52 NOT SET
- START ITERATION \${x} = 1 52 NOT SET
- START IF \${x} == 1 53 NOT SET
- START KEYWORD Log 54 NOT SET
- END KEYWORD Log 54 PASS
- END IF \${x} == 1 53 PASS
- START ELSE ${EMPTY} 55 NOT RUN
- START KEYWORD Fail 56 NOT RUN
- END KEYWORD Fail 56 NOT RUN
- END ELSE ${EMPTY} 55 NOT RUN
- END ITERATION \${x} = 1 52 PASS
- START ITERATION \${x} = 2 52 NOT SET
- START IF \${x} == 1 53 NOT RUN
- START KEYWORD Log 54 NOT RUN
- END KEYWORD Log 54 NOT RUN
- END IF \${x} == 1 53 NOT RUN
- START ELSE ${EMPTY} 55 NOT SET
- START KEYWORD Fail 56 NOT SET
- END KEYWORD Fail 56 FAIL
- END ELSE ${EMPTY} 55 FAIL
- END ITERATION \${x} = 2 52 FAIL
- END FOR \${x} IN [ 1 | 2 ] 52 FAIL
+ START FOR \${x} IN 1 2 52 NOT SET
+ START ITERATION \${x} = 1 52 NOT SET
+ START IF \${x} == 1 53 NOT SET
+ START KEYWORD Log 54 NOT SET
+ \END KEYWORD Log 54 PASS
+ \END IF \${x} == 1 53 PASS
+ START ELSE \ 55 NOT RUN
+ START KEYWORD Fail 56 NOT RUN
+ \END KEYWORD Fail 56 NOT RUN
+ \END ELSE \ 55 NOT RUN
+ \END ITERATION \${x} = 1 52 PASS
+ START ITERATION \${x} = 2 52 NOT SET
+ START IF \${x} == 1 53 NOT RUN
+ START KEYWORD Log 54 NOT RUN
+ \END KEYWORD Log 54 NOT RUN
+ \END IF \${x} == 1 53 NOT RUN
+ START ELSE \ 55 NOT SET
+ START KEYWORD Fail 56 NOT SET
+ \END KEYWORD Fail 56 FAIL
+ \END ELSE \ 55 FAIL
+ \END ITERATION \${x} = 2 52 FAIL
+ \END FOR \${x} IN 1 2 52 FAIL
IF in resource
- START KEYWORD IF In Resource 61 NOT SET
- START IF True 11 NOT SET source=${RESOURCE FILE}
- START KEYWORD No Operation 12 NOT SET source=${RESOURCE FILE}
- END KEYWORD No Operation 12 PASS source=${RESOURCE FILE}
- END IF True 11 PASS source=${RESOURCE FILE}
- END KEYWORD IF In Resource 61 PASS
+ START KEYWORD IF In Resource 61 NOT SET
+ START IF True 11 NOT SET source=${RESOURCE FILE}
+ START KEYWORD No Operation 12 NOT SET source=${RESOURCE FILE}
+ \END KEYWORD No Operation 12 PASS source=${RESOURCE FILE}
+ \END IF True 11 PASS source=${RESOURCE FILE}
+ \END KEYWORD IF In Resource 61 PASS
TRY
START TRY ${EMPTY} 65 NOT SET
START KEYWORD Fail 66 NOT SET
- END KEYWORD Fail 66 FAIL
- END TRY ${EMPTY} 65 FAIL
+ \END KEYWORD Fail 66 FAIL
+ \END TRY ${EMPTY} 65 FAIL
START EXCEPT AS \${name} 67 NOT SET
START TRY ${EMPTY} 68 NOT SET
START KEYWORD Fail 69 NOT SET
- END KEYWORD Fail 69 FAIL
- END TRY ${EMPTY} 68 FAIL
+ \END KEYWORD Fail 69 FAIL
+ \END TRY ${EMPTY} 68 FAIL
START FINALLY ${EMPTY} 70 NOT SET
START KEYWORD Should Be Equal 71 NOT SET
- END KEYWORD Should Be Equal 71 PASS
- END FINALLY ${EMPTY} 70 PASS
- END EXCEPT AS \${name} 67 FAIL
+ \END KEYWORD Should Be Equal 71 PASS
+ \END FINALLY ${EMPTY} 70 PASS
+ \END EXCEPT AS \${name} 67 FAIL
START ELSE ${EMPTY} 73 NOT RUN
START KEYWORD Fail 74 NOT RUN
- END KEYWORD Fail 74 NOT RUN
- END ELSE ${EMPTY} 73 NOT RUN
+ \END KEYWORD Fail 74 NOT RUN
+ \END ELSE ${EMPTY} 73 NOT RUN
TRY in keyword
START KEYWORD TRY In Keyword 78 NOT SET
- START TRY ${EMPTY} 100 NOT SET
- START RETURN ${EMPTY} 101 NOT SET
- END RETURN ${EMPTY} 101 PASS
- START KEYWORD Fail 102 NOT RUN
- END KEYWORD Fail 102 NOT RUN
- END TRY ${EMPTY} 100 PASS
- START EXCEPT No match AS \${var} 103 NOT RUN
- START KEYWORD Fail 104 NOT RUN
- END KEYWORD Fail 104 NOT RUN
- END EXCEPT No match AS \${var} 103 NOT RUN
- START EXCEPT No | Match | 2 AS \${x} 105 NOT RUN
- START KEYWORD Fail 106 NOT RUN
- END KEYWORD Fail 106 NOT RUN
- END EXCEPT No | Match | 2 AS \${x} 105 NOT RUN
- START EXCEPT ${EMPTY} 107 NOT RUN
- START KEYWORD Fail 108 NOT RUN
- END KEYWORD Fail 108 NOT RUN
- END EXCEPT ${EMPTY} 107 NOT RUN
- END KEYWORD TRY In Keyword 78 PASS
+ START TRY ${EMPTY} 116 NOT SET
+ START RETURN ${EMPTY} 117 NOT SET
+ \END RETURN ${EMPTY} 117 PASS
+ START KEYWORD Fail 118 NOT RUN
+ \END KEYWORD Fail 118 NOT RUN
+ \END TRY ${EMPTY} 116 PASS
+ START EXCEPT No match AS \${var} 119 NOT RUN
+ START KEYWORD Fail 120 NOT RUN
+ \END KEYWORD Fail 120 NOT RUN
+ \END EXCEPT No match AS \${var} 119 NOT RUN
+ START EXCEPT No Match 2 AS \${x} 121 NOT RUN
+ START KEYWORD Fail 122 NOT RUN
+ \END KEYWORD Fail 122 NOT RUN
+ \END EXCEPT No Match 2 AS \${x} 121 NOT RUN
+ START EXCEPT ${EMPTY} 123 NOT RUN
+ START KEYWORD Fail 124 NOT RUN
+ \END KEYWORD Fail 124 NOT RUN
+ \END EXCEPT ${EMPTY} 123 NOT RUN
+ \END KEYWORD TRY In Keyword 78 PASS
TRY in resource
START KEYWORD TRY In Resource 81 NOT SET
START TRY ${EMPTY} 16 NOT SET source=${RESOURCE FILE}
START KEYWORD Log 17 NOT SET source=${RESOURCE FILE}
- END KEYWORD Log 17 PASS source=${RESOURCE FILE}
- END TRY ${EMPTY} 16 PASS source=${RESOURCE FILE}
+ \END KEYWORD Log 17 PASS source=${RESOURCE FILE}
+ \END TRY ${EMPTY} 16 PASS source=${RESOURCE FILE}
START FINALLY ${EMPTY} 18 NOT SET source=${RESOURCE FILE}
START KEYWORD Log 19 NOT SET source=${RESOURCE FILE}
- END KEYWORD Log 19 PASS source=${RESOURCE FILE}
- END FINALLY ${EMPTY} 18 PASS source=${RESOURCE FILE}
- END KEYWORD TRY In Resource 81 PASS
+ \END KEYWORD Log 19 PASS source=${RESOURCE FILE}
+ \END FINALLY ${EMPTY} 18 PASS source=${RESOURCE FILE}
+ \END KEYWORD TRY In Resource 81 PASS
-Test
- [Template] Expect test
- Keyword 5
- User keyword 8
- User keyword in resource 11
- Not run keyword 14 FAIL
- \FOR 20
- FOR in keyword 25
- FOR in IF 28
- FOR in resource 35
- \IF 38
- IF in keyword 47
- IF in FOR 50 FAIL
- IF in resource 60
- \TRY 63 FAIL
- TRY in keyword 77
- TRY in resource 80
- [Teardown] Validate tests
+Run Keyword
+ START KEYWORD Run Keyword 84 NOT SET
+ START KEYWORD Log 84 NOT SET
+ \END KEYWORD Log 84 PASS
+ \END KEYWORD Run Keyword 84 PASS
+ START KEYWORD Run Keyword If 85 NOT SET
+ START KEYWORD User Keyword 85 NOT SET
+ START KEYWORD No Operation 101 NOT SET
+ \END KEYWORD No Operation 101 PASS
+ START RETURN ${EMPTY} 102 NOT SET
+ \END RETURN ${EMPTY} 102 PASS
+ \END KEYWORD User Keyword 85 PASS
+ \END KEYWORD Run Keyword If 85 PASS
+
+Run Keyword in keyword
+ START KEYWORD Run Keyword in keyword 89 NOT SET
+ START KEYWORD Run Keyword 128 NOT SET
+ START KEYWORD No Operation 128 NOT SET
+ \END KEYWORD No Operation 128 PASS
+ \END KEYWORD Run Keyword 128 PASS
+ \END KEYWORD Run Keyword in keyword 89 PASS
+
+Run Keyword in resource
+ START KEYWORD Run Keyword in resource 92 NOT SET
+ START KEYWORD Run Keyword 23 NOT SET source=${RESOURCE FILE}
+ START KEYWORD Log 23 NOT SET source=${RESOURCE FILE}
+ \END KEYWORD Log 23 PASS source=${RESOURCE FILE}
+ \END KEYWORD Run Keyword 23 PASS source=${RESOURCE FILE}
+ \END KEYWORD Run Keyword in resource 92 PASS
+
+In setup and teardown
+ START SETUP User Keyword 95 NOT SET
+ START KEYWORD No Operation 101 NOT SET
+ \END KEYWORD No Operation 101 PASS
+ START RETURN ${EMPTY} 102 NOT SET
+ \END RETURN ${EMPTY} 102 PASS
+ \END SETUP User Keyword 95 PASS
+ START KEYWORD No Operation 96 NOT SET
+ \END KEYWORD No Operation 96 PASS
+ START TEARDOWN Run Keyword 97 NOT SET
+ START KEYWORD Log 97 NOT SET
+ \END KEYWORD Log 97 PASS
+ \END TEARDOWN Run Keyword 97 PASS
Suite
- START SUITE Lineno And Source
- END SUITE Lineno And Source status=FAIL
+ START SUITE Lineno And Source
+ \END SUITE Lineno And Source status=FAIL
[Teardown] Validate suite
+Test
+ [Template] Expect test
+ Keyword 5
+ User keyword 8
+ User keyword in resource 11
+ Not run keyword 14 FAIL
+ \FOR 20
+ FOR in keyword 25
+ FOR in IF 28
+ FOR in resource 35
+ \IF 38
+ IF in keyword 47
+ IF in FOR 50 FAIL
+ IF in resource 60
+ \TRY 63 FAIL
+ TRY in keyword 77
+ TRY in resource 80
+ Run Keyword 83
+ Run Keyword in keyword 88
+ Run Keyword in resource 91
+ In setup and teardown 94
+ [Teardown] Validate tests
+
*** Keywords ***
Expect
[Arguments] ${event} ${type} ${name} ${lineno}=-1 ${status}= ${source}=${TEST CASE FILE}
diff --git a/atest/robot/output/listener_interface/listener_failing.robot b/atest/robot/output/listener_interface/listener_failing.robot
index 9ac1fd3eded..da2f6870fb2 100644
--- a/atest/robot/output/listener_interface/listener_failing.robot
+++ b/atest/robot/output/listener_interface/listener_failing.robot
@@ -43,12 +43,11 @@ Listener errors should be reported
Library listener errors should be reported
FOR ${index} ${method} IN ENUMERATE
- ... start_suite start_test start_keyword log_message
+ ... message start_suite start_test start_keyword log_message
... end_keyword end_test end_suite
Error should be reported in execution errors ${index} ${method} failing_listener
END
Error should be reported in stderr close failing_listener
- ... Error in library 'LibraryWithFailingListener':
Error should be reported in execution errors
[Arguments] ${index} ${method} ${listener}
@@ -58,9 +57,8 @@ Error should be reported in execution errors
Check log message ${ERRORS}[${index}] ${error} ERROR
Error should be reported in stderr
- [Arguments] ${method} ${listener} @{prefix}
+ [Arguments] ${method} ${listener}
${error} = Catenate
- ... @{prefix}
... Calling method '${method}' of listener '${listener}' failed:
... Expected failure in ${method}!
Stderr Should Contain [ ERROR ] ${error}
diff --git a/atest/robot/output/listener_interface/listener_logging.robot b/atest/robot/output/listener_interface/listener_logging.robot
index a6acb42e6df..bbeafc6c077 100644
--- a/atest/robot/output/listener_interface/listener_logging.robot
+++ b/atest/robot/output/listener_interface/listener_logging.robot
@@ -10,16 +10,29 @@ Logging from listener does not break output file
All start and end methods can log warnings to execution errors
Correct warnings should be shown in execution errors
-Methods inside start_keyword and end_keyword can log normal messages
+Methods under tests can log normal messages
Correct messages should be logged to normal log
-Methods outside start_keyword and end_keyword can log messages to syslog
+Methods outside tests can log messages to syslog
+ Correct messages should be logged to syslog
+
+Logging from listener when using JSON output
+ [Setup] Run Tests With Logging Listener format=json
+ Test statuses should be correct
+ Log and report should be created
+ Correct messages should be logged to normal log
+ Correct warnings should be shown in execution errors
Correct messages should be logged to syslog
*** Keywords ***
Run Tests With Logging Listener
- ${path} = Normalize Path ${LISTENER DIR}/logging_listener.py
- Run Tests --listener ${path} -l l.html -r r.html misc/pass_and_fail.robot
+ [Arguments] ${format}=xml
+ Should Be True $format in ('xml', 'json')
+ VAR ${output} ${OUTDIR}/output.${format}
+ VAR ${options}
+ ... --listener ${LISTENER DIR}/logging_listener.py
+ ... -o ${output} -l l.html -r r.html
+ Run Tests ${options} misc/pass_and_fail.robot output=${output}
Test statuses should be correct
Check Test Case Pass
@@ -40,14 +53,17 @@ Execution errors should have messages from message and log_message methods
Correct start/end warnings should be shown in execution errors
${msgs} = Get start/end messages ${ERRORS}
@{kw} = Create List start keyword end keyword
+ @{var} = Create List start var end var
@{return} = Create List start return end return
- @{setup} = Create List start setup @{kw} @{kw} @{kw} @{return} end setup
- @{uk} = Create List start keyword @{kw} @{kw} @{kw} @{return} end keyword
+ @{setup} = Create List start setup @{kw} @{kw} @{kw} @{var} @{kw} @{return} end setup
+ @{uk} = Create List start keyword @{kw} @{kw} @{kw} @{var} @{kw} @{return} end keyword
FOR ${index} ${method} IN ENUMERATE
... start_suite
... @{setup}
... start_test
... @{uk}
+ ... start keyword start keyword end keyword end keyword
+ ... @{kw}
... end_test
... start_test
... @{uk}
@@ -70,62 +86,79 @@ Get start/end messages
Correct messages should be logged to normal log
'My Keyword' has correct messages ${SUITE.setup} Suite Setup
${tc} = Check Test Case Pass
- 'My Keyword' has correct messages ${tc.kws[0]} Pass
+ Check Log Message ${tc[0]} start_test INFO
+ Check Log Message ${tc[1]} start_test WARN
+ 'My Keyword' has correct messages ${tc[2]} Pass
+ Check Log Message ${tc[5]} end_test INFO
+ Check Log Message ${tc[6]} end_test WARN
${tc} = Check Test Case Fail
- 'My Keyword' has correct messages ${tc.kws[0]} Fail
- 'Fail' has correct messages ${tc.kws[1]}
+ Check Log Message ${tc[0]} start_test INFO
+ Check Log Message ${tc[1]} start_test WARN
+ 'My Keyword' has correct messages ${tc[2]} Fail
+ 'Fail' has correct messages ${tc[3]}
+ Check Log Message ${tc[4]} end_test INFO
+ Check Log Message ${tc[5]} end_test WARN
'My Keyword' has correct messages
[Arguments] ${kw} ${name}
IF '${name}' == 'Suite Setup'
- ${type} = Set Variable setup
+ VAR ${type} setup
ELSE
- ${type} = Set Variable keyword
+ VAR ${type} keyword
END
- Check Log Message ${kw.body[0]} start ${type} INFO
- Check Log Message ${kw.body[1]} start ${type} WARN
- Check Log Message ${kw.body[2].body[0]} start keyword INFO
- Check Log Message ${kw.body[2].body[1]} start keyword WARN
- Check Log Message ${kw.body[2].body[2]} log_message: INFO Hello says "${name}"! INFO
- Check Log Message ${kw.body[2].body[3]} log_message: INFO Hello says "${name}"! WARN
- Check Log Message ${kw.body[2].body[4]} Hello says "${name}"! INFO
- Check Log Message ${kw.body[2].body[5]} end keyword INFO
- Check Log Message ${kw.body[2].body[6]} end keyword WARN
- Check Log Message ${kw.body[3].body[0]} start keyword INFO
- Check Log Message ${kw.body[3].body[1]} start keyword WARN
- Check Log Message ${kw.body[3].body[2]} end keyword INFO
- Check Log Message ${kw.body[3].body[3]} end keyword WARN
- Check Log Message ${kw.body[4].body[0]} start keyword INFO
- Check Log Message ${kw.body[4].body[1]} start keyword WARN
- Check Log Message ${kw.body[4].body[2]} log_message: INFO \${assign} = JUST TESTING... INFO
- Check Log Message ${kw.body[4].body[3]} log_message: INFO \${assign} = JUST TESTING... WARN
- Check Log Message ${kw.body[4].body[4]} \${assign} = JUST TESTING... INFO
- Check Log Message ${kw.body[4].body[5]} end keyword INFO
- Check Log Message ${kw.body[4].body[6]} end keyword WARN
- Check Log Message ${kw.body[5].body[0]} start return INFO
- Check Log Message ${kw.body[5].body[1]} start return WARN
- Check Log Message ${kw.body[5].body[2]} end return INFO
- Check Log Message ${kw.body[5].body[3]} end return WARN
- Check Log Message ${kw.body[6]} end ${type} INFO
- Check Log Message ${kw.body[7]} end ${type} WARN
+ Check Log Message ${kw[0]} start ${type} INFO
+ Check Log Message ${kw[1]} start ${type} WARN
+ Check Log Message ${kw[2, 0]} start keyword INFO
+ Check Log Message ${kw[2, 1]} start keyword WARN
+ Check Log Message ${kw[2, 2]} log_message: INFO Hello says "${name}"! INFO
+ Check Log Message ${kw[2, 3]} log_message: INFO Hello says "${name}"! WARN
+ Check Log Message ${kw[2, 4]} Hello says "${name}"! INFO
+ Check Log Message ${kw[2, 5]} end keyword INFO
+ Check Log Message ${kw[2, 6]} end keyword WARN
+ Check Log Message ${kw[3, 0]} start keyword INFO
+ Check Log Message ${kw[3, 1]} start keyword WARN
+ Check Log Message ${kw[3, 2]} end keyword INFO
+ Check Log Message ${kw[3, 3]} end keyword WARN
+ Check Log Message ${kw[4, 0]} start keyword INFO
+ Check Log Message ${kw[4, 1]} start keyword WARN
+ Check Log Message ${kw[4, 2]} log_message: INFO \${assign} = JUST TESTING... INFO
+ Check Log Message ${kw[4, 3]} log_message: INFO \${assign} = JUST TESTING... WARN
+ Check Log Message ${kw[4, 4]} \${assign} = JUST TESTING... INFO
+ Check Log Message ${kw[4, 5]} end keyword INFO
+ Check Log Message ${kw[4, 6]} end keyword WARN
+ Check Log Message ${kw[5, 0]} start var INFO
+ Check Log Message ${kw[5, 1]} start var WARN
+ Check Log Message ${kw[5, 2]} log_message: INFO \${expected} = JUST TESTING... INFO
+ Check Log Message ${kw[5, 3]} log_message: INFO \${expected} = JUST TESTING... WARN
+ Check Log Message ${kw[5, 4]} \${expected} = JUST TESTING... INFO
+ Check Log Message ${kw[5, 5]} end var INFO
+ Check Log Message ${kw[5, 6]} end var WARN
+ Check Log Message ${kw[6, 0]} start keyword INFO
+ Check Log Message ${kw[6, 1]} start keyword WARN
+ Check Log Message ${kw[6, 2]} end keyword INFO
+ Check Log Message ${kw[6, 3]} end keyword WARN
+ Check Log Message ${kw[7, 0]} start return INFO
+ Check Log Message ${kw[7, 1]} start return WARN
+ Check Log Message ${kw[7, 2]} end return INFO
+ Check Log Message ${kw[7, 3]} end return WARN
+ Check Log Message ${kw[8]} end ${type} INFO
+ Check Log Message ${kw[9]} end ${type} WARN
'Fail' has correct messages
[Arguments] ${kw}
- Check Log Message ${kw.body[0]} start keyword INFO
- Check Log Message ${kw.body[1]} start keyword WARN
- Check Log Message ${kw.body[2]} log_message: FAIL Expected failure INFO
- Check Log Message ${kw.body[3]} log_message: FAIL Expected failure WARN
- Check Log Message ${kw.body[4]} Expected failure FAIL
- Check Log Message ${kw.body[5]} end keyword INFO
- Check Log Message ${kw.body[6]} end keyword WARN
+ Check Log Message ${kw[0]} start keyword INFO
+ Check Log Message ${kw[1]} start keyword WARN
+ Check Log Message ${kw[2]} log_message: FAIL Expected failure INFO
+ Check Log Message ${kw[3]} log_message: FAIL Expected failure WARN
+ Check Log Message ${kw[4]} Expected failure FAIL
+ Check Log Message ${kw[5]} end keyword INFO
+ Check Log Message ${kw[6]} end keyword WARN
Correct messages should be logged to syslog
FOR ${msg} IN
... message: INFO Robot Framework
... start_suite
... end_suite
- ... start_test
- ... end_test
... output_file
... log_file
... report_file
diff --git a/atest/robot/output/listener_interface/listener_methods.robot b/atest/robot/output/listener_interface/listener_methods.robot
index ff96e7c101a..b97e6414441 100644
--- a/atest/robot/output/listener_interface/listener_methods.robot
+++ b/atest/robot/output/listener_interface/listener_methods.robot
@@ -21,12 +21,12 @@ Listen Some
Correct Attributes To Listener Methods
${status} = Log File %{TEMPDIR}/${ATTR_TYPE_FILE}
- Stderr Should Not Contain attributeverifyinglistener
+ Stderr Should Not Contain VerifyAttributes
Should Not Contain ${status} FAILED
Keyword Tags
${status} = Log File %{TEMPDIR}/${ATTR_TYPE_FILE}
- Should Contain X Times ${status} PASSED | tags: [force, keyword, tags] 6
+ Should Contain X Times ${status} passed | tags: [force, keyword, tags] 6
Suite And Test Counts
Run Tests --listener listeners.SuiteAndTestCounts misc/suites/subsuites misc/suites/subsuites2
@@ -47,8 +47,8 @@ Keyword Status
Executing Keywords from Listeners
Run Tests --listener listeners.KeywordExecutingListener misc/pass_and_fail.robot
${tc}= Get Test Case Pass
- Check Log Message ${tc.kws[0].msgs[0]} Start Pass
- Check Log Message ${tc.kws[2].msgs[0]} End Pass
+ Check Log Message ${tc[0, 0]} Start Pass
+ Check Log Message ${tc[4, 0]} End Pass
Test Template
${listener} = Normalize Path ${LISTENER DIR}/verify_template_listener.py
@@ -56,13 +56,19 @@ Test Template
Stderr Should Be Empty
Keyword Arguments Are Always Strings
- ${result} = Run Tests --listener attributeverifyinglistener ${LISTENER DIR}/keyword_argument_types.robot
+ ${result} = Run Tests --listener VerifyAttributes ${LISTENER DIR}/keyword_argument_types.robot
Should Be Empty ${result.stderr}
Check Test Tags Run Keyword with already resolved non-string arguments in test data 1 2
Check Test Case Run Keyword with non-string arguments in library
${status} = Log File %{TEMPDIR}/${ATTR_TYPE_FILE}
Should Not Contain ${status} FAILED
+Keyword Attributes For Control Structures
+ Run Tests --listener VerifyAttributes misc/for_loops.robot misc/while.robot misc/try_except.robot misc/if_else.robot
+ Stderr Should Be Empty
+ ${status} = Log File %{TEMPDIR}/${ATTR_TYPE_FILE}
+ Should Not Contain ${status} FAILED
+
TimeoutError occurring during listener method is propagaged
[Documentation] Timeouts can only occur inside `log_message`.
... Cannot reliable set timeouts to occur during it, so the listener
@@ -79,7 +85,7 @@ Run Tests With Listeners
... --listener ListenAll:%{TEMPDIR}${/}${ALL_FILE2}
... --listener module_listener
... --listener listeners.ListenSome
- ... --listener attributeverifyinglistener
+ ... --listener VerifyAttributes
... --metadata ListenerMeta:Hello
Run Tests ${args} misc/pass_and_fail.robot
@@ -88,51 +94,74 @@ Check Listen All File
@{expected}= Create List Got settings on level: INFO
... SUITE START: Pass And Fail (s1) 'Some tests here' [ListenerMeta: Hello]
... SETUP START: My Keyword ['Suite Setup'] (line 3)
- ... KEYWORD START: BuiltIn.Log ['Hello says "\${who}"!', '\${LEVEL1}'] (line 27)
+ ... KEYWORD START: BuiltIn.Log ['Hello says "\${who}"!', '\${LEVEL1}'] (line 31)
... LOG MESSAGE: [INFO] Hello says "Suite Setup"!
... KEYWORD END: PASS
- ... KEYWORD START: BuiltIn.Log ['Debug message', '\${LEVEL2}'] (line 28)
+ ... KEYWORD START: BuiltIn.Log ['Debug message', '\${LEVEL2}'] (line 32)
... KEYWORD END: PASS
- ... KEYWORD START: \${assign} = String.Convert To Upper Case ['Just testing...'] (line 29)
+ ... KEYWORD START: \${assign} = String.Convert To Upper Case ['Just testing...'] (line 33)
... LOG MESSAGE: [INFO] \${assign} = JUST TESTING...
... KEYWORD END: PASS
- ... RETURN START: (line 30)
+ ... VAR START: \${expected}${SPACE*4}JUST TESTING... (line 34)
+ ... LOG MESSAGE: [INFO] \${expected} = JUST TESTING...
+ ... VAR END: PASS
+ ... KEYWORD START: BuiltIn.Should Be Equal ['\${assign}', '\${expected}'] (line 35)
+ ... KEYWORD END: PASS
+ ... RETURN START: (line 36)
... RETURN END: PASS
... SETUP END: PASS
- ... TEST START: Pass (s1-t1, line 12) '' ['force', 'pass']
- ... KEYWORD START: My Keyword ['Pass'] (line 15)
- ... KEYWORD START: BuiltIn.Log ['Hello says "\${who}"!', '\${LEVEL1}'] (line 27)
+ ... TEST START: Pass (s1-t1, line 14) '' ['force', 'pass']
+ ... KEYWORD START: My Keyword ['Pass'] (line 17)
+ ... KEYWORD START: BuiltIn.Log ['Hello says "\${who}"!', '\${LEVEL1}'] (line 31)
... LOG MESSAGE: [INFO] Hello says "Pass"!
... KEYWORD END: PASS
- ... KEYWORD START: BuiltIn.Log ['Debug message', '\${LEVEL2}'] (line 28)
+ ... KEYWORD START: BuiltIn.Log ['Debug message', '\${LEVEL2}'] (line 32)
... KEYWORD END: PASS
- ... KEYWORD START: \${assign} = String.Convert To Upper Case ['Just testing...'] (line 29)
+ ... KEYWORD START: \${assign} = String.Convert To Upper Case ['Just testing...'] (line 33)
... LOG MESSAGE: [INFO] \${assign} = JUST TESTING...
... KEYWORD END: PASS
- ... RETURN START: (line 30)
+ ... VAR START: \${expected}${SPACE*4}JUST TESTING... (line 34)
+ ... LOG MESSAGE: [INFO] \${expected} = JUST TESTING...
+ ... VAR END: PASS
+ ... KEYWORD START: BuiltIn.Should Be Equal ['\${assign}', '\${expected}'] (line 35)
+ ... KEYWORD END: PASS
+ ... RETURN START: (line 36)
... RETURN END: PASS
... KEYWORD END: PASS
+ ... KEYWORD START: example.Resource Keyword (line 18)
+ ... KEYWORD START: BuiltIn.Log ['Hello, resource!'] (line 3)
+ ... LOG MESSAGE: [INFO] Hello, resource!
+ ... KEYWORD END: PASS
+ ... KEYWORD END: PASS
+ ... KEYWORD START: BuiltIn.Should Be Equal ['\${VARIABLE}', 'From variables.py with arg 1'] (line 19)
+ ... KEYWORD END: PASS
... TEST END: PASS
- ... TEST START: Fail (s1-t2, line 17) 'FAIL Expected failure' ['fail', 'force']
- ... KEYWORD START: My Keyword ['Fail'] (line 20)
- ... KEYWORD START: BuiltIn.Log ['Hello says "\${who}"!', '\${LEVEL1}'] (line 27)
+ ... TEST START: Fail (s1-t2, line 21) 'FAIL Expected failure' ['fail', 'force']
+ ... KEYWORD START: My Keyword ['Fail'] (line 24)
+ ... KEYWORD START: BuiltIn.Log ['Hello says "\${who}"!', '\${LEVEL1}'] (line 31)
... LOG MESSAGE: [INFO] Hello says "Fail"!
... KEYWORD END: PASS
- ... KEYWORD START: BuiltIn.Log ['Debug message', '\${LEVEL2}'] (line 28)
+ ... KEYWORD START: BuiltIn.Log ['Debug message', '\${LEVEL2}'] (line 32)
... KEYWORD END: PASS
- ... KEYWORD START: \${assign} = String.Convert To Upper Case ['Just testing...'] (line 29)
+ ... KEYWORD START: \${assign} = String.Convert To Upper Case ['Just testing...'] (line 33)
... LOG MESSAGE: [INFO] \${assign} = JUST TESTING...
... KEYWORD END: PASS
- ... RETURN START: (line 30)
+ ... VAR START: \${expected}${SPACE*4}JUST TESTING... (line 34)
+ ... LOG MESSAGE: [INFO] \${expected} = JUST TESTING...
+ ... VAR END: PASS
+ ... KEYWORD START: BuiltIn.Should Be Equal ['\${assign}', '\${expected}'] (line 35)
+ ... KEYWORD END: PASS
+ ... RETURN START: (line 36)
... RETURN END: PASS
... KEYWORD END: PASS
- ... KEYWORD START: BuiltIn.Fail ['Expected failure'] (line 21)
+ ... KEYWORD START: BuiltIn.Fail ['Expected failure'] (line 25)
... LOG MESSAGE: [FAIL] Expected failure
... KEYWORD END: FAIL
... TEST END: FAIL Expected failure
... SUITE END: FAIL 2 tests, 1 passed, 1 failed
... Output: output.xml Closing...
Check Listener File ${filename} @{expected}
+ Stderr Should Be Empty
Calling listener failed
[Arguments] ${method} ${error}
diff --git a/atest/robot/output/listener_interface/listener_order.robot b/atest/robot/output/listener_interface/listener_order.robot
new file mode 100644
index 00000000000..162b4a50154
--- /dev/null
+++ b/atest/robot/output/listener_interface/listener_order.robot
@@ -0,0 +1,57 @@
+*** Settings ***
+Suite Setup Run Tests With Ordered Listeners
+Resource atest_resource.robot
+
+*** Variables ***
+${LISTENER} ${DATADIR}/output/listener_interface/ListenerOrder.py
+
+*** Test Cases ***
+Validate normal order
+ VAR ${expected}
+ ... LIB 3 (999.9): start_suite
+ ... CLI 2 (3.14): start_suite
+ ... CLI 3 (None): start_suite
+ ... LIB 1 (0): start_suite
+ ... LIB 2 (None): start_suite
+ ... CLI 1 (-1): start_suite
+ ... LIB 3 (999.9): log_message
+ ... CLI 2 (3.14): log_message
+ ... CLI 3 (None): log_message
+ ... LIB 1 (0): log_message
+ ... LIB 2 (None): log_message
+ ... CLI 1 (-1): log_message
+ ... LIB 3 (999.9): end_test
+ ... CLI 2 (3.14): end_test
+ ... CLI 3 (None): end_test
+ ... LIB 1 (0): end_test
+ ... LIB 2 (None): end_test
+ ... CLI 1 (-1): end_test
+ ... separator=\n
+ File Should Be Equal To %{TEMPDIR}/listener_order.log ${expected}\n
+
+Validate close order
+ [Documentation] Library listeners are closed when libraries go out of scope.
+ VAR ${expected}
+ ... LIB 1 (0): close
+ ... LIB 2 (None): close
+ ... LIB 3 (999.9): close
+ ... CLI 2 (3.14): close
+ ... CLI 3 (None): close
+ ... CLI 1 (-1): close
+ ... separator=\n
+ File Should Be Equal To %{TEMPDIR}/listener_close_order.log ${expected}\n
+
+Invalid priority
+ ${listener} = Normalize Path ${LISTENER}
+ Check Log Message ${ERRORS}[0] Taking listener '${listener}:NOT USED:invalid' into use failed: Invalid listener priority 'invalid'. ERROR
+ Check Log Message ${ERRORS}[1] Error in library 'BAD': Registering listeners failed: Taking listener 'SELF' into use failed: Invalid listener priority 'bad'. ERROR
+
+*** Keywords ***
+Run Tests With Ordered Listeners
+ ${listener} = Normalize Path ${LISTENER}
+ VAR ${options}
+ ... --listener "${listener}:CLI 1:-1"
+ ... --listener "${listener}:CLI 2:3.14"
+ ... --listener "${listener}:NOT USED:invalid"
+ ... --listener "${listener}:CLI 3"
+ Run Tests ${options} output/listener_interface/listener_order.robot
diff --git a/atest/robot/output/listener_interface/listener_resource.robot b/atest/robot/output/listener_interface/listener_resource.robot
index 6b94649e09a..23d4da59ca3 100644
--- a/atest/robot/output/listener_interface/listener_resource.robot
+++ b/atest/robot/output/listener_interface/listener_resource.robot
@@ -42,4 +42,4 @@ Get Listener File
[Arguments] ${file}
${path} = Join Path %{TEMPDIR} ${file}
${content} = Get File ${path}
- [Return] ${content}
+ RETURN ${content}
diff --git a/atest/robot/output/listener_interface/listener_v3.robot b/atest/robot/output/listener_interface/listener_v3.robot
index d533bbd7c35..031dd246aaa 100644
--- a/atest/robot/output/listener_interface/listener_v3.robot
+++ b/atest/robot/output/listener_interface/listener_v3.robot
@@ -1,5 +1,5 @@
*** Settings ***
-Suite Setup Run Tests --listener ${LISTENER DIR}/v3.py -l l -r r -b d -x x misc/pass_and_fail.robot
+Suite Setup Run Tests --listener ${LISTENER DIR}/v3.py -l l -r r -b d -x x -L trace misc/pass_and_fail.robot
Resource listener_resource.robot
*** Variables ***
@@ -8,11 +8,11 @@ ${SEPARATOR} ${EMPTY + '-' * 78}
*** Test Cases ***
New tests and keywords can be added
${tc} = Check test case Added by start_suite [start suite] FAIL [start] [end]
- Check keyword data ${tc.kws[0]} BuiltIn.No Operation
+ Check keyword data ${tc[0]} BuiltIn.No Operation
${tc} = Check test case Added by startTest PASS Dynamically added! [end]
- Check keyword data ${tc.kws[0]} BuiltIn.Fail args=Dynamically added! status=FAIL
+ Check keyword data ${tc[0]} BuiltIn.Fail args=Dynamically added! status=FAIL
${tc} = Check test case Added by end_Test FAIL [start] [end]
- Check keyword data ${tc.kws[0]} BuiltIn.Log args=Dynamically added!, INFO
+ Check keyword data ${tc[0]} BuiltIn.Log args=Dynamically added!, INFO
Stdout Should Contain SEPARATOR=\n
... Added by start_suite [start suite] :: [start suite] ${SPACE*17} | FAIL |
... [start] [end]
@@ -63,12 +63,34 @@ Changing current element docs does not change console output, but does change ou
Check Test Doc Pass [start suite] [start suite] [start test] [end test]
Log messages and timestamps can be changed
- ${tc} = Get test case Pass [start suite]
- Check log message ${tc.kws[0].kws[0].msgs[0]} HELLO SAYS "PASS"!
- Should be equal ${tc.kws[0].kws[0].msgs[0].timestamp} 20151216 15:51:20.141
+ ${tc} = Get Test Case Pass [start suite]
+ Check Keyword Data ${tc[0, 0]} BuiltIn.Log args=Hello says "\${who}"!, \${LEVEL1}
+ Check Log Message ${tc[0, 0, 0]} HELLO SAYS "PASS"!
+ Should Be Equal ${tc[0, 0, 0].timestamp} ${datetime(2015, 12, 16, 15, 51, 20, 141000)}
+
+Log message can be removed by setting message to `None`
+ ${tc} = Get Test Case Fail [start suite]
+ Check Keyword Data ${tc[0, 0]} BuiltIn.Log args=Hello says "\${who}"!, \${LEVEL1}
+ Should Be Empty ${tc[0, 0].body}
+ File Should Not Contain ${OUTDIR}/d.txt HELLO SAYS "FAIL"!
+ File Should Not Contain ${OUTDIR}/d.txt None
Syslog messages can be changed
- Syslog Should Contain Match 20151216 15:51:20.141 | INFO \ | TESTS EXECUTION ENDED. STATISTICS:
+ Syslog Should Contain Match 2015-12-16 15:51:20.141000 | INFO \ | TESTS EXECUTION ENDED. STATISTICS:
+
+Library import
+ Stdout Should Contain Imported library 'BuiltIn' with 107 keywords.
+ Stdout Should Contain Imported library 'String' with 32 keywords.
+ ${tc} = Get Test Case Pass [start suite]
+ Check Keyword Data ${tc[0, 0]} BuiltIn.Log doc=Changed! args=Hello says "\${who}"!, \${LEVEL1}
+
+Resource import
+ Stdout Should Contain Imported resource 'example' with 2 keywords.
+ ${tc} = Get Test Case Pass [start suite]
+ Check Keyword Data ${tc[1, 1]} example.New! doc=Dynamically created.
+
+Variables import
+ Stdout Should Contain Imported variables 'variables.py' without much info.
File methods and close are called
Stderr Should Be Equal To SEPARATOR=\n
@@ -78,3 +100,9 @@ File methods and close are called
... Log: l.html
... Report: r.html
... Close\n
+
+File methods when files are disabled
+ Run Tests Without Processing Output --listener ${LISTENER DIR}/v3.py -o NONE -r NONE -l NONE misc/pass_and_fail.robot
+ Stderr Should Be Equal To SEPARATOR=\n
+ ... Output: None
+ ... Close\n
diff --git a/atest/robot/output/listener_interface/log_levels.robot b/atest/robot/output/listener_interface/log_levels.robot
index e48ad12f0ac..3bf2727bb0e 100644
--- a/atest/robot/output/listener_interface/log_levels.robot
+++ b/atest/robot/output/listener_interface/log_levels.robot
@@ -11,10 +11,14 @@ Log messages are collected on INFO level by default
Logged messages should be
... INFO: Hello says "Suite Setup"!
... INFO: \${assign} = JUST TESTING...
+ ... INFO: \${expected} = JUST TESTING...
... INFO: Hello says "Pass"!
... INFO: \${assign} = JUST TESTING...
+ ... INFO: \${expected} = JUST TESTING...
+ ... INFO: Hello, resource!
... INFO: Hello says "Fail"!
... INFO: \${assign} = JUST TESTING...
+ ... INFO: \${expected} = JUST TESTING...
... FAIL: Expected failure
Log messages are collected on specified level
@@ -23,12 +27,28 @@ Log messages are collected on specified level
... INFO: Hello says "Suite Setup"!
... DEBUG: Debug message
... INFO: \${assign} = JUST TESTING...
+ ... INFO: \${expected} = JUST TESTING...
+ ... DEBUG: Argument types are:
+ ... Logs the given message with the given level.\\x3c/p>"
- Should Contain ${log} "* Logs the given message with the given level.\\x3c/p>\\n Content flattened.\\x3c/b>\\x3c/i>\\x3c/p>"
+ Should Contain ${log} "*Content flattened."
diff --git a/atest/robot/rebot/start_and_endtime_from_cli.robot b/atest/robot/rebot/start_and_endtime_from_cli.robot
index 6f2f424f7a3..b22cdc84d2b 100644
--- a/atest/robot/rebot/start_and_endtime_from_cli.robot
+++ b/atest/robot/rebot/start_and_endtime_from_cli.robot
@@ -9,67 +9,67 @@ ${INPUT2} %{TEMPDIR}${/}rebot-test-b.xml
${COMBINED} %{TEMPDIR}${/}combined.xml
*** Test Cases ***
-Combine With Both Starttime and endtime should Set Correct Elapsed Time
+Combine with both start time and end time
Log Many ${INPUT1} ${INPUT2}
- Run Rebot --starttime 2007:09:25:21:51 --endtime 2007:09:26:01:12:30.200 ${INPUT1} ${INPUT2}
- Should Be Equal ${SUITE.starttime} 20070925 21:51:00.000
- Should Be Equal ${SUITE.endtime} 20070926 01:12:30.200
- Should Be True ${SUITE.elapsedtime} == (3*60*60 + 21*60 + 30) * 1000 + 200
+ Run Rebot --starttime 2007:09:25:21:51 --endtime 2007-09-26T01:12:30.200 ${INPUT1} ${INPUT2}
+ Should Be Equal ${SUITE.start_time} ${datetime(2007, 9, 25, 21, 51)}
+ Should Be Equal ${SUITE.end_time} ${datetime(2007, 9, 26, 1, 12, 30, 200000)}
+ Should Be Equal ${SUITE.elapsed_time} ${timedelta(seconds=3*60*60 + 21*60 + 30.2)}
-Combine With Only Starttime Should Only Affect Starttime
+Combine with only start time
Run Rebot --starttime 20070925-2151 ${INPUT1} ${INPUT2}
- Should Be Equal ${SUITE.starttime} 20070925 21:51:00.000
- Should Be Equal ${SUITE.endtime} ${ORIG_END}
- Should Be Equal ${SUITE.elapsedtime} ${ORIG_ELAPSED}
+ Should Be Equal ${SUITE.start_time} ${datetime(2007, 9, 25, 21, 51)}
+ Should Be Equal ${SUITE.end_time} ${{datetime.datetime(2007, 9, 25, 21, 51) + $ORIG_ELAPSED}}
+ Should Be Equal ${SUITE.elapsed_time} ${ORIG_ELAPSED}
-Combine With Only Endtime Should Only Affect Endtime
+Combine with only end time
Run Rebot --endtime 2010_01.01:12-33 ${INPUT1} ${INPUT2}
- Should Be Equal ${SUITE.starttime} ${ORIG_START}
- Should Be Equal ${SUITE.endtime} 20100101 12:33:00.000
- Should Be Equal ${SUITE.elapsedtime} ${ORIG_ELAPSED}
+ Should Be Equal ${SUITE.start_time} ${{datetime.datetime(2010, 1, 1, 12, 33) - $ORIG_ELAPSED}}
+ Should Be Equal ${SUITE.end_time} ${datetime(2010, 1, 1, 12, 33)}
+ Should Be Equal ${SUITE.elapsed_time} ${ORIG_ELAPSED}
-Recombining Should Work
+Recombining
${options} = Catenate
... --starttime 2007:09:25:21:51
... --endtime 2007:09:26:01:12:30:200
... --output ${COMBINED}
Run Rebot Without Processing Output ${options} ${INPUT1} ${INPUT2}
Run Rebot ${EMPTY} ${INPUT1} ${INPUT2} ${COMBINED}
- Should Be True '${SUITE.elapsedtime}' > '03:21:30.200'
+ Should Be True $SUITE.elapsed_time > datetime.timedelta(hours=3, minutes=21, seconds=30.2)
-It should Be possible to Omit Time Altogether
+Omit time part altogether
Run Rebot --starttime 2007-10-01 --endtime 20071006 ${INPUT1} ${INPUT2}
- Should Be Equal ${SUITE.starttime} 20071001 00:00:00.000
- Should Be Equal ${SUITE.endtime} 20071006 00:00:00.000
- Should Be True ${SUITE.elapsedtime} == 120*60*60 * 1000
+ Should Be Equal ${SUITE.start_time} ${datetime(2007, 10, 1)}
+ Should Be Equal ${SUITE.end_time} ${datetime(2007, 10, 6)}
+ Should Be Equal ${SUITE.elapsed_time} ${timedelta(days=5)}
-Use Starttime With Single Output
- Run Rebot --starttime 20070925-2151 ${INPUT1}
- Should Be Equal ${SUITE.starttime} 20070925 21:51:00.000
- Should Be Equal ${SUITE.endtime} ${SINGLE_SUITE_ORIG_END}
- Should Be True ${SUITE.elapsedtime} > ${SINGLE SUITE ORIG ELAPSED}
+Start time and end time with single output
+ Run Rebot --starttime 20070925-2151 --endtime 20070925-2252 ${INPUT1}
+ Should Be Equal ${SUITE.start_time} ${datetime(2007, 9, 25, 21, 51)}
+ Should Be Equal ${SUITE.end_time} ${datetime(2007, 9, 25, 22, 52)}
+ Should Be Equal ${SUITE.elapsed_time} ${timedelta(hours=1, minutes=1)}
-Use Endtime With Single Output
- Run Rebot --endtime 20070925-2151 ${INPUT1}
- Should Be Equal ${SUITE.starttime} ${SINGLE_SUITE_ORIG_START}
- Should Be Equal ${SUITE.endtime} 20070925 21:51:00.000
- Should Be True ${SUITE.elapsedtime} < ${SINGLE SUITE ORIG ELAPSED}
+Start time with single output
+ Run Rebot --starttime 20070925-2151 ${INPUT1}
+ Should Be Equal ${SUITE.start_time} ${datetime(2007, 9, 25, 21, 51)}
+ Should Be Equal ${SUITE.end_time} ${SINGLE_SUITE_ORIG_END}
+ Should Be True $SUITE.elapsed_time > $SINGLE_SUITE_ORIG_ELAPSED
-Use Starttime And Endtime With Single Output
- Run Rebot --starttime 20070925-2151 --endtime 20070925-2252 ${INPUT1}
- Should Be Equal ${SUITE.starttime} 20070925 21:51:00.000
- Should Be Equal ${SUITE.endtime} 20070925 22:52:00.000
- Should Be Equal ${SUITE.elapsedtime} ${3660000}
+End time with single output
+ Run Rebot --endtime '2023-09-07 19:31:01.234' ${INPUT1}
+ Should Be Equal ${SUITE.start_time} ${SINGLE_SUITE_ORIG_START}
+ Should Be Equal ${SUITE.end_time} ${datetime(2023, 9, 7, 19, 31, 1, 234000)}
+ Should Be True $SUITE.elapsed_time < $SINGLE_SUITE_ORIG_ELAPSED
*** Keywords ***
Create Input Files
Create Output With Robot ${INPUT1} ${EMPTY} misc/normal.robot
Create Output With Robot ${INPUT2} ${EMPTY} misc/suites/tsuite1.robot
Run Rebot ${EMPTY} ${INPUT1} ${INPUT2}
- Set Suite Variable $ORIG_START ${SUITE.starttime}
- Set Suite Variable $ORIG_END ${SUITE.endtime}
- Set Suite Variable $ORIG_ELAPSED ${SUITE.elapsedtime}
+ Set Suite Variable $ORIG_START ${SUITE.start_time}
+ Set Suite Variable $ORIG_END ${SUITE.end_time}
+ Set Suite Variable $ORIG_ELAPSED ${SUITE.elapsed_time}
Run Rebot ${EMPTY} ${INPUT1}
- Set Suite Variable $SINGLE_SUITE_ORIG_START ${SUITE.starttime}
- Set Suite Variable $SINGLE_SUITE_ORIG_END ${SUITE.endtime}
- Set Suite Variable $SINGLE_SUITE_ORIG_ELAPSED ${SUITE.elapsedtime}
+ Set Suite Variable $SINGLE_SUITE_ORIG_START ${SUITE.start_time}
+ Set Suite Variable $SINGLE_SUITE_ORIG_END ${SUITE.end_time}
+ Set Suite Variable $SINGLE_SUITE_ORIG_ELAPSED ${SUITE.elapsed_time}
diff --git a/atest/robot/rebot/xunit.robot b/atest/robot/rebot/xunit.robot
index c3b2293a391..81567c6960e 100644
--- a/atest/robot/rebot/xunit.robot
+++ b/atest/robot/rebot/xunit.robot
@@ -135,7 +135,7 @@ Remove Temps
Suite Stats Should Be
[Arguments] ${tests} ${failures} ${skipped}=0
- ... ${time}=?.??? ${timestamp}=20??-??-??T??:??:??.???000
+ ... ${time}=?.??? ${timestamp}=20??-??-??T??:??:??.??????
... ${xpath}=.
${suite} = Get Element ${OUTDIR}/xunit.xml xpath=${xpath}
Element Attribute Should Be ${suite} tests ${tests}
diff --git a/atest/robot/rpa/run_rpa_tasks.robot b/atest/robot/rpa/run_rpa_tasks.robot
index cd64a3a7221..b2d9b8a762e 100644
--- a/atest/robot/rpa/run_rpa_tasks.robot
+++ b/atest/robot/rpa/run_rpa_tasks.robot
@@ -5,7 +5,7 @@ Test Template Run and validate RPA tasks
Resource atest_resource.robot
*** Variables ***
-@{ALL TASKS} Task Another task Task Failing Passing Test
+@{ALL TASKS} Defaults Override Task Another task Task Failing Passing Test
... Defaults Override Task timeout exceeded Invalid task timeout
*** Test Cases ***
@@ -16,30 +16,30 @@ Task header in multiple files
${EMPTY} rpa/tasks1.robot rpa/tasks2.robot Task Failing Passing
Task header in directory
- ${EMPTY} rpa/tasks Task Another task
+ ${EMPTY} rpa/tasks Task Another task Defaults Override
Test header with --rpa
--rpa rpa/tests.robot Test
Task header with --norpa
[Template] Run and validate test cases
- --norpa rpa/tasks Task Another task
+ --norpa rpa/tasks Task Another task Defaults Override
Conflicting headers cause error
[Template] Run and validate conflict
- rpa/tests.robot rpa/tasks rpa/tasks/stuff.robot tasks tests
- rpa/ rpa/tests.robot tests tasks
- ... [[] ERROR ] Error in file '*[/\\]task_setup_teardown_template_timeout.robot' on line 6:
+ rpa/tests.robot rpa/tasks Rpa.Tests Rpa.Tasks tests tasks
+ rpa/ Rpa.Task Aliases Rpa.Tests tasks tests
+ ... [[] ERROR ] Error in file '*[/\\]task_aliases.robot' on line 7:
... Non-existing setting 'Tesk Setup'. Did you mean:\n
... ${SPACE*3}Test Setup\n
... ${SPACE*3}Task Setup\n
Conflicting headers with --rpa are fine
- --RPA rpa/tasks rpa/tests.robot Task Another task Test
+ --RPA rpa/tasks rpa/tests.robot Task Another task Defaults Override Test
Conflicting headers with --norpa are fine
[Template] Run and validate test cases
- --NorPA -v TIMEOUT:Test rpa/ @{ALL TASKS}
+ --NorPA -v TIMEOUT:Test -v RPA:False rpa/ @{ALL TASKS}
Conflicting headers in same file cause error
[Documentation] Using --rpa or --norpa doesn't affect the behavior.
@@ -66,11 +66,15 @@ Conflicting headers in same file cause error when executing directory
--task task rpa/tasks Task
--rpa --task Test --test "An* T???" rpa/ Another task Test
+Suite containing tests is ok if only tasks are selected
+ --task task rpa/tasks rpa/tests.robot Task
+ --suite stuff rpa/tasks rpa/tests.robot Task Another task
+
Error message is correct if no task match --task or other options
[Template] Run and validate no task found
- --task nonex matching name 'nonex'
- --include xxx --exclude yyy matching tag 'xxx' and not matching tag 'yyy'
- --suite nonex --task task matching name 'task' in suite 'nonex'
+ --rpa --task nonex no tasks matching name 'nonex'
+ --norpa --include xxx --exclude yyy no tests matching tag 'xxx' and not matching tag 'yyy'
+ --suite nonex --task task no tests or tasks matching name 'task' in suite 'nonex'
Error message is correct if task name is empty or task contains no keywords
[Template] NONE
@@ -92,20 +96,19 @@ Run and validate test cases
Should contain tests ${SUITE} @{tasks}
Run and validate conflict
- [Arguments] ${paths} ${conflicting} ${this} ${that} @{extra errors}
- Run tests without processing output ${EMPTY} ${paths}
- ${conflicting} = Normalize path ${DATADIR}/${conflicting}
+ [Arguments] ${paths} ${suite1} ${suite2} ${mode1} ${mode2} @{extra errors}
+ Run tests without processing output --name Rpa ${paths}
${extra} = Catenate @{extra errors}
${error} = Catenate
- ... [[] ERROR ] Parsing '${conflicting}' failed: Conflicting execution modes.
- ... File has ${this} but files parsed earlier have ${that}.
- ... Fix headers or use '--rpa' or '--norpa' options to set the execution mode explicitly.
+ ... [[] ERROR ] Conflicting execution modes:
+ ... Suite '${suite1}' has ${mode1} but suite '${suite2}' has ${mode2}.
+ ... Resolve the conflict or use '--rpa' or '--norpa' options to set the execution mode explicitly.
Stderr Should Match ${extra}${error}${USAGE TIP}\n
Run and validate no task found
[Arguments] ${options} ${message}
- Run tests without processing output --rpa ${options} rpa/tasks
- Stderr Should Be Equal To [ ERROR ] Suite 'Tasks' contains no tasks ${message}.${USAGE TIP}\n
+ Run tests without processing output ${options} rpa/tasks rpa/tests.robot
+ Stderr Should Be Equal To [ ERROR ] Suite 'Tasks & Tests' contains ${message}.${USAGE TIP}\n
Outputs should contain correct mode information
[Arguments] ${rpa}
diff --git a/atest/robot/rpa/task_aliases.robot b/atest/robot/rpa/task_aliases.robot
new file mode 100644
index 00000000000..533eab1baa1
--- /dev/null
+++ b/atest/robot/rpa/task_aliases.robot
@@ -0,0 +1,68 @@
+*** Settings ***
+Suite Setup Run Tests --loglevel DEBUG rpa/task_aliases.robot
+Resource atest_resource.robot
+
+*** Test Cases ***
+Defaults
+ ${tc} = Check Test Tags ${TESTNAME} task tags
+ Check timeout message ${tc.setup[0]} 1 minute 10 seconds
+ Check log message ${tc.setup[1]} Setup has an alias!
+ Check timeout message ${tc[0, 0]} 1 minute 10 seconds
+ Check log message ${tc[0, 1]} Using default settings
+ Check log message ${tc.teardown[0]} Also teardown has an alias!!
+ Should be equal ${tc.timeout} 1 minute 10 seconds
+
+Override
+ ${tc} = Check Test Tags ${TESTNAME} task tags own
+ Check log message ${tc.setup[0]} Overriding setup
+ Check log message ${tc[0, 0]} Overriding settings
+ Check log message ${tc.teardown[0]} Overriding teardown as well
+ Should be equal ${tc.timeout} ${NONE}
+
+Task timeout exceeded
+ ${tc} = Check Test Case ${TESTNAME}
+ Check timeout message ${tc[0, 0]} 99 milliseconds
+ Check log message ${tc[0, 1]} Task timeout 99 milliseconds exceeded. FAIL
+
+Invalid task timeout
+ Check Test Case ${TESTNAME}
+
+Task aliases are included in setting recommendations
+ Error In File
+ ... 0 rpa/task_aliases.robot 7
+ ... SEPARATOR=\n
+ ... Non-existing setting 'Tesk Setup'. Did you mean:
+ ... ${SPACE*4}Test Setup
+ ... ${SPACE*4}Task Setup
+
+Task settings are not allowed in resource file
+ [Template] Validate invalid setting error
+ 1 2 Task Setup
+ 2 3 Task Teardown
+ 3 4 Task Template
+ 4 5 Task Timeout
+
+In init file
+ Run Tests --loglevel DEBUG rpa/tasks
+ ${tc} = Check Test Tags Defaults file tag task tags
+ Check timeout message ${tc.setup[0]} 1 minute 10 seconds
+ Check log message ${tc.setup[1]} Setup has an alias!
+ Check timeout message ${tc[0, 0]} 1 minute 10 seconds
+ Check log message ${tc.teardown[0]} Also teardown has an alias!!
+ Should be equal ${tc.timeout} 1 minute 10 seconds
+ ${tc} = Check Test Tags Override file tag task tags own
+ Check log message ${tc.setup[0]} Overriding setup
+ Check log message ${tc.teardown[0]} Overriding teardown as well
+ Should be equal ${tc.timeout} ${NONE}
+ Should be empty ${ERRORS}
+
+*** Keywords ***
+Check timeout message
+ [Arguments] ${msg} ${timeout}
+ Check log message ${msg} Task timeout ${timeout} active. * seconds left. DEBUG pattern=True
+
+Validate invalid setting error
+ [Arguments] ${index} ${lineno} ${setting}
+ Error In File
+ ... ${index} rpa/resource_with_invalid_task_settings.robot ${lineno}
+ ... Setting '${setting}' is not allowed in resource file.
diff --git a/atest/robot/rpa/task_setup_teardown_template_timeout.robot b/atest/robot/rpa/task_setup_teardown_template_timeout.robot
deleted file mode 100644
index 1df6f414510..00000000000
--- a/atest/robot/rpa/task_setup_teardown_template_timeout.robot
+++ /dev/null
@@ -1,54 +0,0 @@
-*** Settings ***
-Suite Setup Run Tests --loglevel DEBUG rpa/task_setup_teardown_template_timeout.robot
-Resource atest_resource.robot
-
-*** Test Cases ***
-Defaults
- ${tc} = Check Test Case ${TESTNAME}
- Check timeout message ${tc.setup.msgs[0]} 1 minute 10 seconds
- Check log message ${tc.setup.msgs[1]} Setup has an alias!
- Check timeout message ${tc.kws[0].msgs[0]} 1 minute 10 seconds
- Check log message ${tc.kws[0].msgs[1]} Using default settings
- Check log message ${tc.teardown.msgs[0]} Also teardown has an alias!!
- Should be equal ${tc.timeout} 1 minute 10 seconds
-
-Override
- ${tc} = Check Test Case ${TESTNAME}
- Check log message ${tc.setup.msgs[0]} Overriding setup
- Check log message ${tc.kws[0].msgs[0]} Overriding settings
- Check log message ${tc.teardown.msgs[0]} Overriding teardown as well
- Should be equal ${tc.timeout} ${NONE}
-
-Task timeout exceeded
- ${tc} = Check Test Case ${TESTNAME}
- Check timeout message ${tc.kws[0].msgs[0]} 99 milliseconds
- Check log message ${tc.kws[0].msgs[1]} Task timeout 99 milliseconds exceeded. FAIL
-
-Invalid task timeout
- Check Test Case ${TESTNAME}
-
-Task aliases are included in setting recommendations
- Error In File
- ... 0 rpa/task_setup_teardown_template_timeout.robot 6
- ... SEPARATOR=\n
- ... Non-existing setting 'Tesk Setup'. Did you mean:
- ... ${SPACE*4}Test Setup
- ... ${SPACE*4}Task Setup
-
-Task settings are not allowed in resource file
- [Template] Validate invalid setting error
- 1 2 Task Setup
- 2 3 Task Teardown
- 3 4 Task Template
- 4 5 Task Timeout
-
-*** Keywords ***
-Check timeout message
- [Arguments] ${msg} ${timeout}
- Check log message ${msg} Task timeout ${timeout} active. * seconds left. DEBUG pattern=True
-
-Validate invalid setting error
- [Arguments] ${index} ${lineno} ${setting}
- Error In File
- ... ${index} rpa/resource_with_invalid_task_settings.robot ${lineno}
- ... Non-existing setting '${setting}'.
diff --git a/atest/robot/running/GetNestingLevel.py b/atest/robot/running/GetNestingLevel.py
new file mode 100644
index 00000000000..1f091c2a720
--- /dev/null
+++ b/atest/robot/running/GetNestingLevel.py
@@ -0,0 +1,21 @@
+from robot.api import SuiteVisitor
+
+
+class Nesting(SuiteVisitor):
+
+ def __init__(self):
+ self.level = 0
+ self.max = 0
+
+ def start_keyword(self, kw):
+ self.level += 1
+ self.max = max(self.level, self.max)
+
+ def end_keyword(self, kw):
+ self.level -= 1
+
+
+def get_nesting_level(test):
+ nesting = Nesting()
+ test.visit(nesting)
+ return nesting.max
diff --git a/atest/robot/running/continue_on_failure.robot b/atest/robot/running/continue_on_failure.robot
index bcffd3559c4..784d162727f 100644
--- a/atest/robot/running/continue_on_failure.robot
+++ b/atest/robot/running/continue_on_failure.robot
@@ -6,93 +6,93 @@ Resource atest_resource.robot
Continue in test
${tc}= Check Test Case ${TESTNAME}
- Check Log Message ${tc.kws[1].msgs[0]} This should be executed
+ Check Log Message ${tc[1, 0]} This should be executed
Continue in user keyword
${tc}= Check Test Case ${TESTNAME}
- Check Log Message ${tc.kws[0].kws[1].msgs[0]} This should be executed in Test Case
+ Check Log Message ${tc[0, 1, 0]} This should be executed in Test Case
Continue in test with several continuable failures
${tc}= Check Test Case ${TESTNAME}
- Check Log Message ${tc.kws[1].msgs[0]} This should be executed
- Check Log Message ${tc.kws[3].msgs[0]} This should also be executed
- Check Log Message ${tc.kws[5].msgs[0]} This too should also be executed
+ Check Log Message ${tc[1, 0]} This should be executed
+ Check Log Message ${tc[3, 0]} This should also be executed
+ Check Log Message ${tc[5, 0]} This too should also be executed
Continue in user keyword with several continuable failures
${tc}= Check Test Case ${TESTNAME}
- Verify all failures in user keyword ${tc.kws[0]} Test Case
- Verify all failures in user keyword ${tc.kws[1]} Test Case, Again
+ Verify all failures in user keyword ${tc[0]} Test Case
+ Verify all failures in user keyword ${tc[1]} Test Case, Again
Continuable and regular failure
${tc}= Check Test Case ${TESTNAME}
- Length Should Be ${tc.kws} 4
- Should Be Equal ${tc.kws[-1].status} NOT RUN
+ Length Should Be ${tc.body} 4
+ Should Be Equal ${tc[-1].status} NOT RUN
Continue in nested user keyword
${tc}= Check Test Case ${TESTNAME}
- Check Log Message ${tc.kws[0].kws[1].msgs[0]} This should be executed in Top Level UK (with ∏ön ÄßÇïï €§)
- Verify all failures in user keyword ${tc.kws[0].kws[2]} Nested UK
+ Check Log Message ${tc[0, 1, 0]} This should be executed in Top Level UK (with ∏ön ÄßÇïï €§)
+ Verify all failures in user keyword ${tc[0, 2]} Nested UK
Continuable and regular failure in UK
Check Test Case ${TESTNAME}
Several continuable failures and regular failure in nested UK
${tc}= Check Test Case ${TESTNAME}
- Verify all failures in user keyword ${tc.kws[0].kws[2]} Nested UK
- Verify all failures in user keyword ${tc.kws[1].kws[1].kws[2]} Nested UK
+ Verify all failures in user keyword ${tc[0, 2]} Nested UK
+ Verify all failures in user keyword ${tc[1, 1, 2]} Nested UK
Continue when setting variables
${tc}= Check Test Case ${TESTNAME}
- Check Log Message ${tc.kws[0].msgs[0]} \${ret} = None
- Check Log Message ${tc.kws[0].msgs[1]} ContinuableApocalypseException: Can be continued FAIL
- Check Log Message ${tc.kws[2].msgs[0]} \${r1} = None
- Check Log Message ${tc.kws[2].msgs[1]} \${r2} = None
- Check Log Message ${tc.kws[2].msgs[2]} \${r3} = None
- Check Log Message ${tc.kws[2].msgs[3]} ContinuableApocalypseException: Can be continued FAIL
- Check Log Message ${tc.kws[4].msgs[0]} \@{list} = [ ]
- Check Log Message ${tc.kws[4].msgs[1]} ContinuableApocalypseException: Can be continued FAIL
- Check Log Message ${tc.kws[6].msgs[0]} No jokes FAIL
- Length Should Be ${tc.kws[6].msgs} 1
+ Check Log Message ${tc[0, 0]} \${ret} = None
+ Check Log Message ${tc[0, 1]} ContinuableApocalypseException: Can be continued FAIL
+ Check Log Message ${tc[2, 0]} \${r1} = None
+ Check Log Message ${tc[2, 1]} \${r2} = None
+ Check Log Message ${tc[2, 2]} \${r3} = None
+ Check Log Message ${tc[2, 3]} ContinuableApocalypseException: Can be continued FAIL
+ Check Log Message ${tc[4, 0]} \@{list} = [ ]
+ Check Log Message ${tc[4, 1]} ContinuableApocalypseException: Can be continued FAIL
+ Check Log Message ${tc[6, 0]} No jokes FAIL
+ Length Should Be ${tc[6].body} 1
Continuable failure in user keyword returning value
Check Test Case ${TESTNAME}
Continue in test setup
${tc}= Check Test Case ${TESTNAME}
- Check Log Message ${tc.setup.kws[1].msgs[0]} This should be executed in Test Setup
- Should Be Empty ${tc.kws}
+ Check Log Message ${tc.setup[1, 0]} This should be executed in Test Setup
+ Should Be Empty ${tc.body}
Continue in test teardown
${tc}= Check Test Case ${TESTNAME}
- Check Log Message ${tc.teardown.kws[1].msgs[0]} This should be executed in Test Teardown
+ Check Log Message ${tc.teardown[1, 0]} This should be executed in Test Teardown
Continue many times in test setup and teardown
${tc}= Check Test Case ${TESTNAME}
- Verify all failures in user keyword ${tc.setup} Test Setup
- Should Be Empty ${tc.kws}
+ Verify all failures in user keyword ${tc.setup} Test Setup
+ Should Be Empty ${tc.body}
Verify all failures in user keyword ${tc.teardown} Test Teardown
Continue in suite teardown
${suite}= Get Test Suite Continue On Failure
- Check Log Message ${suite.teardown.kws[1].msgs[0]} This should be executed in Suite Teardown
+ Check Log Message ${suite.teardown[1, 0]} This should be executed in Suite Teardown
Continue in suite setup
${suite}= Get Test Suite Continue On Failure In Suite Setup
- Check Log Message ${suite.setup.kws[1].msgs[0]} This should be executed in Suite Setup (with ∏ön ÄßÇïï €§)
+ Check Log Message ${suite.setup[1, 0]} This should be executed in Suite Setup (with ∏ön ÄßÇïï €§)
Continue in for loop
${tc}= Check Test Case ${TESTNAME}
- Check Log Message ${tc.kws[0].kws[0].kws[0].msgs[0]} ContinuableApocalypseException: 0 FAIL
- Check Log Message ${tc.kws[0].kws[0].kws[1].msgs[0]} This should be executed inside for loop
- Check Log Message ${tc.kws[0].kws[1].kws[0].msgs[0]} ContinuableApocalypseException: 1 FAIL
- Check Log Message ${tc.kws[0].kws[1].kws[1].msgs[0]} This should be executed inside for loop
- Check Log Message ${tc.kws[0].kws[2].kws[0].msgs[0]} ContinuableApocalypseException: 2 FAIL
- Check Log Message ${tc.kws[0].kws[2].kws[1].msgs[0]} This should be executed inside for loop
- Check Log Message ${tc.kws[0].kws[3].kws[0].msgs[0]} ContinuableApocalypseException: 3 FAIL
- Check Log Message ${tc.kws[0].kws[3].kws[1].msgs[0]} This should be executed inside for loop
- Check Log Message ${tc.kws[0].kws[4].kws[0].msgs[0]} ContinuableApocalypseException: 4 FAIL
- Check Log Message ${tc.kws[0].kws[4].kws[1].msgs[0]} This should be executed inside for loop
- Check Log Message ${tc.kws[1].msgs[0]} This should be executed after for loop
+ Check Log Message ${tc[0, 0, 0, 0]} ContinuableApocalypseException: 0 FAIL
+ Check Log Message ${tc[0, 0, 1, 0]} This should be executed inside for loop
+ Check Log Message ${tc[0, 1, 0, 0]} ContinuableApocalypseException: 1 FAIL
+ Check Log Message ${tc[0, 1, 1, 0]} This should be executed inside for loop
+ Check Log Message ${tc[0, 2, 0, 0]} ContinuableApocalypseException: 2 FAIL
+ Check Log Message ${tc[0, 2, 1, 0]} This should be executed inside for loop
+ Check Log Message ${tc[0, 3, 0, 0]} ContinuableApocalypseException: 3 FAIL
+ Check Log Message ${tc[0, 3, 1, 0]} This should be executed inside for loop
+ Check Log Message ${tc[0, 4, 0, 0]} ContinuableApocalypseException: 4 FAIL
+ Check Log Message ${tc[0, 4, 1, 0]} This should be executed inside for loop
+ Check Log Message ${tc[1, 0]} This should be executed after for loop
Continuable and regular failure in for loop
Check Test Case ${TESTNAME}
@@ -102,9 +102,9 @@ robot.api.ContinuableFailure
*** Keywords ***
Verify all failures in user keyword [Arguments] ${kw} ${where}
- Check Log Message ${kw.kws[0].msgs[0]} ContinuableApocalypseException: 1 FAIL
- Check Log Message ${kw.kws[1].msgs[0]} This should be executed in ${where} (with ∏ön ÄßÇïï €§)
- Check Log Message ${kw.kws[2].msgs[0]} ContinuableApocalypseException: 2 FAIL
- Check Log Message ${kw.kws[3].msgs[0]} This should also be executed in ${where}
- Check Log Message ${kw.kws[4].msgs[0]} ContinuableApocalypseException: 3 FAIL
- Check Log Message ${kw.kws[5].msgs[0]} This too should also be executed in ${where}
+ Check Log Message ${kw[0, 0]} ContinuableApocalypseException: 1 FAIL
+ Check Log Message ${kw[1, 0]} This should be executed in ${where} (with ∏ön ÄßÇïï €§)
+ Check Log Message ${kw[2, 0]} ContinuableApocalypseException: 2 FAIL
+ Check Log Message ${kw[3, 0]} This should also be executed in ${where}
+ Check Log Message ${kw[4, 0]} ContinuableApocalypseException: 3 FAIL
+ Check Log Message ${kw[5, 0]} This too should also be executed in ${where}
diff --git a/atest/robot/running/continue_on_failure_tag.robot b/atest/robot/running/continue_on_failure_tag.robot
index 336ed2d7a1a..17443e98f2d 100644
--- a/atest/robot/running/continue_on_failure_tag.robot
+++ b/atest/robot/running/continue_on_failure_tag.robot
@@ -3,25 +3,25 @@ Suite Setup Run Tests ${EMPTY} running/continue_on_failure_tag.robot
Resource atest_resource.robot
*** Test Cases ***
-Continue in test with tag
+Continue in test with continue tag
Check Test Case ${TESTNAME}
Continue in test with Set Tags
Check Test Case ${TESTNAME}
-Continue in user keyword with tag
+Continue in user keyword with continue tag
Check Test Case ${TESTNAME}
-Continue in test with tag and UK without tag
+Continue in test with continue tag and UK without tag
Check Test Case ${TESTNAME}
-Continue in test with tag and nested UK with and without tag
+Continue in test with continue tag and nested UK with and without tag
Check Test Case ${TESTNAME}
-Continue in test with tag and two nested UK with tag
+Continue in test with continue tag and two nested UK with continue tag
Check Test Case ${TESTNAME}
-Continue in FOR loop with tag
+Continue in FOR loop with continue tag
Check Test Case ${TESTNAME}
Continue in FOR loop with Set Tags
@@ -30,13 +30,13 @@ Continue in FOR loop with Set Tags
No continue in FOR loop without tag
Check Test Case ${TESTNAME}
-Continue in FOR loop in UK with tag
+Continue in FOR loop in UK with continue tag
Check Test Case ${TESTNAME}
Continue in FOR loop in UK without tag
Check Test Case ${TESTNAME}
-Continue in IF with tag
+Continue in IF with continue tag
Check Test Case ${TESTNAME}
Continue in IF with set and remove tag
@@ -45,22 +45,28 @@ Continue in IF with set and remove tag
No continue in IF without tag
Check Test Case ${TESTNAME}
-Continue in IF in UK with tag
+Continue in IF in UK with continue tag
Check Test Case ${TESTNAME}
No continue in IF in UK without tag
Check Test Case ${TESTNAME}
-Continue in Run Keywords with tag
+Continue in Run Keywords with continue tag
Check Test Case ${TESTNAME}
-Recursive continue in test with tag and two nested UK without tag
+Recursive continue in test with continue tag and two nested UK without tag
Check Test Case ${TESTNAME}
Recursive continue in test with Set Tags and two nested UK without tag
Check Test Case ${TESTNAME}
-Recursive continue in test with tag and two nested UK with and without tag
+Recursive continue in test with continue tag and two nested UK with and without tag
+ Check Test Case ${TESTNAME}
+
+Recursive continue in test with continue tag and UK with stop tag
+ Check Test Case ${TESTNAME}
+
+Recursive continue in test with continue tag and UK with recursive stop tag
Check Test Case ${TESTNAME}
Recursive continue in user keyword
@@ -68,3 +74,69 @@ Recursive continue in user keyword
Recursive continue in nested keyword
Check Test Case ${TESTNAME}
+
+stop-on-failure in keyword in Teardown
+ Check Test Case ${TESTNAME}
+
+stop-on-failure with continuable failure in keyword in Teardown
+ Check Test Case ${TESTNAME}
+
+stop-on-failure with run-kw-and-continue failure in keyword in Teardown
+ Check Test Case ${TESTNAME}
+
+stop-on-failure with run-kw-and-continue failure in keyword
+ Check Test Case ${TESTNAME}
+
+Test teardown using run keywords with stop tag in test case
+ Check Test Case ${TESTNAME}
+
+Test teardown using user keyword with recursive stop tag in test case
+ Check Test Case ${TESTNAME}
+
+Test teardown using user keyword with stop tag in test case
+ Check Test Case ${TESTNAME}
+
+Test Teardown with stop tag in user keyword
+ Check Test Case ${TESTNAME}
+
+Test Teardown with recursive stop tag in user keyword
+ Check Test Case ${TESTNAME}
+
+Test Teardown with recursive stop tag and UK with continue tag
+ Check Test Case ${TESTNAME}
+
+Test Teardown with recursive stop tag and UK with recursive continue tag
+ Check Test Case ${TESTNAME}
+
+stop-on-failure with Template
+ Check Test Case ${TESTNAME}
+
+recursive-stop-on-failure with Template
+ Check Test Case ${TESTNAME}
+
+stop-on-failure with Template and Teardown
+ Check Test Case ${TESTNAME}
+
+stop-on-failure does not stop continuable failure in test
+ Check Test Case ${TESTNAME}
+
+Test recursive-continue-recursive-stop
+ Check Test Case ${TESTNAME}
+
+Test recursive-stop-recursive-continue
+ Check Test Case ${TESTNAME}
+
+Test recursive-stop-recursive-continue-recursive-stop
+ Check Test Case ${TESTNAME}
+
+Test test setup with continue-on-failure
+ Check Test Case ${TESTNAME}
+
+Test test setup with recursive-continue-on-failure
+ Check Test Case ${TESTNAME}
+
+recursive-stop-on-failure with continue-on-failure
+ Check Test Case ${TESTNAME}
+
+recursive-continue-on-failure with stop-on-failure
+ Check Test Case ${TESTNAME}
diff --git a/atest/robot/running/detect_recursion.robot b/atest/robot/running/detect_recursion.robot
new file mode 100644
index 00000000000..cd50dc5d443
--- /dev/null
+++ b/atest/robot/running/detect_recursion.robot
@@ -0,0 +1,41 @@
+*** Settings ***
+Suite Setup Run Tests ${EMPTY} running/detect_recursion.robot
+Library GetNestingLevel.py
+Resource atest_resource.robot
+
+*** Test Cases ***
+Infinite recursion
+ Check Test Case ${TESTNAME}
+
+Infinite cyclic recursion
+ Check Test Case ${TESTNAME}
+
+Infinite recursion with Run Keyword
+ Check Test Case ${TESTNAME}
+
+Infinitely recursive for loop
+ Check Test Case ${TESTNAME}
+
+Recursion below the recursion limit is ok
+ [Documentation] Also verifies that recursion limit blown earlier doesn't affect subsequent tests
+ Check Test Case ${TESTNAME}
+
+Recursion limit is over 140 started keywords
+ ${tc} = Check Test Case Infinite recursion
+ ${level} = Get Nesting Level ${tc}
+ Should Be True 140 < ${level} < 160
+
+Recursion limit can be raised with `sys.setrecursionlimit`
+ [Setup] Should Be True sys.getrecursionlimit() == 1000
+ # Raise limit with executed tests using sitecustomize.py.
+ Create File %{TEMPDIR}/sitecustomize.py import sys; sys.setrecursionlimit(1500)
+ Set Environment Variable PYTHONPATH %{TEMPDIR}
+ # Also raise limit here to be able to process created outputs.
+ Evaluate sys.setrecursionlimit(1500)
+ Run Tests -t "Infinite recursion" running/detect_recursion.robot
+ ${tc} = Check Test Case Infinite recursion
+ ${level} = Get Nesting Level ${tc}
+ Should Be True 220 < ${level} < 240
+ [Teardown] Run Keywords
+ ... Remove File %{TEMPDIR}/sitecustomize.py AND
+ ... Evaluate sys.setrecursionlimit(1000)
diff --git a/atest/robot/running/duplicate_suite_name.robot b/atest/robot/running/duplicate_suite_name.robot
index 3eedb5d31bc..6386a65e05e 100644
--- a/atest/robot/running/duplicate_suite_name.robot
+++ b/atest/robot/running/duplicate_suite_name.robot
@@ -1,8 +1,8 @@
-*** Setting ***
+*** Settings ***
Suite Setup Run Tests --exclude exclude running/duplicate_suite_name
Resource atest_resource.robot
-*** Test Case ***
+*** Test Cases ***
Suites with same name shoul be executed
Should Contain Suites ${SUITE}
... Test
diff --git a/atest/robot/running/duplicate_test_name.robot b/atest/robot/running/duplicate_test_name.robot
index 56f686b4e8a..68133471a24 100644
--- a/atest/robot/running/duplicate_test_name.robot
+++ b/atest/robot/running/duplicate_test_name.robot
@@ -1,26 +1,37 @@
-*** Setting ***
+*** Settings ***
Suite Setup Run Tests --exclude exclude running/duplicate_test_name.robot
Resource atest_resource.robot
-*** Test Case ***
-Tests with same name should be executed
+*** Test Cases ***
+Tests with same name are executed
Should Contain Tests ${SUITE}
- ... Same Test Multiple Times
- ... Same Test Multiple Times
- ... Same Test Multiple Times
- ... Same Test With Different Case And Spaces
- ... SameTestwith Different CASE and s p a c e s
- ... Same Test In Data But Only One Executed
+ ... Duplicates
+ ... Duplicates
+ ... Duplicates
+ ... Duplicates with different case and spaces
+ ... Duplicates with different CASE ands p a c e s
+ ... Duplicates but only one executed
+ ... Test 1 Test 2 Test 3
+ ... Duplicates after resolving variables
+ ... Duplicates after resolving variables
-There should be warning when multiple tests with same name are executed
- Check Multiple Tests Log Message ${ERRORS[0]} Same Test Multiple Times
- Check Multiple Tests Log Message ${ERRORS[1]} Same Test Multiple Times
- Check Multiple Tests Log Message ${ERRORS[2]} SameTestwith Different CASE and s p a c e s
+There is warning when multiple tests with same name are executed
+ Check Multiple Tests Log Message ${ERRORS[0]} Duplicates
+ Check Multiple Tests Log Message ${ERRORS[1]} Duplicates
+ Check Multiple Tests Log Message ${ERRORS[2]} Duplicates with different CASE ands p a c e s
-There should be no warning when there are multiple tests with same name in data but only one is executed
- ${tc} = Check Test Case Same Test In Data But Only One Executed
- Check Log Message ${tc.kws[0].msgs[0]} This is executed!
- Length Should Be ${ERRORS} 3
+There is warning if names are same after resolving variables
+ Check Multiple Tests Log Message ${ERRORS[3]} Duplicates after resolving variables
+
+There is no warning when there are multiple tests with same name but only one is executed
+ Check Test Case Duplicates but only one executed
+ Length Should Be ${ERRORS} 4
+
+Original name can be same if there is variable and its value changes
+ Check Test Case Test 1
+ Check Test Case Test 2
+ Check Test Case Test 3
+ Length Should Be ${ERRORS} 4
*** Keywords ***
Check Multiple Tests Log Message
diff --git a/atest/robot/running/exit_on_failure_tag.robot b/atest/robot/running/exit_on_failure_tag.robot
new file mode 100644
index 00000000000..f95649083ca
--- /dev/null
+++ b/atest/robot/running/exit_on_failure_tag.robot
@@ -0,0 +1,17 @@
+*** Settings ***
+Suite Setup Run Tests ${EMPTY} running/exit_on_failure_tag.robot
+Resource atest_resource.robot
+
+*** Test Cases ***
+Passing test with the tag has not special effect
+ Check Test Case ${TESTNAME}
+
+Failing test without the tag has no special effect
+ Check Test Case ${TESTNAME}
+
+Failing test with the tag initiates exit-on-failure
+ Check Test Case ${TESTNAME}
+
+Subsequent tests are not run
+ Check Test Case ${TESTNAME} 1
+ Check Test Case ${TESTNAME} 2
diff --git a/atest/robot/running/failures_in_teardown.robot b/atest/robot/running/failures_in_teardown.robot
index a38425dee86..175af344269 100644
--- a/atest/robot/running/failures_in_teardown.robot
+++ b/atest/robot/running/failures_in_teardown.robot
@@ -6,63 +6,83 @@ Resource atest_resource.robot
*** Test Cases ***
One Failure
${tc} = Check Test Case ${TESTNAME}
- Check Log Message ${tc.teardown.kws[1].msgs[0]} This should be executed
+ Check Log Message ${tc.teardown[1, 0]} This should be executed
Multiple Failures
${tc} = Check Test Case ${TESTNAME}
- Check Log Message ${tc.teardown.kws[2].msgs[0]} This should also be executed
+ Check Log Message ${tc.teardown[2, 0]} This should also be executed
Failure When Setting Variables
${tc} = Check Test Case ${TESTNAME}
- Check Log Message ${tc.teardown.kws[0].msgs[0]} \${ret} = None
- Check Log Message ${tc.teardown.kws[0].msgs[1]} Return values is None FAIL
+ Check Log Message ${tc.teardown[0, 0]} \${ret} = None
+ Check Log Message ${tc.teardown[0, 1]} Return values is None FAIL
Failure In For Loop
Check Test Case ${TESTNAME}
Execution Continues After Test Timeout
${tc} = Check Test Case ${TESTNAME}
- Should Be True ${tc.elapsedtime} >= 300
+ Elapsed Time Should Be Valid ${tc.elapsed_time} minimum=0.3
Execution Stops After Keyword Timeout
${tc} = Check Test Case ${TESTNAME}
- Length Should Be ${tc.teardown.kws} 2
- Should Be Equal ${tc.teardown.kws[-1].status} NOT RUN
+ Length Should Be ${tc.teardown.body} 2
+ Should Be Equal ${tc.teardown[-1].status} NOT RUN
-Execution Continues After Keyword Timeout Occurs In Executed Keyword
+Execution continues if executed keyword fails for keyword timeout
${tc} = Check Test Case ${TESTNAME}
- Length Should Be ${tc.teardown.body} 2
- Length Should Be ${tc.teardown.body[0].body} 2
- Should Be Equal ${tc.teardown.body[0].body[0].status} FAIL
- Should Be Equal ${tc.teardown.body[0].body[1].status} NOT RUN
- Should Be Equal ${tc.teardown.body[0].status} FAIL
- Should Be Equal ${tc.teardown.body[1].status} FAIL
+ Length Should Be ${tc.teardown.body} 2
+ Should Be Equal ${tc.teardown.body[0].status} FAIL
+ Should Be Equal ${tc.teardown.body[1].status} FAIL
+ Length Should Be ${tc.teardown.body[0].body} 2
+ Should Be Equal ${tc.teardown[0, 0].status} FAIL
+ Check Log Message ${tc.teardown}[0, 0, 0] Keyword timeout 42 milliseconds exceeded. FAIL
+ Should Be Equal ${tc.teardown[0, 1].status} NOT RUN
+ Length Should Be ${tc.teardown.body[1].body} 1
+ Check Log Message ${tc.teardown}[1, 0] This should be executed FAIL
+
+Execution stops after keyword timeout if keyword uses WUKS
+ ${tc} = Check Test Case ${TESTNAME}
+ Length Should Be ${tc.teardown.body} 2
+ Should Be Equal ${tc.teardown.body[0].status} FAIL
+ Should Be Equal ${tc.teardown.body[1].status} NOT RUN
+ Length Should Be ${tc.teardown.body[0].body} 2
+ Should Be Equal ${tc.teardown[0, 0].status} FAIL
+ Should Be Equal ${tc.teardown[0, 1].status} FAIL
+ Length Should Be ${tc.teardown[0, 0].body} 2
+ Should Be Equal ${tc.teardown[0, 0, 0].status} PASS
+ Should Be Equal ${tc.teardown[0, 0, 1].status} FAIL
+ Check Log Message ${tc.teardown}[0, 0, 1, 0] Failing! FAIL
+ Length Should Be ${tc.teardown[0, 1].body} 2
+ Should Be Equal ${tc.teardown[0, 1, 0].status} FAIL
+ Check Log Message ${tc.teardown}[0, 1, 0, 0] Keyword timeout 100 milliseconds exceeded. FAIL
+ Should Be Equal ${tc.teardown[0, 1, 1].status} NOT RUN
Execution Continues If Variable Does Not Exist
${tc} = Check Test Case ${TESTNAME}
- Length Should Be ${tc.teardown.kws} 3
+ Length Should Be ${tc.teardown.body} 3
Execution Continues After Keyword Errors
${tc} = Check Test Case ${TESTNAME}
- Length Should Be ${tc.teardown.kws} 3
+ Length Should Be ${tc.teardown.body} 3
Execution Stops After Syntax Error
${tc} = Check Test Case ${TESTNAME}
- Length Should Be ${tc.teardown.kws} 2
- Should Be Equal ${tc.teardown.kws[-1].status} NOT RUN
+ Length Should Be ${tc.teardown.body} 2
+ Should Be Equal ${tc.teardown[-1].status} NOT RUN
Fatal Error
${tc} = Check Test Case ${TESTNAME} 1
- Length Should Be ${tc.teardown.kws} 2
- Should Be Equal ${tc.teardown.kws[-1].status} NOT RUN
- Check Test Case ${TESTNAME} 2
+ Length Should Be ${tc.teardown.body} 2
+ Should Be Equal ${tc.teardown[-1].status} NOT RUN
+ Check Test Case ${TESTNAME} 2
Suite Teardown Is Executed Fully
${td} = Set Variable ${SUITE.teardown}
- Check Log Message ${td.kws[0].msgs[0]} Suite Message 1 FAIL
- Check Log Message ${td.kws[1].msgs[0]} Suite Message 2 (with ∏ön ÄßÇïï €§) FAIL
- Check Log Message ${td.kws[2].msgs[0]} Variable '\${it is ok not to exist}' not found. FAIL
- Check Log Message ${td.kws[3].msgs[0]} This should be executed
+ Check Log Message ${td[0, 0]} Suite Message 1 FAIL
+ Check Log Message ${td[1, 0]} Suite Message 2 (with ∏ön ÄßÇïï €§) FAIL
+ Check Log Message ${td[2, 0]} Variable '\${it is ok not to exist}' not found. FAIL
+ Check Log Message ${td[3, 0]} This should be executed
${msg} = Catenate SEPARATOR=\n\n
... Suite teardown failed:\nSeveral failures occurred:
... 1) Suite Message 1
@@ -73,5 +93,5 @@ Suite Teardown Is Executed Fully
Suite Teardown Should Stop At Fatal Error
Run Tests ${EMPTY} running/fatal_error_in_suite_teardown.robot
${ts} = Get Test Suite fatal error in suite teardown
- Length Should Be ${ts.teardown.kws} 2
- Should Be Equal ${ts.teardown.kws[-1].status} NOT RUN
+ Length Should Be ${ts.teardown.body} 2
+ Should Be Equal ${ts.teardown[-1].status} NOT RUN
diff --git a/atest/robot/running/fatal_exception.robot b/atest/robot/running/fatal_exception.robot
index 7166f538ad2..6c3898b42df 100644
--- a/atest/robot/running/fatal_exception.robot
+++ b/atest/robot/running/fatal_exception.robot
@@ -5,7 +5,7 @@ Resource atest_resource.robot
Exit From Python Keyword
Run Tests ${EMPTY} running/fatal_exception/01__python_library_kw.robot
${tc}= Check Test Case ${TESTNAME}
- Check Log Message ${tc.teardown.msgs[0]} This should be executed
+ Check Log Message ${tc.teardown[0]} This should be executed
Check Test Case Test That Should Not Be Run 1
robot.api.FatalError
@@ -42,7 +42,7 @@ Multiple Suite Aware Exiting From Suite Setup
Run Tests ${EMPTY} running/fatal_exception_suite_setup/
Check Test Case Test That Should Not Be Run 1
${ts1} = Get Test Suite Suite Setup
- Should End With ${ts1.teardown.msgs[0].message} Tearing down 1
+ Should End With ${ts1.teardown[0].message} Tearing down 1
Check Test Case Test That Should Not Be Run 2.1
Check Test Case Test That Should Not Be Run 2.2
${ts2} = Get Test Suite Irrelevant
diff --git a/atest/robot/running/flatten.robot b/atest/robot/running/flatten.robot
new file mode 100644
index 00000000000..dd3ab863fd9
--- /dev/null
+++ b/atest/robot/running/flatten.robot
@@ -0,0 +1,51 @@
+*** Settings ***
+Suite Setup Run Tests --loglevel trace --listener flatten_listener.Listener running/flatten.robot
+Resource atest_resource.robot
+
+*** Test Cases ***
+A single user keyword
+ ${tc}= User keyword content should be flattened 1
+ Check Log Message ${tc[0, 0]} From the main kw
+
+Nested UK
+ ${tc}= User keyword content should be flattened 2
+ Check Log Message ${tc[0, 0]} arg
+ Check Log Message ${tc[0, 1]} from nested kw
+
+Loops and stuff
+ ${tc}= User keyword content should be flattened 13
+ Check Log Message ${tc[0, 0]} inside for 0
+ Check Log Message ${tc[0, 1]} inside for 1
+ Check Log Message ${tc[0, 2]} inside for 2
+ Check Log Message ${tc[0, 3]} inside while 0
+ Check Log Message ${tc[0, 4]} \${LIMIT} = 1
+ Check Log Message ${tc[0, 5]} inside while 1
+ Check Log Message ${tc[0, 6]} \${LIMIT} = 2
+ Check Log Message ${tc[0, 7]} inside while 2
+ Check Log Message ${tc[0, 8]} \${LIMIT} = 3
+ Check Log Message ${tc[0, 9]} inside if
+ Check Log Message ${tc[0, 10]} fail inside try FAIL
+ Check Log Message ${tc[0, 11]} Traceback (most recent call last):* DEBUG pattern=True
+ Check Log Message ${tc[0, 12]} inside except
+
+Recursion
+ User keyword content should be flattened 8
+
+Listener methods start and end keyword are called
+ Stderr Should Be Empty
+
+Log levels
+ Run Tests ${EMPTY} running/flatten.robot
+ ${tc}= User keyword content should be flattened 4
+ Check Log Message ${tc[0, 0]} INFO 1
+ Check Log Message ${tc[0, 1]} Log level changed from INFO to DEBUG. DEBUG
+ Check Log Message ${tc[0, 2]} INFO 2
+ Check Log Message ${tc[0, 3]} DEBUG 2 level=DEBUG
+
+*** Keywords ***
+User keyword content should be flattened
+ [Arguments] ${expected_message_count}=0
+ ${tc}= Check Test Case ${TESTNAME}
+ Length Should Be ${tc[0].body} ${expected_message_count}
+ Length Should Be ${tc[0].messages} ${expected_message_count}
+ RETURN ${tc}
diff --git a/atest/robot/running/for/break_and_continue.robot b/atest/robot/running/for/break_and_continue.robot
index 262fdc5bf8d..e79998042f0 100644
--- a/atest/robot/running/for/break_and_continue.robot
+++ b/atest/robot/running/for/break_and_continue.robot
@@ -4,60 +4,56 @@ Resource for.resource
Test Template Test and all keywords should have passed
*** Test Cases ***
-With CONTINUE
+CONTINUE
allow not run=True
-With CONTINUE inside IF
- [Template] None
- ${tc}= Check test case ${TEST NAME}
- Should be FOR loop ${tc.body[0]} 5 FAIL IN RANGE
+CONTINUE inside IF
+ allow not run=True allowed failure=Oh no, got 4
-With CONTINUE inside TRY
+CONTINUE inside TRY
allow not run=True
-With CONTINUE inside EXCEPT and TRY-ELSE
- allow not run=True
+CONTINUE inside EXCEPT and TRY-ELSE
+ allow not run=True allowed failure=4 == 4
-With BREAK
+BREAK
allow not run=True
-With BREAK inside IF
+BREAK inside IF
allow not run=True
-With BREAK inside TRY
+BREAK inside TRY
allow not run=True
-With BREAK inside EXCEPT
- allow not run=True
+BREAK inside EXCEPT
+ allow not run=True allowed failure=This is excepted!
-With BREAK inside TRY-ELSE
+BREAK inside TRY-ELSE
allow not run=True
-With CONTINUE in UK
+CONTINUE in UK
allow not run=True
-With CONTINUE inside IF in UK
- [Template] None
- ${tc}= Check test case ${TEST NAME}
- Should be FOR loop ${tc.body[0].body[0]} 5 FAIL IN RANGE
+CONTINUE inside IF in UK
+ allow not run=True allowed failure=Oh no, got 4
-With CONTINUE inside TRY in UK
+CONTINUE inside TRY in UK
allow not run=True
-With CONTINUE inside EXCEPT and TRY-ELSE in UK
- allow not run=True
+CONTINUE inside EXCEPT and TRY-ELSE in UK
+ allow not run=True allowed failure=4 == 4
-With BREAK in UK
+BREAK in UK
allow not run=True
-With BREAK inside IF in UK
+BREAK inside IF in UK
allow not run=True
-With BREAK inside TRY in UK
+BREAK inside TRY in UK
allow not run=True
-With BREAK inside EXCEPT in UK
- allow not run=True
+BREAK inside EXCEPT in UK
+ allow not run=True allowed failure=This is excepted!
-With BREAK inside TRY-ELSE in UK
+BREAK inside TRY-ELSE in UK
allow not run=True
diff --git a/atest/robot/running/for/continue_for_loop.robot b/atest/robot/running/for/continue_for_loop.robot
index 5f50a6bd23e..59ab08e67b4 100644
--- a/atest/robot/running/for/continue_for_loop.robot
+++ b/atest/robot/running/for/continue_for_loop.robot
@@ -26,8 +26,8 @@ Continue For Loop In User Keyword Without For Loop Should Fail
Continue For Loop Keyword Should Log Info
${tc} = Check Test Case Simple Continue For Loop
- Should Be Equal ${tc.kws[0].kws[0].kws[0].name} BuiltIn.Continue For Loop
- Check Log Message ${tc.kws[0].kws[0].kws[0].msgs[0]} Continuing for loop from the next iteration.
+ Should Be Equal ${tc[0, 0, 0].full_name} BuiltIn.Continue For Loop
+ Check Log Message ${tc[0, 0, 0, 0]} Continuing for loop from the next iteration.
Continue For Loop In Test Teardown
Test And All Keywords Should Have Passed
diff --git a/atest/robot/running/for/exit_for_loop.robot b/atest/robot/running/for/exit_for_loop.robot
index 00d3062c9de..aff02d26484 100644
--- a/atest/robot/running/for/exit_for_loop.robot
+++ b/atest/robot/running/for/exit_for_loop.robot
@@ -29,8 +29,8 @@ Exit For Loop In User Keyword Without For Loop Should Fail
Exit For Loop Keyword Should Log Info
${tc} = Check Test Case Simple Exit For Loop
- Should Be Equal ${tc.kws[0].kws[0].kws[0].name} BuiltIn.Exit For Loop
- Check Log Message ${tc.kws[0].kws[0].kws[0].msgs[0]} Exiting for loop altogether.
+ Should Be Equal ${tc[0, 0, 0].full_name} BuiltIn.Exit For Loop
+ Check Log Message ${tc[0, 0, 0, 0]} Exiting for loop altogether.
Exit For Loop In Test Teardown
Test And All Keywords Should Have Passed
diff --git a/atest/robot/running/for/for.resource b/atest/robot/running/for/for.resource
index 258045e7995..9b29093eda6 100644
--- a/atest/robot/running/for/for.resource
+++ b/atest/robot/running/for/for.resource
@@ -5,33 +5,40 @@ Resource atest_resource.robot
Check test and get loop
[Arguments] ${test name} ${loop index}=0
${tc} = Check Test Case ${test name}
- [Return] ${tc.kws}[${loop index}]
+ RETURN ${tc.body}[${loop index}]
Check test and failed loop
- [Arguments] ${test name} ${type}=FOR ${loop index}=0
+ [Arguments] ${test name} ${type}=FOR ${loop index}=0 &{config}
${loop} = Check test and get loop ${test name} ${loop index}
- Run Keyword Should Be ${type} loop ${loop} 0 FAIL
+ Length Should Be ${loop.body} 2
+ Should Be Equal ${loop[0].type} ITERATION
+ Should Be Equal ${loop[1].type} MESSAGE
+ Run Keyword Should Be ${type} loop ${loop} 1 FAIL &{config}
Should be FOR loop
[Arguments] ${loop} ${iterations} ${status}=PASS ${flavor}=IN
- Should Be Equal ${loop.type} FOR
- Should Be Equal ${loop.flavor} ${flavor}
- Length Should Be ${loop.kws} ${iterations}
- Should Be Equal ${loop.status} ${status}
+ ... ${start}=${None} ${mode}=${None} ${fill}=${None}
+ Should Be Equal ${loop.type} FOR
+ Should Be Equal ${loop.flavor} ${flavor}
+ Should Be Equal ${loop.start} ${start}
+ Should Be Equal ${loop.mode} ${mode}
+ Should Be Equal ${loop.fill} ${fill}
+ Length Should Be ${loop.non_messages} ${iterations}
+ Should Be Equal ${loop.status} ${status}
Should be IN RANGE loop
[Arguments] ${loop} ${iterations} ${status}=PASS
- Should Be FOR Loop ${loop} ${iterations} ${status} flavor=IN RANGE
+ Should Be FOR Loop ${loop} ${iterations} ${status} IN RANGE
Should be IN ZIP loop
- [Arguments] ${loop} ${iterations} ${status}=PASS
- Should Be FOR Loop ${loop} ${iterations} ${status} flavor=IN ZIP
+ [Arguments] ${loop} ${iterations} ${status}=PASS ${mode}=${None} ${fill}=${None}
+ Should Be FOR Loop ${loop} ${iterations} ${status} IN ZIP mode=${mode} fill=${fill}
Should be IN ENUMERATE loop
- [Arguments] ${loop} ${iterations} ${status}=PASS
- Should Be FOR Loop ${loop} ${iterations} ${status} flavor=IN ENUMERATE
+ [Arguments] ${loop} ${iterations} ${status}=PASS ${start}=${None}
+ Should Be FOR Loop ${loop} ${iterations} ${status} IN ENUMERATE start=${start}
Should be FOR iteration
- [Arguments] ${iteration} &{variables}
+ [Arguments] ${iteration} &{assign}
Should Be Equal ${iteration.type} ITERATION
- Should Be Equal ${iteration.variables} ${variables}
+ Should Be Equal ${iteration.assign} ${assign}
diff --git a/atest/robot/running/for/for.robot b/atest/robot/running/for/for.robot
index 4551f2f371f..93e7769b44b 100644
--- a/atest/robot/running/for/for.robot
+++ b/atest/robot/running/for/for.robot
@@ -1,28 +1,28 @@
*** Settings ***
-Suite Setup Run Tests ${EMPTY} running/for/for.robot
+Suite Setup Run Tests --log log-tests-also-string-reprs.html running/for/for.robot
+Suite Teardown File Should Exist ${OUTDIR}/log-tests-also-string-reprs.html
Resource for.resource
*** Test Cases ***
Simple loop
- ${tc} = Check test case ${TEST NAME}
- ${loop} = Set variable ${tc.body[1]}
- Check log message ${tc.body[0].msgs[0]} Not yet in FOR
- Should be FOR loop ${loop} 2
- Should be FOR iteration ${loop.body[0]} \${var}=one
- Check log message ${loop.body[0].body[0].msgs[0]} var: one
- Should be FOR iteration ${loop.body[1]} \${var}=two
- Check log message ${loop.body[1].body[0].msgs[0]} var: two
- Check log message ${tc.body[2].body[0]} Not in FOR anymore
+ ${tc} = Check Test Case ${TEST NAME}
+ Check Log Message ${tc[0, 0]} Not yet in FOR
+ Should be FOR loop ${tc[1]} 2
+ Should be FOR iteration ${tc[1, 0]} \${var}=one
+ Check Log Message ${tc[1, 0, 0, 0]} var: one
+ Should be FOR iteration ${tc[1, 1]} \${var}=two
+ Check Log Message ${tc[1, 1, 0, 0]} var: two
+ Check Log Message ${tc[2, 0]} Not in FOR anymore
Variables in values
${loop} = Check test and get loop ${TEST NAME}
- Should be FOR loop ${loop} 6
- "Variables in values" helper ${loop.kws[0]} 1
- "Variables in values" helper ${loop.kws[1]} 2
- "Variables in values" helper ${loop.kws[2]} 3
- "Variables in values" helper ${loop.kws[3]} 4
- "Variables in values" helper ${loop.kws[4]} 5
- "Variables in values" helper ${loop.kws[5]} 6
+ Should be FOR loop ${loop} 6
+ "Variables in values" helper ${loop[0]} 1
+ "Variables in values" helper ${loop[1]} 2
+ "Variables in values" helper ${loop[2]} 3
+ "Variables in values" helper ${loop[3]} 4
+ "Variables in values" helper ${loop[4]} 5
+ "Variables in values" helper ${loop[5]} 6
Indentation is not required
${loop} = Check test and get loop ${TEST NAME} 1
@@ -30,203 +30,203 @@ Indentation is not required
Values on multiple rows
${loop} = Check test and get loop ${TEST NAME}
- Should be FOR loop ${loop} 10
- Check log message ${loop.kws[0].kws[0].msgs[0]} 1
+ Should be FOR loop ${loop} 10
+ Check Log Message ${loop[0, 0, 0]} 1
FOR ${i} IN RANGE 10
- Check log message ${loop.kws[${i}].kws[0].msgs[0]} ${{str($i + 1)}}
+ Check Log Message ${loop[${i}, 0, 0]} ${{str($i + 1)}}
END
# Sanity check
- Check log message ${loop.kws[0].kws[0].msgs[0]} 1
- Check log message ${loop.kws[4].kws[0].msgs[0]} 5
- Check log message ${loop.kws[9].kws[0].msgs[0]} 10
+ Check Log Message ${loop[0, 0, 0]} 1
+ Check Log Message ${loop[4, 0, 0]} 5
+ Check Log Message ${loop[9, 0, 0]} 10
Keyword arguments on multiple rows
${loop} = Check test and get loop ${TEST NAME}
- Should be FOR loop ${loop} 2
- Check log message ${loop.kws[0].kws[1].msgs[0]} 1 2 3 4 5 6 7 one
- Check log message ${loop.kws[1].kws[1].msgs[0]} 1 2 3 4 5 6 7 two
+ Should be FOR loop ${loop} 2
+ Check Log Message ${loop[0, 1, 0]} 1 2 3 4 5 6 7 one
+ Check Log Message ${loop[1, 1, 0]} 1 2 3 4 5 6 7 two
Multiple loops in a test
- ${tc} = Check test case ${TEST NAME}
- Should be FOR loop ${tc.kws[0]} 2
- Check log message ${tc.kws[0].kws[0].kws[0].msgs[0]} In first loop with "foo"
- Check log message ${tc.kws[0].kws[1].kws[0].msgs[0]} In first loop with "bar"
- Should be FOR loop ${tc.kws[1]} 1
- Check kw "My UK 2" ${tc.kws[1].kws[0].kws[0]} Hello, world!
- Check log message ${tc.kws[2].msgs[0]} Outside loop
- Should be FOR loop ${tc.kws[3]} 2
- Check log message ${tc.kws[3].kws[0].kws[0].msgs[0]} Third loop
- Check log message ${tc.kws[3].kws[0].kws[2].msgs[0]} Value: a
- Check log message ${tc.kws[3].kws[1].kws[0].msgs[0]} Third loop
- Check log message ${tc.kws[3].kws[1].kws[2].msgs[0]} Value: b
- Check log message ${tc.kws[4].msgs[0]} The End
+ ${tc} = Check Test Case ${TEST NAME}
+ Should be FOR loop ${tc[0]} 2
+ Check Log Message ${tc[0, 0, 0, 0]} In first loop with "foo"
+ Check Log Message ${tc[0, 1, 0, 0]} In first loop with "bar"
+ Should be FOR loop ${tc[1]} 1
+ Check kw "My UK 2" ${tc[1, 0, 0]} Hello, world!
+ Check Log Message ${tc[2, 0]} Outside loop
+ Should be FOR loop ${tc[3]} 2
+ Check Log Message ${tc[3, 0, 0, 0]} Third loop
+ Check Log Message ${tc[3, 0, 2, 0]} Value: a
+ Check Log Message ${tc[3, 1, 0, 0]} Third loop
+ Check Log Message ${tc[3, 1, 2, 0]} Value: b
+ Check Log Message ${tc[4, 0]} The End
Nested loop syntax
- ${tc} = Check test case ${TEST NAME}
- Should be FOR loop ${tc.kws[0]} 3
- Should be FOR loop ${tc.kws[0].kws[0].kws[1]} 3
- Check log message ${tc.kws[0].kws[0].kws[0].msgs[0]} 1 in
- Check log message ${tc.kws[0].kws[0].kws[1].kws[0].kws[0].msgs[0]} values 1 a
- Check log message ${tc.kws[0].kws[0].kws[1].kws[1].kws[0].msgs[0]} values 1 b
- Check log message ${tc.kws[0].kws[0].kws[1].kws[2].kws[0].msgs[0]} values 1 c
- Check log message ${tc.kws[0].kws[0].kws[2].msgs[0]} 1 out
- Check log message ${tc.kws[0].kws[1].kws[0].msgs[0]} 2 in
- Check log message ${tc.kws[0].kws[1].kws[1].kws[0].kws[0].msgs[0]} values 2 a
- Check log message ${tc.kws[0].kws[1].kws[1].kws[1].kws[0].msgs[0]} values 2 b
- Check log message ${tc.kws[0].kws[1].kws[1].kws[2].kws[0].msgs[0]} values 2 c
- Check log message ${tc.kws[0].kws[1].kws[2].msgs[0]} 2 out
- Check log message ${tc.kws[0].kws[2].kws[0].msgs[0]} 3 in
- Check log message ${tc.kws[0].kws[2].kws[1].kws[0].kws[0].msgs[0]} values 3 a
- Check log message ${tc.kws[0].kws[2].kws[1].kws[1].kws[0].msgs[0]} values 3 b
- Check log message ${tc.kws[0].kws[2].kws[1].kws[2].kws[0].msgs[0]} values 3 c
- Check log message ${tc.kws[0].kws[2].kws[2].msgs[0]} 3 out
- Check log message ${tc.kws[1].msgs[0]} The End
+ ${tc} = Check Test Case ${TEST NAME}
+ Should be FOR loop ${tc[0]} 3
+ Should be FOR loop ${tc[0, 0, 1]} 3
+ Check Log Message ${tc[0, 0, 0, 0]} 1 in
+ Check Log Message ${tc[0, 0, 1, 0, 0, 0]} values 1 a
+ Check Log Message ${tc[0, 0, 1, 1, 0, 0]} values 1 b
+ Check Log Message ${tc[0, 0, 1, 2, 0, 0]} values 1 c
+ Check Log Message ${tc[0, 0, 2, 0]} 1 out
+ Check Log Message ${tc[0, 1, 0, 0]} 2 in
+ Check Log Message ${tc[0, 1, 1, 0, 0, 0]} values 2 a
+ Check Log Message ${tc[0, 1, 1, 1, 0, 0]} values 2 b
+ Check Log Message ${tc[0, 1, 1, 2, 0, 0]} values 2 c
+ Check Log Message ${tc[0, 1, 2, 0]} 2 out
+ Check Log Message ${tc[0, 2, 0, 0]} 3 in
+ Check Log Message ${tc[0, 2, 1, 0, 0, 0]} values 3 a
+ Check Log Message ${tc[0, 2, 1, 1, 0, 0]} values 3 b
+ Check Log Message ${tc[0, 2, 1, 2, 0, 0]} values 3 c
+ Check Log Message ${tc[0, 2, 2, 0]} 3 out
+ Check Log Message ${tc[1, 0]} The End
Multiple loops in a loop
- Check test case ${TEST NAME}
+ Check Test Case ${TEST NAME}
Deeply nested loops
- Check test case ${TEST NAME}
+ Check Test Case ${TEST NAME}
Settings after FOR
- ${tc} = Check test case ${TEST NAME}
- Should be FOR loop ${tc.kws[0]} 1
- Check log message ${tc.teardown.msgs[0]} Teardown was found and eXecuted.
+ ${tc} = Check Test Case ${TEST NAME}
+ Should be FOR loop ${tc[0]} 1
+ Check Log Message ${tc.teardown[0]} Teardown was found and eXecuted.
Looping over empty list variable is OK
- ${tc} = Check test case ${TEST NAME}
- Should be FOR loop ${tc.kws[0]} 1 NOT RUN
- Should be FOR iteration ${tc.body[0].body[0]} \${var}=
- Check keyword data ${tc.body[0].body[0].body[0]} BuiltIn.Fail args=Not executed status=NOT RUN
+ ${tc} = Check Test Case ${TEST NAME}
+ Should be FOR loop ${tc[0]} 1 NOT RUN
+ Should be FOR iteration ${tc[0, 0]} \${var}=
+ Check keyword data ${tc[0, 0, 0]} BuiltIn.Fail args=Not executed status=NOT RUN
Other iterables
- ${tc} = Check test case ${TEST NAME}
- Should be FOR loop ${tc.kws[2]} 10
+ ${tc} = Check Test Case ${TEST NAME}
+ Should be FOR loop ${tc[2]} 10
Failure inside FOR
${loop} = Check test and get loop ${TEST NAME} 1
- Should be FOR loop ${loop} 1 FAIL
- Check log message ${loop.kws[0].kws[0].msgs[0]} Hello before failing kw
- Should be equal ${loop.kws[0].kws[0].status} PASS
- Check log message ${loop.kws[0].kws[1].msgs[0]} Here we fail! FAIL
- Should be equal ${loop.kws[0].kws[1].status} FAIL
- Should be equal ${loop.kws[0].kws[2].status} NOT RUN
- Should be equal ${loop.kws[0].status} FAIL
- Length should be ${loop.kws[0].kws} 3
+ Should be FOR loop ${loop} 1 FAIL
+ Check Log Message ${loop[0, 0, 0]} Hello before failing kw
+ Should Be Equal ${loop[0, 0].status} PASS
+ Check Log Message ${loop[0, 1, 0]} Here we fail! FAIL
+ Should Be Equal ${loop[0, 1].status} FAIL
+ Should Be Equal ${loop[0, 2].status} NOT RUN
+ Should Be Equal ${loop[0].status} FAIL
+ Length Should Be ${loop[0].body} 3
${loop} = Check test and get loop ${TEST NAME} 2
- Should be FOR loop ${loop} 4 FAIL
- Check log message ${loop.kws[0].kws[0].msgs[0]} Before Check
- Check log message ${loop.kws[0].kws[2].msgs[0]} After Check
- Length should be ${loop.kws[0].kws} 3
- Should be equal ${loop.kws[0].status} PASS
- Should be equal ${loop.kws[1].status} PASS
- Should be equal ${loop.kws[2].status} PASS
- Check log message ${loop.kws[3].kws[0].msgs[0]} Before Check
- Check log message ${loop.kws[3].kws[1].msgs[0]} Failure with 4 FAIL
- Should be equal ${loop.kws[3].kws[2].status} NOT RUN
- Length should be ${loop.kws[3].kws} 3
- Should be equal ${loop.kws[3].status} FAIL
+ Should be FOR loop ${loop} 4 FAIL
+ Check Log Message ${loop[0, 0, 0]} Before Check
+ Check Log Message ${loop[0, 2, 0]} After Check
+ Length Should Be ${loop[0].body} 3
+ Should Be Equal ${loop[0].status} PASS
+ Should Be Equal ${loop[1].status} PASS
+ Should Be Equal ${loop[2].status} PASS
+ Check Log Message ${loop[3, 0, 0]} Before Check
+ Check Log Message ${loop[3, 1, 0]} Failure with <4> FAIL
+ Should Be Equal ${loop[3, 2].status} NOT RUN
+ Length Should Be ${loop[3].body} 3
+ Should Be Equal ${loop[3].status} FAIL
Loop with user keywords
${loop} = Check test and get loop ${TEST NAME}
Should be FOR loop ${loop} 2
- Check kw "My UK" ${loop.kws[0].kws[0]}
- Check kw "My UK 2" ${loop.kws[0].kws[1]} foo
- Check kw "My UK" ${loop.kws[1].kws[0]}
- Check kw "My UK 2" ${loop.kws[1].kws[1]} bar
+ Check kw "My UK" ${loop[0, 0]}
+ Check kw "My UK 2" ${loop[0, 1]} foo
+ Check kw "My UK" ${loop[1, 0]}
+ Check kw "My UK 2" ${loop[1, 1]} bar
Loop with failures in user keywords
- ${tc} = Check test case ${TEST NAME}
- Should be FOR loop ${tc.kws[0]} 2 FAIL
+ ${tc} = Check Test Case ${TEST NAME}
+ Should be FOR loop ${tc[0]} 2 FAIL
Loop in user keyword
- ${tc} = Check test case ${TEST NAME}
- Check kw "For In UK" ${tc.kws[0]}
- Check kw "For In UK with Args" ${tc.kws[1]} 4 one
+ ${tc} = Check Test Case ${TEST NAME}
+ Check kw "For In UK" ${tc[0]}
+ Check kw "For In UK with Args" ${tc[1]} 4 one
Keyword with loop calling other keywords with loops
- ${tc} = Check test case ${TEST NAME}
- Check kw "Nested For In UK" ${tc.kws[0]} foo
+ ${tc} = Check Test Case ${TEST NAME}
+ Check kw "Nested For In UK" ${tc[0]} foo
Test with loop calling keywords with loops
${loop} = Check test and get loop ${TEST NAME} 1
- Should be FOR loop ${loop} 1 FAIL
- Check kw "For In UK" ${loop.kws[0].kws[0]}
- Check kw "For In UK with Args" ${loop.kws[0].kws[1]} 2 one
- Check kw "Nested For In UK" ${loop.kws[0].kws[2]} one
+ Should be FOR loop ${loop} 1 FAIL
+ Check kw "For In UK" ${loop[0, 0]}
+ Check kw "For In UK with Args" ${loop[0, 1]} 2 one
+ Check kw "Nested For In UK" ${loop[0, 2]} one
Loop variables is available after loop
- Check test case ${TEST NAME}
+ Check Test Case ${TEST NAME}
Assign inside loop
${loop} = Check test and get loop ${TEST NAME}
Should be FOR loop ${loop} 2
- Check log message ${loop.kws[0].kws[0].msgs[0]} \${v1} = v1
- Check log message ${loop.kws[0].kws[1].msgs[0]} \${v2} = v2
- Check log message ${loop.kws[0].kws[1].msgs[1]} \${v3} = vY
- Check log message ${loop.kws[0].kws[2].msgs[0]} \@{list} = [ v1 | v2 | vY | Y ]
- Check log message ${loop.kws[1].kws[0].msgs[0]} \${v1} = v1
- Check log message ${loop.kws[1].kws[1].msgs[0]} \${v2} = v2
- Check log message ${loop.kws[1].kws[1].msgs[1]} \${v3} = vZ
- Check log message ${loop.kws[1].kws[2].msgs[0]} \@{list} = [ v1 | v2 | vZ | Z ]
+ Check Log Message ${loop[0, 0, 0]} \${v1} = v1
+ Check Log Message ${loop[0, 1, 0]} \${v2} = v2
+ Check Log Message ${loop[0, 1, 1]} \${v3} = vY
+ Check Log Message ${loop[0, 2, 0]} \@{list} = [ v1 | v2 | vY | Y ]
+ Check Log Message ${loop[1, 0, 0]} \${v1} = v1
+ Check Log Message ${loop[1, 1, 0]} \${v2} = v2
+ Check Log Message ${loop[1, 1, 1]} \${v3} = vZ
+ Check Log Message ${loop[1, 2, 0]} \@{list} = [ v1 | v2 | vZ | Z ]
Invalid assign inside loop
- ${tc} = Check test case ${TEST NAME}
- Should be FOR loop ${tc.kws[0]} 1 FAIL
+ ${tc} = Check Test Case ${TEST NAME}
+ Should be FOR loop ${tc[0]} 1 FAIL
Loop with non-existing keyword
- Check test case ${TEST NAME}
+ Check Test Case ${TEST NAME}
Loop with non-existing variable
- Check test case ${TEST NAME}
+ Check Test Case ${TEST NAME}
Loop value with non-existing variable
- Check test case ${TEST NAME}
+ Check Test Case ${TEST NAME}
Multiple loop variables
${tc} = Check Test Case ${TEST NAME}
- ${loop} = Set Variable ${tc.body[0]}
- Should be FOR loop ${loop} 4
- Should be FOR iteration ${loop.body[0]} \${x}=1 \${y}=a
- Check log message ${loop.body[0].body[0].msgs[0]} 1a
- Should be FOR iteration ${loop.body[1]} \${x}=2 \${y}=b
- Check log message ${loop.body[1].body[0].msgs[0]} 2b
- Should be FOR iteration ${loop.body[2]} \${x}=3 \${y}=c
- Check log message ${loop.body[2].body[0].msgs[0]} 3c
- Should be FOR iteration ${loop.body[3]} \${x}=4 \${y}=d
- Check log message ${loop.body[3].body[0].msgs[0]} 4d
- ${loop} = Set Variable ${tc.body[2]}
- Should be FOR loop ${loop} 2
- Should be FOR iteration ${loop.body[0]} \${a}=1 \${b}=2 \${c}=3 \${d}=4 \${e}=5
- Should be FOR iteration ${loop.body[1]} \${a}=1 \${b}=2 \${c}=3 \${d}=4 \${e}=5
+ ${loop} = Set Variable ${tc[0]}
+ Should be FOR loop ${loop} 4
+ Should be FOR iteration ${loop[0]} \${x}=1 \${y}=a
+ Check Log Message ${loop[0, 0, 0]} 1a
+ Should be FOR iteration ${loop[1]} \${x}=2 \${y}=b
+ Check Log Message ${loop[1, 0, 0]} 2b
+ Should be FOR iteration ${loop[2]} \${x}=3 \${y}=c
+ Check Log Message ${loop[2, 0, 0]} 3c
+ Should be FOR iteration ${loop[3]} \${x}=4 \${y}=d
+ Check Log Message ${loop[3, 0, 0]} 4d
+ ${loop} = Set Variable ${tc[2]}
+ Should be FOR loop ${loop} 2
+ Should be FOR iteration ${loop[0]} \${a}=1 \${b}=2 \${c}=3 \${d}=4 \${e}=5
+ Should be FOR iteration ${loop[1]} \${a}=1 \${b}=2 \${c}=3 \${d}=4 \${e}=5
Wrong number of loop variables
Check test and failed loop ${TEST NAME} 1
Check test and failed loop ${TEST NAME} 2
Cut long iteration variable values
- ${tc} = Check test case ${TEST NAME}
- ${loop} = Set Variable ${tc.body[6]}
+ ${tc} = Check Test Case ${TEST NAME}
+ ${loop} = Set Variable ${tc[6]}
${exp10} = Set Variable 0123456789
${exp100} = Evaluate "${exp10}" * 10
${exp200} = Evaluate "${exp10}" * 20
${exp200+} = Set Variable ${exp200}...
- Should be FOR loop ${loop} 6
- Should be FOR iteration ${loop.body[0]} \${var}=${exp10}
- Should be FOR iteration ${loop.body[1]} \${var}=${exp100}
- Should be FOR iteration ${loop.body[2]} \${var}=${exp200}
- Should be FOR iteration ${loop.body[3]} \${var}=${exp200+}
- Should be FOR iteration ${loop.body[4]} \${var}=${exp200+}
- Should be FOR iteration ${loop.body[5]} \${var}=${exp200+}
- ${loop} = Set Variable ${tc.body[7]}
- Should be FOR loop ${loop} 2
- Should be FOR iteration ${loop.body[0]} \${var1}=${exp10} \${var2}=${exp100} \${var3}=${exp200}
- Should be FOR iteration ${loop.body[1]} \${var1}=${exp200+} \${var2}=${exp200+} \${var3}=${exp200+}
+ Should be FOR loop ${loop} 6
+ Should be FOR iteration ${loop[0]} \${var}=${exp10}
+ Should be FOR iteration ${loop[1]} \${var}=${exp100}
+ Should be FOR iteration ${loop[2]} \${var}=${exp200}
+ Should be FOR iteration ${loop[3]} \${var}=${exp200+}
+ Should be FOR iteration ${loop[4]} \${var}=${exp200+}
+ Should be FOR iteration ${loop[5]} \${var}=${exp200+}
+ ${loop} = Set Variable ${tc[7]}
+ Should be FOR loop ${loop} 2
+ Should be FOR iteration ${loop[0]} \${var1}=${exp10} \${var2}=${exp100} \${var3}=${exp200}
+ Should be FOR iteration ${loop[1]} \${var1}=${exp200+} \${var2}=${exp200+} \${var3}=${exp200+}
Characters that are illegal in XML
- ${tc} = Check test case ${TEST NAME}
- Should be FOR iteration ${tc.body[0].body[0]} \${var}=illegal:
- Should be FOR iteration ${tc.body[0].body[1]} \${var}=more:
+ ${tc} = Check Test Case ${TEST NAME}
+ Should be FOR iteration ${tc[0, 0]} \${var}=illegal:
+ Should be FOR iteration ${tc[0, 1]} \${var}=more:
Old :FOR syntax is not supported
Check Test Case ${TESTNAME}
@@ -235,12 +235,12 @@ Escaping with backslash is not supported
Check Test Case ${TESTNAME}
FOR is case and space sensitive
- Check test case ${TEST NAME} 1
- Check test case ${TEST NAME} 2
+ Check Test Case ${TEST NAME} 1
+ Check Test Case ${TEST NAME} 2
Invalid END usage
- Check test case ${TEST NAME} 1
- Check test case ${TEST NAME} 2
+ Check Test Case ${TEST NAME} 1
+ Check Test Case ${TEST NAME} 2
Empty body
Check test and failed loop ${TEST NAME}
@@ -252,11 +252,10 @@ Invalid END
Check test and failed loop ${TEST NAME}
No loop values
- ${tc} = Check test case ${TEST NAME}
- Should be FOR loop ${tc.body[0]} 0 FAIL
+ Check test and failed loop ${TEST NAME}
No loop variables
- Check Test Case ${TESTNAME}
+ Check test and failed loop ${TEST NAME}
Invalid loop variable
Check test and failed loop ${TEST NAME} 1
@@ -267,13 +266,13 @@ Invalid loop variable
Check test and failed loop ${TEST NAME} 6
Invalid separator
- Check test case ${TEST NAME}
+ Check Test Case ${TEST NAME}
Separator is case- and space-sensitive
- Check test case ${TEST NAME} 1
- Check test case ${TEST NAME} 2
- Check test case ${TEST NAME} 3
- Check test case ${TEST NAME} 4
+ Check Test Case ${TEST NAME} 1
+ Check Test Case ${TEST NAME} 2
+ Check Test Case ${TEST NAME} 3
+ Check Test Case ${TEST NAME} 4
FOR without any paramenters
Check Test Case ${TESTNAME}
@@ -284,10 +283,10 @@ Syntax error in nested loop
Unexecuted
${tc} = Check Test Case ${TESTNAME}
- Should be FOR loop ${tc.body[1].body[0].body[0]} 1 NOT RUN
- Should be FOR iteration ${tc.body[1].body[0].body[0].body[0]} \${x}= \${y}=
- Should be FOR loop ${tc.body[5]} 1 NOT RUN
- Should be FOR iteration ${tc.body[5].body[0]} \${x}= \${y}=
+ Should be FOR loop ${tc[1, 0, 0]} 1 NOT RUN
+ Should be FOR iteration ${tc[1, 0, 0, 0]} \${x}= \${y}=
+ Should be FOR loop ${tc[5]} 1 NOT RUN
+ Should be FOR iteration ${tc[5, 0]} \${x}= \${y}=
Header at the end of file
Check Test Case ${TESTNAME}
@@ -295,49 +294,49 @@ Header at the end of file
*** Keywords ***
"Variables in values" helper
[Arguments] ${kw} ${num}
- Check log message ${kw.kws[0].msgs[0]} ${num}
- Check log message ${kw.kws[1].msgs[0]} Hello from for loop
- Should be equal ${kw.kws[2].name} BuiltIn.No Operation
+ Check Log Message ${kw[0, 0]} ${num}
+ Check Log Message ${kw[1, 0]} Hello from for loop
+ Should Be Equal ${kw[2].full_name} BuiltIn.No Operation
Check kw "My UK"
[Arguments] ${kw}
- Should be equal ${kw.name} My UK
- Should be equal ${kw.kws[0].name} BuiltIn.No Operation
- Check log message ${kw.kws[1].msgs[0]} We are in My UK
+ Should Be Equal ${kw.full_name} My UK
+ Should Be Equal ${kw[0].full_name} BuiltIn.No Operation
+ Check Log Message ${kw[1, 0]} We are in My UK
Check kw "My UK 2"
[Arguments] ${kw} ${arg}
- Should be equal ${kw.name} My UK 2
- Check kw "My UK" ${kw.kws[0]}
- Check log message ${kw.kws[1].msgs[0]} My UK 2 got argument "${arg}"
- Check kw "My UK" ${kw.kws[2]}
+ Should Be Equal ${kw.full_name} My UK 2
+ Check kw "My UK" ${kw[0]}
+ Check Log Message ${kw[1, 0]} My UK 2 got argument "${arg}"
+ Check kw "My UK" ${kw[2]}
Check kw "For In UK"
[Arguments] ${kw}
- Should be equal ${kw.name} For In UK
- Check log message ${kw.kws[0].msgs[0]} Not for yet
- Should be FOR loop ${kw.kws[1]} 2
- Check log message ${kw.kws[1].kws[0].kws[0].msgs[0]} This is for with 1
- Check kw "My UK" ${kw.kws[1].kws[0].kws[1]}
- Check log message ${kw.kws[1].kws[1].kws[0].msgs[0]} This is for with 2
- Check kw "My UK" ${kw.kws[1].kws[1].kws[1]}
- Check log message ${kw.kws[2].msgs[0]} Not for anymore
+ Should Be Equal ${kw.full_name} For In UK
+ Check Log Message ${kw[0, 0]} Not for yet
+ Should be FOR loop ${kw[1]} 2
+ Check Log Message ${kw[1, 0, 0, 0]} This is for with 1
+ Check kw "My UK" ${kw[1, 0, 1]}
+ Check Log Message ${kw[1, 1, 0, 0]} This is for with 2
+ Check kw "My UK" ${kw[1, 1, 1]}
+ Check Log Message ${kw[2, 0]} Not for anymore
Check kw "For In UK With Args"
[Arguments] ${kw} ${arg_count} ${first_arg}
- Should be equal ${kw.name} For In UK With Args
- Should be FOR loop ${kw.kws[0]} ${arg_count}
- Check kw "My UK 2" ${kw.kws[0].kws[0].kws[0]} ${first_arg}
- Should be FOR loop ${kw.kws[2]} 1
- Check log message ${kw.kws[2].kws[0].kws[0].msgs[0]} This for loop is executed only once
+ Should Be Equal ${kw.full_name} For In UK With Args
+ Should be FOR loop ${kw[0]} ${arg_count}
+ Check kw "My UK 2" ${kw[0, 0, 0]} ${first_arg}
+ Should be FOR loop ${kw[2]} 1
+ Check Log Message ${kw[2, 0, 0, 0]} This for loop is executed only once
Check kw "Nested For In UK"
[Arguments] ${kw} ${first_arg}
- Should be FOR loop ${kw.kws[0]} 1 FAIL
- Check kw "For In UK" ${kw.kws[0].kws[0].kws[0]}
- ${nested2} = Set Variable ${kw.kws[0].kws[0].kws[1]}
- Should be equal ${nested2.name} Nested For In UK 2
- Should be FOR loop ${nested2.kws[0]} 2
- Check kw "For In UK" ${nested2.kws[0].kws[0].kws[0]}
- Check log message ${nested2.kws[0].kws[0].kws[1].msgs[0]} Got arg: ${first_arg}
- Check log message ${nested2.kws[1].msgs[0]} This ought to be enough FAIL
+ Should be FOR loop ${kw[0]} 1 FAIL
+ Check kw "For In UK" ${kw[0, 0, 0]}
+ ${nested2} = Set Variable ${kw[0, 0, 1]}
+ Should Be Equal ${nested2.full_name} Nested For In UK 2
+ Should be FOR loop ${nested2[0]} 2
+ Check kw "For In UK" ${nested2[0, 0, 0]}
+ Check Log Message ${nested2[0, 0, 1, 0]} Got arg: ${first_arg}
+ Check Log Message ${nested2[1, 0]} This ought to be enough FAIL
diff --git a/atest/robot/running/for/for_dict_iteration.robot b/atest/robot/running/for/for_dict_iteration.robot
index ceb0c5caf45..a8b840726fc 100644
--- a/atest/robot/running/for/for_dict_iteration.robot
+++ b/atest/robot/running/for/for_dict_iteration.robot
@@ -43,7 +43,7 @@ FOR IN ENUMERATE loop with three variables
FOR IN ENUMERATE loop with start
${loop} = Check test and get loop ${TESTNAME}
- Should be IN ENUMERATE loop ${loop} 3
+ Should be IN ENUMERATE loop ${loop} 3 start=42
FOR IN ENUMERATE loop with more than three variables is invalid
Check test and failed loop ${TESTNAME} IN ENUMERATE
@@ -72,14 +72,14 @@ Equal sign in variable
... FOR loop iteration over values that are all in 'name=value' format like 'a=1' is deprecated.
... In the future this syntax will mean iterating over names and values separately like when iterating over '\&{dict} variables.
... Escape at least one of the values like 'a\\=1' to use normal FOR loop iteration and to disable this warning.
- Check Log Message ${tc.body[0].msgs[0]} ${message} WARN
- Check Log Message ${ERRORS}[0] ${message} WARN
+ Check Log Message ${tc[0, 0]} ${message} WARN
+ Check Log Message ${ERRORS}[0] ${message} WARN
${message} = Catenate
... FOR loop iteration over values that are all in 'name=value' format like 'x==1' is deprecated.
... In the future this syntax will mean iterating over names and values separately like when iterating over '\&{dict} variables.
... Escape at least one of the values like 'x\\==1' to use normal FOR loop iteration and to disable this warning.
- Check Log Message ${tc.body[2].msgs[0]} ${message} WARN
- Check Log Message ${ERRORS}[1] ${message} WARN
+ Check Log Message ${tc[2, 0]} ${message} WARN
+ Check Log Message ${ERRORS}[1] ${message} WARN
Non-string keys
Check Test Case ${TESTNAME}
diff --git a/atest/robot/running/for/for_in_enumerate.robot b/atest/robot/running/for/for_in_enumerate.robot
index a47d2a809d0..67ddc3c3134 100644
--- a/atest/robot/running/for/for_in_enumerate.robot
+++ b/atest/robot/running/for/for_in_enumerate.robot
@@ -21,7 +21,7 @@ Values from list variable
Start
${loop} = Check test and get loop ${TEST NAME}
- Should be IN ENUMERATE loop ${loop} 5
+ Should be IN ENUMERATE loop ${loop} 5 start=1
Should be FOR iteration ${loop.body[0]} \${index}=1 \${item}=1
Should be FOR iteration ${loop.body[1]} \${index}=2 \${item}=2
Should be FOR iteration ${loop.body[2]} \${index}=3 \${item}=3
@@ -33,12 +33,14 @@ Escape start
Should be IN ENUMERATE loop ${loop} 2
Invalid start
- ${loop} = Check test and get loop ${TEST NAME}
- Should be IN ENUMERATE loop ${loop} 0 status=FAIL
+ Check test and failed loop ${TEST NAME} IN ENUMERATE start=invalid
Invalid variable in start
+ Check test and failed loop ${TEST NAME} IN ENUMERATE start=\${invalid}
+
+Start multiple times
${loop} = Check test and get loop ${TEST NAME}
- Should be IN ENUMERATE loop ${loop} 0 status=FAIL
+ Should be IN ENUMERATE loop ${loop} 1 start=2
Index and two items
${loop} = Check test and get loop ${TEST NAME} 1
@@ -66,4 +68,4 @@ No values
Check test and failed loop ${TEST NAME} IN ENUMERATE
No values with start
- Check test and failed loop ${TEST NAME} IN ENUMERATE
+ Check test and failed loop ${TEST NAME} IN ENUMERATE start=0
diff --git a/atest/robot/running/for/for_in_range.robot b/atest/robot/running/for/for_in_range.robot
index 9c3f2c12a7a..0defe65d78d 100644
--- a/atest/robot/running/for/for_in_range.robot
+++ b/atest/robot/running/for/for_in_range.robot
@@ -5,65 +5,65 @@ Resource for.resource
*** Test Cases ***
Only stop
${loop} = Check test and get loop ${TEST NAME}
- Should be IN RANGE loop ${loop} 100
- Should be FOR iteration ${loop.body[0]} \${i}=0
- Check log message ${loop.body[0].body[1].msgs[0]} i: 0
- Should be FOR iteration ${loop.body[1]} \${i}=1
- Check log message ${loop.body[1].body[1].msgs[0]} i: 1
- Should be FOR iteration ${loop.body[42]} \${i}=42
- Check log message ${loop.body[42].body[1].msgs[0]} i: 42
- Should be FOR iteration ${loop.body[-1]} \${i}=99
- Check log message ${loop.body[-1].body[1].msgs[0]} i: 99
+ Should be IN RANGE loop ${loop} 100
+ Should be FOR iteration ${loop[0]} \${i}=0
+ Check log message ${loop[0, 1, 0]} i: 0
+ Should be FOR iteration ${loop[1]} \${i}=1
+ Check log message ${loop[1, 1, 0]} i: 1
+ Should be FOR iteration ${loop[42]} \${i}=42
+ Check log message ${loop[42, 1, 0]} i: 42
+ Should be FOR iteration ${loop[-1]} \${i}=99
+ Check log message ${loop[-1, 1, 0]} i: 99
Start and stop
- ${loop} = Check test and get loop ${TEST NAME}
- Should be IN RANGE loop ${loop} 4
+ ${loop} = Check test and get loop ${TEST NAME}
+ Should be IN RANGE loop ${loop} 4
Start, stop and step
- ${loop} = Check test and get loop ${TEST NAME}
- Should be IN RANGE loop ${loop} 3
- Should be FOR iteration ${loop.body[0]} \${item}=10
- Should be FOR iteration ${loop.body[1]} \${item}=7
- Should be FOR iteration ${loop.body[2]} \${item}=4
+ ${loop} = Check test and get loop ${TEST NAME}
+ Should be IN RANGE loop ${loop} 3
+ Should be FOR iteration ${loop[0]} \${item}=10
+ Should be FOR iteration ${loop[1]} \${item}=7
+ Should be FOR iteration ${loop[2]} \${item}=4
Float stop
- ${loop} = Check test and get loop ${TEST NAME} 1
- Should be IN RANGE loop ${loop} 4
- Should be FOR iteration ${loop.body[0]} \${item}=0.0
- Should be FOR iteration ${loop.body[1]} \${item}=1.0
- Should be FOR iteration ${loop.body[2]} \${item}=2.0
- Should be FOR iteration ${loop.body[3]} \${item}=3.0
- ${loop} = Check test and get loop ${TEST NAME} 2
- Should be IN RANGE loop ${loop} 3
- Should be FOR iteration ${loop.body[0]} \${item}=0.0
- Should be FOR iteration ${loop.body[1]} \${item}=1.0
- Should be FOR iteration ${loop.body[2]} \${item}=2.0
+ ${loop} = Check test and get loop ${TEST NAME} 1
+ Should be IN RANGE loop ${loop} 4
+ Should be FOR iteration ${loop[0]} \${item}=0.0
+ Should be FOR iteration ${loop[1]} \${item}=1.0
+ Should be FOR iteration ${loop[2]} \${item}=2.0
+ Should be FOR iteration ${loop[3]} \${item}=3.0
+ ${loop} = Check test and get loop ${TEST NAME} 2
+ Should be IN RANGE loop ${loop} 3
+ Should be FOR iteration ${loop[0]} \${item}=0.0
+ Should be FOR iteration ${loop[1]} \${item}=1.0
+ Should be FOR iteration ${loop[2]} \${item}=2.0
Float start and stop
- ${loop} = Check test and get loop ${TEST NAME} 1
- Should be IN RANGE loop ${loop} 3
- Should be FOR iteration ${loop.body[0]} \${item}=-1.5
- Should be FOR iteration ${loop.body[1]} \${item}=-0.5
- Should be FOR iteration ${loop.body[2]} \${item}=0.5
- ${loop} = Check test and get loop ${TEST NAME} 2 0
- Should be IN RANGE loop ${loop} 4
- Should be FOR iteration ${loop.body[0]} \${item}=-1.5
- Should be FOR iteration ${loop.body[1]} \${item}=-0.5
- Should be FOR iteration ${loop.body[2]} \${item}=0.5
- Should be FOR iteration ${loop.body[3]} \${item}=1.5
+ ${loop} = Check test and get loop ${TEST NAME} 1
+ Should be IN RANGE loop ${loop} 3
+ Should be FOR iteration ${loop[0]} \${item}=-1.5
+ Should be FOR iteration ${loop[1]} \${item}=-0.5
+ Should be FOR iteration ${loop[2]} \${item}=0.5
+ ${loop} = Check test and get loop ${TEST NAME} 2 0
+ Should be IN RANGE loop ${loop} 4
+ Should be FOR iteration ${loop[0]} \${item}=-1.5
+ Should be FOR iteration ${loop[1]} \${item}=-0.5
+ Should be FOR iteration ${loop[2]} \${item}=0.5
+ Should be FOR iteration ${loop[3]} \${item}=1.5
Float start, stop and step
- ${loop} = Check test and get loop ${TEST NAME}
- Should be IN RANGE loop ${loop} 3
- Should be FOR iteration ${loop.body[0]} \${item}=10.99
- Should be FOR iteration ${loop.body[1]} \${item}=7.95
- Should be FOR iteration ${loop.body[2]} \${item}=4.91
+ ${loop} = Check test and get loop ${TEST NAME}
+ Should be IN RANGE loop ${loop} 3
+ Should be FOR iteration ${loop[0]} \${item}=10.99
+ Should be FOR iteration ${loop[1]} \${item}=7.95
+ Should be FOR iteration ${loop[2]} \${item}=4.91
Variables in arguments
- ${loop} = Check test and get loop ${TEST NAME} 0
- Should be IN RANGE loop ${loop} 2
- ${loop} = Check test and get loop ${TEST NAME} 2
- Should be IN RANGE loop ${loop} 1
+ ${loop} = Check test and get loop ${TEST NAME} 0
+ Should be IN RANGE loop ${loop} 2
+ ${loop} = Check test and get loop ${TEST NAME} 2
+ Should be IN RANGE loop ${loop} 1
Calculations
Check test case ${TEST NAME}
@@ -72,15 +72,15 @@ Calculations with floats
Check test case ${TEST NAME}
Multiple variables
- ${loop} = Check test and get loop ${TEST NAME} 0
- Should be IN RANGE loop ${loop} 1
- Should be FOR iteration ${loop.body[0]} \${a}=0 \${b}=1 \${c}=2 \${d}=3 \${e}=4
- ${loop} = Check test and get loop ${TEST NAME} 2
- Should be IN RANGE loop ${loop} 4
- Should be FOR iteration ${loop.body[0]} \${i}=-1 \${j}=0 \${k}=1
- Should be FOR iteration ${loop.body[1]} \${i}=2 \${j}=3 \${k}=4
- Should be FOR iteration ${loop.body[2]} \${i}=5 \${j}=6 \${k}=7
- Should be FOR iteration ${loop.body[3]} \${i}=8 \${j}=9 \${k}=10
+ ${loop} = Check test and get loop ${TEST NAME} 0
+ Should be IN RANGE loop ${loop} 1
+ Should be FOR iteration ${loop[0]} \${a}=0 \${b}=1 \${c}=2 \${d}=3 \${e}=4
+ ${loop} = Check test and get loop ${TEST NAME} 2
+ Should be IN RANGE loop ${loop} 4
+ Should be FOR iteration ${loop[0]} \${i}=-1 \${j}=0 \${k}=1
+ Should be FOR iteration ${loop[1]} \${i}=2 \${j}=3 \${k}=4
+ Should be FOR iteration ${loop[2]} \${i}=5 \${j}=6 \${k}=7
+ Should be FOR iteration ${loop[3]} \${i}=8 \${j}=9 \${k}=10
Too many arguments
Check test and failed loop ${TEST NAME} IN RANGE
diff --git a/atest/robot/running/for/for_in_zip.robot b/atest/robot/running/for/for_in_zip.robot
index b512c9a9a21..987e080ff40 100644
--- a/atest/robot/running/for/for_in_zip.robot
+++ b/atest/robot/running/for/for_in_zip.robot
@@ -5,70 +5,122 @@ Resource for.resource
*** Test Cases ***
Two variables and lists
${loop} = Check test and get loop ${TEST NAME}
- Should be IN ZIP loop ${loop} 3
- Should be FOR iteration ${loop.body[0]} \${x}=a \${y}=x
- Should be FOR iteration ${loop.body[1]} \${x}=b \${y}=y
- Should be FOR iteration ${loop.body[2]} \${x}=c \${y}=z
+ Should be IN ZIP loop ${loop} 3
+ Should be FOR iteration ${loop[0]} \${x}=a \${y}=x
+ Should be FOR iteration ${loop[1]} \${x}=b \${y}=y
+ Should be FOR iteration ${loop[2]} \${x}=c \${y}=z
-Uneven lists
+Uneven lists cause deprecation warning by default
${loop} = Check test and get loop ${TEST NAME}
- Should be IN ZIP loop ${loop} 3
- Should be FOR iteration ${loop.body[0]} \${x}=a \${y}=1
- Should be FOR iteration ${loop.body[1]} \${x}=b \${y}=2
- Should be FOR iteration ${loop.body[2]} \${x}=c \${y}=3
+ Should be IN ZIP loop ${loop} 3
+ Check Log Message ${loop[0]}
+ ... FOR IN ZIP default mode will be changed from SHORTEST to STRICT in Robot Framework 8.0. Use 'mode=SHORTEST' to keep using the SHORTEST mode. If the mode is not changed, execution will fail like this in the future: FOR IN ZIP items must have equal lengths in the STRICT mode, but lengths are 3 and 5. WARN
+ Should be FOR iteration ${loop[1]} \${x}=a \${y}=1
+ Should be FOR iteration ${loop[2]} \${x}=b \${y}=2
+ Should be FOR iteration ${loop[3]} \${x}=c \${y}=3
Three variables and lists
${loop} = Check test and get loop ${TEST NAME}
- Should be IN ZIP loop ${loop} 3
- Should be FOR iteration ${loop.body[0]} \${x}=a \${y}=x \${z}=1
- Should be FOR iteration ${loop.body[1]} \${x}=b \${y}=y \${z}=2
- Should be FOR iteration ${loop.body[2]} \${x}=c \${y}=z \${z}=3
+ Should be IN ZIP loop ${loop} 3
+ Should be FOR iteration ${loop[0]} \${x}=a \${y}=x \${z}=1
+ Should be FOR iteration ${loop[1]} \${x}=b \${y}=y \${z}=2
+ Should be FOR iteration ${loop[2]} \${x}=c \${y}=z \${z}=3
Six variables and lists
${loop} = Check test and get loop ${TEST NAME}
- Should be IN ZIP loop ${loop} 3
- Should be FOR iteration ${loop.body[0]} \${x}=a \${y}=x \${z}=1 \${å}=1 \${ä}=x \${ö}=a
- Should be FOR iteration ${loop.body[1]} \${x}=b \${y}=y \${z}=2 \${å}=2 \${ä}=y \${ö}=b
- Should be FOR iteration ${loop.body[2]} \${x}=c \${y}=z \${z}=3 \${å}=3 \${ä}=z \${ö}=c
+ Should be IN ZIP loop ${loop} 3
+ Should be FOR iteration ${loop[0]} \${x}=a \${y}=x \${z}=1 \${å}=1 \${ä}=x \${ö}=a
+ Should be FOR iteration ${loop[1]} \${x}=b \${y}=y \${z}=2 \${å}=2 \${ä}=y \${ö}=b
+ Should be FOR iteration ${loop[2]} \${x}=c \${y}=z \${z}=3 \${å}=3 \${ä}=z \${ö}=c
One variable and list
${loop} = Check test and get loop ${TEST NAME}
- Should be IN ZIP loop ${loop} 3
- Should be FOR iteration ${loop.body[0]} \${x}=a
- Should be FOR iteration ${loop.body[1]} \${x}=b
- Should be FOR iteration ${loop.body[2]} \${x}=c
+ Should be IN ZIP loop ${loop} 3
+ Should be FOR iteration ${loop[0]} \${x}=a
+ Should be FOR iteration ${loop[1]} \${x}=b
+ Should be FOR iteration ${loop[2]} \${x}=c
One variable and two lists
${loop} = Check test and get loop ${TEST NAME}
- Should be IN ZIP loop ${loop} 3
- Should be FOR iteration ${loop.body[0]} \${x}=('a', 'x')
- Should be FOR iteration ${loop.body[1]} \${x}=('b', 'y')
- Should be FOR iteration ${loop.body[2]} \${x}=('c', 'z')
+ Should be IN ZIP loop ${loop} 3
+ Should be FOR iteration ${loop[0]} \${x}=('a', 'x')
+ Should be FOR iteration ${loop[1]} \${x}=('b', 'y')
+ Should be FOR iteration ${loop[2]} \${x}=('c', 'z')
One variable and six lists
${loop} = Check test and get loop ${TEST NAME}
- Should be IN ZIP loop ${loop} 3
- Should be FOR iteration ${loop.body[0]} \${x}=('a', 'x', '1', '1', 'x', 'a')
- Should be FOR iteration ${loop.body[1]} \${x}=('b', 'y', '2', '2', 'y', 'b')
- Should be FOR iteration ${loop.body[2]} \${x}=('c', 'z', '3', '3', 'z', 'c')
+ Should be IN ZIP loop ${loop} 3
+ Should be FOR iteration ${loop[0]} \${x}=('a', 'x', 1, 1, 'x', 'a')
+ Should be FOR iteration ${loop[1]} \${x}=('b', 'y', 2, 2, 'y', 'b')
+ Should be FOR iteration ${loop[2]} \${x}=('c', 'z', 3, 3, 'z', 'c')
Other iterables
Check Test Case ${TEST NAME}
List variable containing iterables
- ${loop} = Check test and get loop ${TEST NAME} 2
- Should be IN ZIP loop ${loop} 3
- Should be FOR iteration ${loop.body[0]} \${x}=a \${y}=x \${z}=f
- Should be FOR iteration ${loop.body[1]} \${x}=b \${y}=y \${z}=o
- Should be FOR iteration ${loop.body[2]} \${x}=c \${y}=z \${z}=o
+ ${loop} = Check test and get loop ${TEST NAME} 1
+ Should be IN ZIP loop ${loop} 3
+ Should be FOR iteration ${loop[0]} \${x}=a \${y}=x \${z}=f
+ Should be FOR iteration ${loop[1]} \${x}=b \${y}=y \${z}=o
+ Should be FOR iteration ${loop[2]} \${x}=c \${y}=z \${z}=o
List variable with iterables can be empty
${tc} = Check Test Case ${TEST NAME}
- Should be IN ZIP loop ${tc.body[0]} 1 NOT RUN
- Should be FOR iteration ${tc.body[0].body[0]} \${x}=
- Should be IN ZIP loop ${tc.body[1]} 1 NOT RUN
- Should be FOR iteration ${tc.body[1].body[0]} \${x}= \${y}= \${z}=
- Check Log Message ${tc.body[2].msgs[0]} Executed!
+ Should be IN ZIP loop ${tc[0]} 1 NOT RUN
+ Should be FOR iteration ${tc[0, 0]} \${x}=
+ Should be IN ZIP loop ${tc[1]} 1 NOT RUN
+ Should be FOR iteration ${tc[1, 0]} \${x}= \${y}= \${z}=
+ Check Log Message ${tc[2, 0]} Executed!
+
+Strict mode
+ ${tc} = Check Test Case ${TEST NAME}
+ Should be IN ZIP loop ${tc[0]} 3 PASS mode=STRICT
+ Should be IN ZIP loop ${tc[2]} 1 FAIL mode=strict
+
+Strict mode requires items to have length
+ ${tc} = Check Test Case ${TEST NAME}
+ Should be IN ZIP loop ${tc[0]} 1 FAIL mode=STRICT
+
+Shortest mode
+ ${tc} = Check Test Case ${TEST NAME}
+ Should be IN ZIP loop ${tc[0]} 3 PASS mode=SHORTEST fill=ignored
+ Should be IN ZIP loop ${tc[3]} 3 PASS mode=\${{'shortest'}}
+
+Shortest mode supports infinite iterators
+ ${tc} = Check Test Case ${TEST NAME}
+ Should be IN ZIP loop ${tc[0]} 3 PASS mode=SHORTEST
+
+Longest mode
+ ${tc} = Check Test Case ${TEST NAME}
+ Should be IN ZIP loop ${tc[0]} 3 PASS mode=LONGEST
+ Should be IN ZIP loop ${tc[3]} 5 PASS mode=LoNgEsT
+
+Longest mode with custom fill value
+ ${tc} = Check Test Case ${TEST NAME}
+ Should be IN ZIP loop ${tc[0]} 5 PASS mode=longest fill=?
+ Should be IN ZIP loop ${tc[3]} 3 PASS mode=longest fill=\${0}
+
+Invalid mode
+ ${tc} = Check Test Case ${TEST NAME}
+ Should be IN ZIP loop ${tc[0]} 1 FAIL mode=bad
+
+Invalid mode from variable
+ ${tc} = Check Test Case ${TEST NAME}
+ Should be IN ZIP loop ${tc[0]} 1 FAIL mode=\${{'bad'}}
+
+Config more than once
+ ${tc} = Check Test Case ${TEST NAME} 1
+ Should be IN ZIP loop ${tc[0]} 1 FAIL mode=shortest
+ ${tc} = Check Test Case ${TEST NAME} 2
+ Should be IN ZIP loop ${tc[0]} 1 FAIL fill=z
+
+Non-existing variable in mode
+ ${tc} = Check Test Case ${TEST NAME}
+ Should be IN ZIP loop ${tc[0]} 1 FAIL mode=\${bad} fill=\${ignored}
+
+Non-existing variable in fill value
+ ${tc} = Check Test Case ${TEST NAME}
+ Should be IN ZIP loop ${tc[0]} 1 FAIL mode=longest fill=\${bad}
Not iterable value
Check test and failed loop ${TEST NAME} IN ZIP
diff --git a/atest/robot/running/group/group.robot b/atest/robot/running/group/group.robot
new file mode 100644
index 00000000000..f579f090cf5
--- /dev/null
+++ b/atest/robot/running/group/group.robot
@@ -0,0 +1,42 @@
+*** Settings ***
+Suite Setup Run Tests ${EMPTY} running/group/group.robot
+Resource atest_resource.robot
+
+*** Test Cases ***
+Basics
+ ${tc}= Check Test Case ${TESTNAME}
+ Check Body Item Data ${tc[0]} type=GROUP name=1st group children=2
+ Check Body Item Data ${tc[0, 0]} type=KEYWORD name=Log args=Inside group
+ Check Body Item Data ${tc[0, 1]} type=KEYWORD name=Log args=Still inside
+ Check Body Item Data ${tc[1]} type=GROUP name=second children=1
+ Check Body Item Data ${tc[1, 0]} type=KEYWORD name=Log args=Inside second group
+ Check Body Item Data ${tc[2]} type=KEYWORD name=Log args=After
+
+Failing
+ ${tc}= Check Test Case ${TESTNAME}
+ Check Body Item Data ${tc[0]} type=GROUP name=Fails children=2 status=FAIL message=Failing inside GROUP!
+ Check Body Item Data ${tc[0, 0]} type=KEYWORD name=Fail children=1 status=FAIL message=Failing inside GROUP!
+ Check Body Item Data ${tc[0, 1]} type=KEYWORD name=Fail children=0 status=NOT RUN
+ Check Body Item Data ${tc[1]} type=GROUP name=Not run children=1 status=NOT RUN
+ Check Body Item Data ${tc[1, 0]} type=KEYWORD name=Fail children=0 status=NOT RUN
+
+Anonymous
+ ${tc}= Check Test Case ${TESTNAME}
+ Check Body Item Data ${tc[0]} type=GROUP name=${EMPTY} children=1
+ Check Body Item Data ${tc[0, 0]} type=KEYWORD name=Log args=Inside unnamed group
+
+Variable in name
+ ${tc}= Check Test Case ${TESTNAME}
+ Check Body Item Data ${tc[0]} type=GROUP name=Test is named: ${TEST NAME} children=1
+ Check Body Item Data ${tc[0, 0]} type=KEYWORD name=Log args=\${TEST_NAME}
+ Check Log Message ${tc[0, 0, 0]} ${TEST NAME}
+ Check Body Item Data ${tc[1]} type=GROUP name=42 children=1
+ Check Body Item Data ${tc[1, 0]} type=KEYWORD name=Log args=Should be 42
+
+In user keyword
+ ${tc}= Check Test Case ${TESTNAME}
+ Check Body Item Data ${tc[0]} type=KEYWORD name=Keyword children=4
+ Check Body Item Data ${tc[0, 0]} type=KEYWORD name=Log args=Before
+ Check Body Item Data ${tc[0, 1]} type=GROUP name=First children=2
+ Check Body Item Data ${tc[0, 2]} type=GROUP name=Second children=1
+ Check Body Item Data ${tc[0, 3]} type=KEYWORD name=Log args=After
diff --git a/atest/robot/running/group/invalid_group.robot b/atest/robot/running/group/invalid_group.robot
new file mode 100644
index 00000000000..f6d415cdaf9
--- /dev/null
+++ b/atest/robot/running/group/invalid_group.robot
@@ -0,0 +1,44 @@
+*** Settings ***
+Suite Setup Run Tests ${EMPTY} running/group/invalid_group.robot
+Resource atest_resource.robot
+
+*** Test Cases ***
+END missing
+ ${tc} = Check Test Case ${TESTNAME}
+ Length Should Be ${tc.body} 1
+ Check Body Item Data ${tc[0]} GROUP status=FAIL children=2 message=GROUP must have closing END.
+ Check Body Item Data ${tc[0, 0]} KEYWORD status=NOT RUN children=0 name=Fail args=Not run
+ Check Body Item Data ${tc[0, 1]} MESSAGE level=FAIL message=GROUP must have closing END.
+
+Empty
+ ${tc} Check Test Case ${TESTNAME}
+ Length Should Be ${tc.body} 2
+ Check Body Item Data ${tc[0]} GROUP status=FAIL children=1 message=GROUP cannot be empty.
+ Check Body Item Data ${tc[0, 0]} MESSAGE level=FAIL message=GROUP cannot be empty.
+ Check Body Item Data ${tc[1]} KEYWORD status=NOT RUN children=0 name=Log args=Outside
+
+Multiple parameters
+ ${tc} Check Test Case ${TESTNAME}
+ Length Should Be ${tc.body} 2
+ Check Body Item Data ${tc[0]} GROUP status=FAIL children=2 message=GROUP accepts only one argument as name, got 3 arguments 'Too', 'many' and 'values'.
+ Check Body Item Data ${tc[0, 0]} KEYWORD status=NOT RUN children=0 name=Fail args=Not run
+ Check Body Item Data ${tc[0, 1]} MESSAGE level=FAIL message=GROUP accepts only one argument as name, got 3 arguments 'Too', 'many' and 'values'.
+ Check Body Item Data ${tc[1]} KEYWORD status=NOT RUN children=0 name=Log args=Last Keyword
+
+Non-existing variable in name
+ ${tc} Check Test Case ${TESTNAME}
+ Length Should Be ${tc.body} 2
+ Check Body Item Data ${tc[0]} GROUP status=FAIL children=2 message=Variable '\${non_existing_var}' not found. name=\${non_existing_var} in name
+ Check Body Item Data ${tc[0, 0]} KEYWORD status=NOT RUN children=0 name=Fail args=Not run
+ Check Body Item Data ${tc[0, 1]} MESSAGE level=FAIL message=Variable '\${non_existing_var}' not found.
+ Check Body Item Data ${tc[1]} KEYWORD status=NOT RUN children=0 name=Log args=Last Keyword
+
+Invalid data is not reported after failures
+ ${tc} Check Test Case ${TESTNAME}
+ Length Should Be ${tc.body} 4
+ Check Body Item Data ${tc[0]} KEYWORD status=FAIL children=1 name=Fail args=Something bad happened! message=Something bad happened!
+ Check Body Item Data ${tc[1]} GROUP status=NOT RUN children=1 name=\${non_existing_non_executed_variable_is_ok}
+ Check Body Item Data ${tc[1, 0]} KEYWORD status=NOT RUN children=0 name=Fail args=Not run
+ Check Body Item Data ${tc[2]} GROUP status=NOT RUN children=0 name=Empty non-executed GROUP is ok
+ Check Body Item Data ${tc[3]} GROUP status=NOT RUN children=1 name=Even missing END is ok
+ Check Body Item Data ${tc[3, 0]} KEYWORD status=NOT RUN children=0 name=Fail args=Not run
diff --git a/atest/robot/running/group/nesting_group.robot b/atest/robot/running/group/nesting_group.robot
new file mode 100644
index 00000000000..1d612e0c189
--- /dev/null
+++ b/atest/robot/running/group/nesting_group.robot
@@ -0,0 +1,51 @@
+*** Settings ***
+Suite Setup Run Tests ${EMPTY} running/group/nesting_group.robot
+Resource atest_resource.robot
+
+*** Test Cases ***
+Nested
+ ${tc} Check Test Case ${TESTNAME}
+ Check Body Item Data ${tc[0]} type=GROUP name=
+ Check Body Item Data ${tc[0, 0]} type=KEYWORD name=Set Variable
+ Check Body Item Data ${tc[0, 1]} type=GROUP name=This Is A Named Group
+ Check Body Item Data ${tc[0, 1, 0]} type=KEYWORD name=Should Be Equal
+
+With other control structures
+ ${tc} Check Test Case ${TESTNAME}
+ Check Body Item Data ${tc[0]} type=IF/ELSE ROOT
+ Check Body Item Data ${tc[0, 0]} type=IF condition=True children=2
+ Check Body Item Data ${tc[0, 0, 0]} type=GROUP name=Hello children=1
+ Check Body Item Data ${tc[0, 0, 0, 0]} type=VAR name=\${i}
+ Check Body Item Data ${tc[0, 0, 1]} type=GROUP name=With WHILE children=2
+ Check Body Item Data ${tc[0, 0, 1, 0]} type=WHILE condition=$i < 2 children=2
+ Check Body Item Data ${tc[0, 0, 1, 0, 0]} type=ITERATION
+ Check Body Item Data ${tc[0, 0, 1, 0, 0, 0]} type=GROUP name=Group1 Inside WHILE (0) children=1
+ Check Body Item Data ${tc[0, 0, 1, 0, 0, 0, 0]} type=KEYWORD name=Log args=\${i}
+ Check Body Item Data ${tc[0, 0, 1, 0, 0, 1]} type=GROUP name=Group2 Inside WHILE children=1
+ Check Body Item Data ${tc[0, 0, 1, 0, 0, 1, 0]} type=VAR name=\${i} value=\${i + 1}
+ Check Body Item Data ${tc[0, 0, 1, 0, 1]} type=ITERATION
+ Check Body Item Data ${tc[0, 0, 1, 0, 1, 0]} type=GROUP name=Group1 Inside WHILE (1) children=1
+ Check Body Item Data ${tc[0, 0, 1, 0, 1, 0, 0]} type=KEYWORD name=Log args=\${i}
+ Check Body Item Data ${tc[0, 0, 1, 0, 1, 1]} type=GROUP name=Group2 Inside WHILE children=1
+ Check Body Item Data ${tc[0, 0, 1, 0, 1, 1, 0]} type=VAR name=\${i} value=\${i + 1}
+ Check Body Item Data ${tc[0, 0, 1, 1]} type=IF/ELSE ROOT
+ Check Body Item Data ${tc[0, 0, 1, 1, 0]} type=IF status=NOT RUN condition=$i != 2 children=1
+ Check Body Item Data ${tc[0, 0, 1, 1, 0, 0]} type=KEYWORD status=NOT RUN name=Fail args=Shall be logged but NOT RUN
+
+In non-executed branch
+ ${tc} Check Test Case ${TESTNAME}
+ Check Body Item Data ${tc[0]} type=VAR name=\${var} value=value
+ Check Body Item Data ${tc[1]} type=IF/ELSE ROOT
+ Check Body Item Data ${tc[1, 0]} type=IF condition=True children=1
+ Check Body Item Data ${tc[1, 0, 0]} type=GROUP name=GROUP in IF children=2
+ Check Body Item Data ${tc[1, 0, 0, 0]} type=KEYWORD name=Should Be Equal
+ Check Body Item Data ${tc[1, 0, 0, 1]} type=IF/ELSE ROOT
+ Check Body Item Data ${tc[1, 0, 0, 1, 0]} type=IF status=PASS children=1 condition=True
+ Check Body Item Data ${tc[1, 0, 0, 1, 0, 0]} type=KEYWORD status=PASS name=Log args=IF in GROUP
+ Check Body Item Data ${tc[1, 0, 0, 1, 1]} type=ELSE status=NOT RUN
+ Check Body Item Data ${tc[1, 0, 0, 1, 1, 0]} type=GROUP status=NOT RUN children=1 name=GROUP in ELSE
+ Check Body Item Data ${tc[1, 0, 0, 1, 1, 0, 0]} type=KEYWORD status=NOT RUN name=Fail args=Shall be logged but NOT RUN
+ Check Body Item Data ${tc[1, 1]} type=ELSE IF status=NOT RUN
+ Check Body Item Data ${tc[1, 1, 0]} type=GROUP status=NOT RUN children=1 name=\${non_existing_variable_is_fine_here}
+ Check Body Item Data ${tc[1, 2]} type=ELSE status=NOT RUN
+ Check Body Item Data ${tc[1, 2, 0]} type=GROUP status=NOT RUN children=0 name=Even empty GROUP is allowed
diff --git a/atest/robot/running/group/templates.robot b/atest/robot/running/group/templates.robot
new file mode 100644
index 00000000000..b42966b2524
--- /dev/null
+++ b/atest/robot/running/group/templates.robot
@@ -0,0 +1,69 @@
+*** Settings ***
+Suite Setup Run Tests ${EMPTY} running/group/templates.robot
+Resource atest_resource.robot
+
+*** Test Cases ***
+Pass
+ ${tc} = Check Test Case ${TESTNAME}
+ Check Body Item Data ${tc[0]} type=GROUP status=PASS children=1 name=1
+ Check Body Item Data ${tc[0, 0]} type=KEYWORD status=PASS children=1 name=Run Keyword args=Log, 1.1
+ Check Body Item Data ${tc[1]} type=GROUP status=PASS children=2 name=2
+ Check Body Item Data ${tc[1, 0]} type=KEYWORD status=PASS children=1 name=Run Keyword args=Log, 2.1
+ Check Body Item Data ${tc[1, 1]} type=KEYWORD status=PASS children=1 name=Run Keyword args=Log, 2.2
+
+Pass and fail
+ ${tc} = Check Test Case ${TESTNAME}
+ Check Body Item Data ${tc[0]} type=GROUP status=PASS children=1 name=1
+ Check Body Item Data ${tc[0, 0]} type=KEYWORD status=PASS children=1 name=Run Keyword args=Log, 1.1
+ Check Body Item Data ${tc[1]} type=GROUP status=FAIL children=2 name=2 message=2.1
+ Check Body Item Data ${tc[1, 0]} type=KEYWORD status=FAIL children=1 name=Run Keyword args=Fail, 2.1 message=2.1
+ Check Body Item Data ${tc[1, 1]} type=KEYWORD status=PASS children=1 name=Run Keyword args=Log, 2.2
+ Check Body Item Data ${tc[2]} type=GROUP status=PASS children=1 name=3
+ Check Body Item Data ${tc[2, 0]} type=KEYWORD status=PASS children=1 name=Run Keyword args=Log, 3.1
+
+Fail multiple times
+ ${tc} = Check Test Case ${TESTNAME}
+ Check Body Item Data ${tc[0]} type=GROUP status=FAIL children=1 name=1 message=1.1
+ Check Body Item Data ${tc[0, 0]} type=KEYWORD status=FAIL children=1 name=Run Keyword args=Fail, 1.1 message=1.1
+ Check Body Item Data ${tc[1]} type=GROUP status=FAIL children=3 name=2 message=Several failures occurred:\n\n1) 2.1\n\n2) 2.3
+ Check Body Item Data ${tc[1, 0]} type=KEYWORD status=FAIL children=1 name=Run Keyword args=Fail, 2.1 message=2.1
+ Check Body Item Data ${tc[1, 1]} type=KEYWORD status=PASS children=1 name=Run Keyword args=Log, 2.2
+ Check Body Item Data ${tc[1, 2]} type=KEYWORD status=FAIL children=1 name=Run Keyword args=Fail, 2.3 message=2.3
+ Check Body Item Data ${tc[2]} type=GROUP status=PASS children=1 name=3
+ Check Body Item Data ${tc[2, 0]} type=KEYWORD status=PASS children=1 name=Run Keyword args=Log, 3.1
+ Check Body Item Data ${tc[3]} type=GROUP status=FAIL children=1 name=4 message=4.1
+ Check Body Item Data ${tc[3, 0]} type=KEYWORD status=FAIL children=1 name=Run Keyword args=Fail, 4.1 message=4.1
+
+Pass and skip
+ ${tc} = Check Test Case ${TESTNAME}
+ Check Body Item Data ${tc[0]} type=GROUP status=SKIP children=1 name=1 message=1.1
+ Check Body Item Data ${tc[0, 0]} type=KEYWORD status=SKIP children=1 name=Run Keyword args=Skip, 1.1 message=1.1
+ Check Body Item Data ${tc[1]} type=GROUP status=PASS children=1 name=2
+ Check Body Item Data ${tc[1, 0]} type=KEYWORD status=PASS children=1 name=Run Keyword args=Log, 2.1
+ Check Body Item Data ${tc[2]} type=GROUP status=PASS children=2 name=3
+ Check Body Item Data ${tc[2, 0]} type=KEYWORD status=SKIP children=1 name=Run Keyword args=Skip, 3.1 message=3.1
+ Check Body Item Data ${tc[2, 1]} type=KEYWORD status=PASS children=1 name=Run Keyword args=Log, 3.2
+
+Pass, fail and skip
+ ${tc} = Check Test Case ${TESTNAME}
+ Check Body Item Data ${tc[0]} type=GROUP status=FAIL children=3 name=1 message=1.1
+ Check Body Item Data ${tc[0, 0]} type=KEYWORD status=FAIL children=1 name=Run Keyword args=Fail, 1.1 message=1.1
+ Check Body Item Data ${tc[0, 1]} type=KEYWORD status=SKIP children=1 name=Run Keyword args=Skip, 1.2 message=1.2
+ Check Body Item Data ${tc[0, 2]} type=KEYWORD status=PASS children=1 name=Run Keyword args=Log, 1.3
+ Check Body Item Data ${tc[1]} type=GROUP status=SKIP children=1 name=2 message=2.1
+ Check Body Item Data ${tc[1, 0]} type=KEYWORD status=SKIP children=1 name=Run Keyword args=Skip, 2.1 message=2.1
+ Check Body Item Data ${tc[2]} type=GROUP status=PASS children=1 name=3
+ Check Body Item Data ${tc[2, 0]} type=KEYWORD status=PASS children=1 name=Run Keyword args=Log, 3.1
+
+Skip all
+ ${tc} = Check Test Case ${TESTNAME}
+ Check Body Item Data ${tc[0]} type=GROUP status=SKIP children=2 name=1 message=All iterations skipped.
+ Check Body Item Data ${tc[0, 0]} type=KEYWORD status=SKIP children=1 name=Run Keyword args=Skip, 1.1 message=1.1
+ Check Body Item Data ${tc[0, 1]} type=KEYWORD status=SKIP children=1 name=Run Keyword args=Skip, 1.2 message=1.2
+ Check Body Item Data ${tc[1]} type=GROUP status=SKIP children=1 name=2 message=2.1
+ Check Body Item Data ${tc[1, 0]} type=KEYWORD status=SKIP children=1 name=Run Keyword args=Skip, 2.1 message=2.1
+
+Just one that is skipped
+ ${tc} = Check Test Case ${TESTNAME}
+ Check Body Item Data ${tc[0]} type=GROUP status=SKIP children=1 name=1 message=1.1
+ Check Body Item Data ${tc[0, 0]} type=KEYWORD status=SKIP children=1 name=Run Keyword args=Skip, 1.1 message=1.1
diff --git a/atest/robot/running/html_error_message.robot b/atest/robot/running/html_error_message.robot
index 007d5a3cd04..9ff087a0d82 100644
--- a/atest/robot/running/html_error_message.robot
+++ b/atest/robot/running/html_error_message.robot
@@ -9,15 +9,15 @@ ${FAILURE} Robot Framework
*** Test Cases ***
Set Test Message
${tc} = Check Test Case ${TESTNAME}
- Check Log Message ${tc.kws[0].msgs[0]} Set test message to:\n${MESSAGE} html=True
+ Check Log Message ${tc[0, 0]} Set test message to:\n${MESSAGE} html=True
HTML failure
${tc} = Check Test Case ${TESTNAME}
- Check Log Message ${tc.kws[0].msgs[0]} ${FAILURE} FAIL html=True
+ Check Log Message ${tc[0, 0]} ${FAILURE} FAIL html=True
HTML failure with non-generic exception
${tc} = Check Test Case ${TESTNAME}
- Check Log Message ${tc.kws[0].msgs[0]} ValueError: Invalid value FAIL html=True
+ Check Log Message ${tc[0, 0]} ValueError: Invalid value FAIL html=True
HTML failure in setup
Check Test Case ${TESTNAME}
@@ -30,8 +30,8 @@ Normal failure in body and HTML failure in teardown
HTML failure in body and normal failure teardown
${tc} = Check Test Case ${TESTNAME}
- Check Log Message ${tc.kws[0].msgs[0]} Should be HTML FAIL html=True
- Check Log Message ${tc.teardown.msgs[0]} Should NOT be HTML FAIL html=False
+ Check Log Message ${tc[0, 0]} Should be HTML FAIL html=True
+ Check Log Message ${tc.teardown[0]} Should NOT be HTML FAIL html=False
HTML failure in body and in teardown
Check Test Case ${TESTNAME}
diff --git a/atest/robot/running/if/complex_if.robot b/atest/robot/running/if/complex_if.robot
index 49afc29aaca..1e4662b1f12 100644
--- a/atest/robot/running/if/complex_if.robot
+++ b/atest/robot/running/if/complex_if.robot
@@ -14,7 +14,7 @@ If inside for loop
Setting after if
${tc} = Check Test Case ${TESTNAME}
- Check Log Message ${tc.teardown.msgs[0]} Teardown was found and executed.
+ Check Log Message ${tc.teardown[0]} Teardown was found and executed.
For loop inside if
Check Test Case ${TESTNAME}
@@ -24,23 +24,23 @@ For loop inside for loop
Direct Boolean condition
${tc} = Check Test Case ${TESTNAME}
- Should Be Equal ${tc.body[0].status} PASS
- Should Be Equal ${tc.body[0].body[0].status} PASS
- Should Be Equal ${tc.body[0].body[0].body[0].status} PASS
+ Should Be Equal ${tc[0].status} PASS
+ Should Be Equal ${tc[0, 0].status} PASS
+ Should Be Equal ${tc[0, 0, 0].status} PASS
Direct Boolean condition false
${tc} = Check Test Case ${TESTNAME}
- Should Be Equal ${tc.kws[0].status} PASS
- Should Be Equal ${tc.body[0].body[0].status} NOT RUN
- Should Be Equal ${tc.body[0].body[0].body[0].status} NOT RUN
+ Should Be Equal ${tc[0].status} PASS
+ Should Be Equal ${tc[0, 0].status} NOT RUN
+ Should Be Equal ${tc[0, 0, 0].status} NOT RUN
Nesting insanity
Check Test Case ${TESTNAME}
Recursive If
${tc} = Check Test Case ${TESTNAME}
- Should Be Equal ${tc.kws[0].kws[0].status} PASS
- Should Be Equal ${tc.kws[0].kws[0].kws[0].kws[0].status} PASS
+ Should Be Equal ${tc[0, 0].status} PASS
+ Should Be Equal ${tc[0, 0, 0, 0].status} PASS
If creating variable
Check Test Case ${TESTNAME}
diff --git a/atest/robot/running/if/if_else.robot b/atest/robot/running/if/if_else.robot
index e4b489659bf..6331aeb4dfc 100644
--- a/atest/robot/running/if/if_else.robot
+++ b/atest/robot/running/if/if_else.robot
@@ -38,3 +38,10 @@ If failing in keyword
If failing in else keyword
Check Test Case ${TESTNAME}
+
+Expression evaluation time is included in elapsed time
+ ${tc} = Check Test Case ${TESTNAME}
+ Elapsed Time Should Be Valid ${tc[0].elapsed_time} minimum=0.2
+ Elapsed Time Should Be Valid ${tc[0, 0].elapsed_time} minimum=0.1
+ Elapsed Time Should Be Valid ${tc[0, 1].elapsed_time} minimum=0.1
+ Elapsed Time Should Be Valid ${tc[0, 2].elapsed_time} maximum=1.0
diff --git a/atest/robot/running/if/inline_if_else.robot b/atest/robot/running/if/inline_if_else.robot
index 216fb0124a6..d3a5a346db5 100644
--- a/atest/robot/running/if/inline_if_else.robot
+++ b/atest/robot/running/if/inline_if_else.robot
@@ -22,10 +22,10 @@ Not executed after failure
Not executed after failure with assignment
[Template] NONE
${tc} = Check Test Case ${TEST NAME}
- Check IF/ELSE Status NOT RUN NOT RUN root=${tc.body[1]} run=False
- Check IF/ELSE Status NOT RUN NOT RUN root=${tc.body[2]} run=False
- Check Keyword Data ${tc.body[1].body[0].body[0]} Not run assign=\${x} status=NOT RUN
- Check Keyword Data ${tc.body[2].body[0].body[0]} Not run assign=\${x}, \@{y} status=NOT RUN
+ Check IF/ELSE Status NOT RUN NOT RUN root=${tc[1]} run=False
+ Check IF/ELSE Status NOT RUN NOT RUN root=${tc[2]} run=False
+ Check Keyword Data ${tc[1, 0, 0]} Not run assign=\${x} status=NOT RUN
+ Check Keyword Data ${tc[2, 0, 0]} Not run assign=\${x}, \@{y} status=NOT RUN
ELSE IF not executed
NOT RUN NOT RUN PASS index=0
@@ -48,6 +48,11 @@ Assign
NOT RUN PASS NOT RUN index=1
NOT RUN NOT RUN PASS index=2
+Assign with item
+ PASS NOT RUN NOT RUN index=0
+ NOT RUN PASS NOT RUN index=1
+ NOT RUN NOT RUN PASS index=2
+
Multi assign
PASS NOT RUN index=0
FAIL NOT RUN index=4
@@ -59,6 +64,9 @@ List assign
Dict assign
NOT RUN PASS
+Assign based on another variable
+ PASS NOT RUN index=1
+
Assign without ELSE
PASS NOT RUN index=0
NOT RUN PASS NOT RUN index=2
@@ -71,20 +79,20 @@ Assign when no branch is run
Inside FOR
[Template] NONE
${tc} = Check Test Case ${TEST NAME}
- Check IF/ELSE Status NOT RUN PASS root=${tc.body[0].body[0].body[0]}
- Check IF/ELSE Status NOT RUN PASS root=${tc.body[0].body[1].body[0]}
- Check IF/ELSE Status FAIL NOT RUN root=${tc.body[0].body[2].body[0]}
+ Check IF/ELSE Status NOT RUN PASS root=${tc[0, 0, 0]}
+ Check IF/ELSE Status NOT RUN PASS root=${tc[0, 1, 0]}
+ Check IF/ELSE Status FAIL NOT RUN root=${tc[0, 2, 0]}
Inside normal IF
[Template] NONE
${tc} = Check Test Case ${TEST NAME}
- Check IF/ELSE Status NOT RUN PASS root=${tc.body[0].body[0].body[1]}
- Check IF/ELSE Status NOT RUN NOT RUN root=${tc.body[0].body[1].body[0]} run=False
+ Check IF/ELSE Status NOT RUN PASS root=${tc[0, 0, 1]}
+ Check IF/ELSE Status NOT RUN NOT RUN root=${tc[0, 1, 0]} run=False
In keyword
[Template] NONE
${tc} = Check Test Case ${TEST NAME}
- Check IF/ELSE Status PASS NOT RUN root=${tc.body[0].body[0]}
- Check IF/ELSE Status NOT RUN PASS NOT RUN root=${tc.body[0].body[1]}
+ Check IF/ELSE Status PASS NOT RUN root=${tc[0, 0]}
+ Check IF/ELSE Status NOT RUN PASS NOT RUN root=${tc[0, 1]}
Check IF/ELSE Status NOT RUN NOT RUN NOT RUN FAIL
- ... NOT RUN NOT RUN NOT RUN root=${tc.body[0].body[2]}
+ ... NOT RUN NOT RUN NOT RUN root=${tc[0, 2]}
diff --git a/atest/robot/running/if/invalid_if.robot b/atest/robot/running/if/invalid_if.robot
index a720be8347d..714a7b4eb80 100644
--- a/atest/robot/running/if/invalid_if.robot
+++ b/atest/robot/running/if/invalid_if.robot
@@ -16,12 +16,24 @@ IF with invalid condition
IF with invalid condition with ELSE
FAIL NOT RUN
-IF condition with non-existing variable
+IF condition with non-existing ${variable}
+ FAIL NOT RUN
+
+IF condition with non-existing $variable
FAIL NOT RUN
ELSE IF with invalid condition
NOT RUN NOT RUN FAIL NOT RUN NOT RUN
+Recommend $var syntax if invalid condition contains ${var}
+ FAIL index=1
+
+$var recommendation with multiple variables
+ FAIL index=1
+
+Remove quotes around variable in $var recommendation
+ FAIL index=2
+
IF without END
FAIL
@@ -38,7 +50,7 @@ ELSE IF without condition
ELSE IF with multiple conditions
[Template] NONE
${tc} = Branch statuses should be FAIL NOT RUN NOT RUN
- Should Be Equal ${tc.body[0].body[1].condition} \${False}, ooops, \${True}
+ Should Be Equal ${tc[0, 1].condition} \${False}, ooops, \${True}
ELSE with condition
FAIL NOT RUN
@@ -58,19 +70,59 @@ ELSE after ELSE
ELSE IF after ELSE
FAIL NOT RUN NOT RUN
+Dangling ELSE
+ [Template] Check Test Case
+ ${TEST NAME}
+
+Dangling ELSE inside FOR
+ [Template] Check Test Case
+ ${TEST NAME}
+
+Dangling ELSE inside WHILE
+ [Template] Check Test Case
+ ${TEST NAME}
+
+Dangling ELSE IF
+ [Template] Check Test Case
+ ${TEST NAME}
+
+Dangling ELSE IF inside FOR
+ [Template] Check Test Case
+ ${TEST NAME}
+
+Dangling ELSE IF inside WHILE
+ [Template] Check Test Case
+ ${TEST NAME}
+
+Dangling ELSE IF inside TRY
+ [Template] Check Test Case
+ ${TEST NAME}
+
Invalid IF inside FOR
FAIL
Multiple errors
FAIL NOT RUN NOT RUN NOT RUN NOT RUN
+Invalid data causes syntax error
+ [Template] Check Test Case
+ ${TEST NAME}
+
+Invalid condition causes normal error
+ [Template] Check Test Case
+ ${TEST NAME}
+
+Non-existing variable in condition causes normal error
+ [Template] Check Test Case
+ ${TEST NAME}
+
*** Keywords ***
Branch statuses should be
- [Arguments] @{statuses}
+ [Arguments] @{statuses} ${index}=0
${tc} = Check Test Case ${TESTNAME}
- Should Be Equal ${tc.body[0].status} FAIL
- FOR ${branch} ${status} IN ZIP ${tc.body[0].body} ${statuses}
+ ${if} = Set Variable ${tc.body}[${index}]
+ Should Be Equal ${if.status} FAIL
+ FOR ${branch} ${status} IN ZIP ${if.body} ${statuses} mode=STRICT
Should Be Equal ${branch.status} ${status}
END
- Should Be Equal ${{len($tc.body[0].body)}} ${{len($statuses)}}
RETURN ${tc}
diff --git a/atest/robot/running/if/invalid_inline_if.robot b/atest/robot/running/if/invalid_inline_if.robot
index 4609f4f9d88..5a7ba2c0ac0 100644
--- a/atest/robot/running/if/invalid_inline_if.robot
+++ b/atest/robot/running/if/invalid_inline_if.robot
@@ -65,10 +65,11 @@ Unnecessary END
Invalid END after inline header
[Template] NONE
${tc} = Check Test Case ${TEST NAME}
- Check IF/ELSE Status PASS root=${tc.body[0]}
- Check Log Message ${tc.body[0].body[0].body[0].body[0]} Executed inside inline IF
- Check Log Message ${tc.body[1].body[0]} Executed outside IF
- Check Keyword Data ${tc.body[2]} Reserved.End status=FAIL
+ Check IF/ELSE Status PASS root=${tc[0]}
+ Check Log Message ${tc[0, 0, 0, 0]} Executed inside inline IF
+ Check Log Message ${tc[1, 0]} Executed outside IF
+ Should Be Equal ${tc[2].type} ERROR
+ Should Be Equal ${tc[2].status} FAIL
Assign in IF branch
FAIL
@@ -103,7 +104,12 @@ Assign when ELSE IF branch is empty
Assign when ELSE branch is empty
FAIL NOT RUN
-Assign with RETURN
+Control structures are allowed
[Template] NONE
${tc} = Check Test Case ${TESTNAME}
- Check IF/ELSE Status FAIL NOT RUN root=${tc.body[0].body[0]}
+ Check IF/ELSE Status NOT RUN PASS root=${tc[0, 0]}
+
+Control structures are not allowed with assignment
+ [Template] NONE
+ ${tc} = Check Test Case ${TESTNAME}
+ Check IF/ELSE Status FAIL NOT RUN root=${tc[0, 0]}
diff --git a/atest/robot/running/invalid_break_and_continue.robot b/atest/robot/running/invalid_break_and_continue.robot
index 6aeafe8493d..6730a116b6a 100644
--- a/atest/robot/running/invalid_break_and_continue.robot
+++ b/atest/robot/running/invalid_break_and_continue.robot
@@ -26,11 +26,11 @@ CONTINUE in TRY-ELSE
CONTINUE with argument in FOR
${tc} = Check Test Case ${TESTNAME}
- Check Log Message ${tc.body[0].body[0].body[1].body[0]} CONTINUE does not accept arguments, got 'should not work'. FAIL
+ Check Log Message ${tc[0, 0, 1, 0]} CONTINUE does not accept arguments, got 'should not work'. FAIL
CONTINUE with argument in WHILE
${tc} = Check Test Case ${TESTNAME}
- Check Log Message ${tc.body[0].body[0].body[1].body[0]} CONTINUE does not accept arguments, got 'should', 'not' and 'work'. FAIL
+ Check Log Message ${tc[0, 0, 1, 0]} CONTINUE does not accept arguments, got 'should', 'not' and 'work'. FAIL
BREAK in test case
Check Test Case ${TESTNAME}
@@ -55,8 +55,8 @@ BREAK in TRY-ELSE
BREAK with argument in FOR
${tc} = Check Test Case ${TESTNAME}
- Check Log Message ${tc.body[0].body[0].body[1].body[0]} BREAK does not accept arguments, got 'should not work'. FAIL
+ Check Log Message ${tc[0, 0, 1, 0]} BREAK does not accept arguments, got 'should not work'. FAIL
BREAK with argument in WHILE
${tc} = Check Test Case ${TESTNAME}
- Check Log Message ${tc.body[0].body[0].body[1].body[0]} BREAK does not accept arguments, got 'should', 'not' and 'work'. FAIL
+ Check Log Message ${tc[0, 0, 1, 0]} BREAK does not accept arguments, got 'should', 'not' and 'work'. FAIL
diff --git a/atest/robot/running/long_error_messages.robot b/atest/robot/running/long_error_messages.robot
index 3f1187be2f2..abef7c57b16 100644
--- a/atest/robot/running/long_error_messages.robot
+++ b/atest/robot/running/long_error_messages.robot
@@ -42,21 +42,21 @@ Has Been Cut
Should Contain ${test.message} ${EXPLANATION}
Should Match Non Empty Regexp ${test.message} ${eol_dots}
Should Match Non Empty Regexp ${test.message} ${bol_dots}
- Error Message In Log Should Not Have Been Cut ${test.kws}
- [Return] ${test}
+ Error Message In Log Should Not Have Been Cut ${test}
+ RETURN ${test}
Error Message In Log Should Not Have Been Cut
- [Arguments] ${kws}
- @{keywords} = Set Variable ${kws}
- FOR ${kw} IN @{keywords}
- Run Keyword If ${kw.msgs}
- ... Should Not Contain ${kw.msgs[-1].message} ${EXPLANATION}
- Error Message In Log Should Not Have Been Cut ${kw.kws}
+ [Arguments] ${item}
+ FOR ${child} IN @{item.non_messages}
+ FOR ${msg} IN @{child.messages}
+ Should Not Contain ${msg.message} ${EXPLANATION}
+ END
+ Error Message In Log Should Not Have Been Cut ${child}
END
Should Match Non Empty Regexp
[Arguments] ${message} ${pattern}
- Run Keyword If $pattern
+ IF $pattern
... Should Match Regexp ${message} ${pattern}
Has Not Been Cut
diff --git a/atest/robot/running/non_ascii_bytes.robot b/atest/robot/running/non_ascii_bytes.robot
index 18b726cd5c2..b84792d1a6e 100644
--- a/atest/robot/running/non_ascii_bytes.robot
+++ b/atest/robot/running/non_ascii_bytes.robot
@@ -8,26 +8,26 @@ Variables ${DATADIR}/running/expbytevalues.py
*** Test Cases ***
In Message
${tc}= Check Test Case ${TESTNAME}
- Check Log Message ${tc.kws[0].msgs[0]} ${exp_log_msg}
+ Check Log Message ${tc[0, 0]} ${exp_log_msg}
In Multiline Message
${tc}= Check Test Case ${TESTNAME}
- Check Log Message ${tc.kws[0].msgs[0]} ${exp_log_multiline_msg}
+ Check Log Message ${tc[0, 0]} ${exp_log_multiline_msg}
In Return Value
[Documentation] Return value is not altered by the framework and thus it
... contains the exact same bytes that the keyword returned.
${tc} = Check Test Case ${TESTNAME}
- Check Log Message ${tc.kws[0].msgs[0]} \${retval} = ${exp_return_msg}
+ Check Log Message ${tc[0, 0]} \${retval} = ${exp_return_msg}
In Exception
${tc} = Check Test Case ${TESTNAME}
- Check Log Message ${tc.kws[0].msgs[0]} ${exp_error_msg} FAIL
+ Check Log Message ${tc[0, 0]} ${exp_error_msg} FAIL
In Exception In Setup
${tc} = Check Test Case ${TESTNAME}
- Check Log Message ${tc.setup.msgs[0]} ${exp_error_msg} FAIL
+ Check Log Message ${tc.setup[0]} ${exp_error_msg} FAIL
In Exception In Teardown
${tc} = Check Test Case ${TESTNAME}
- Check Log Message ${tc.teardown.msgs[0]} ${exp_error_msg} FAIL
+ Check Log Message ${tc.teardown[0]} ${exp_error_msg} FAIL
diff --git a/atest/robot/running/pass_execution.robot b/atest/robot/running/pass_execution.robot
index 2fa03944982..4bea02d7167 100644
--- a/atest/robot/running/pass_execution.robot
+++ b/atest/robot/running/pass_execution.robot
@@ -1,9 +1,9 @@
*** Settings ***
-Suite Setup Run Tests ${EMPTY} running/pass_execution.robot
-Resource atest_resource.robot
+Suite Setup Run Tests ${EMPTY} running/pass_execution.robot
+Resource atest_resource.robot
*** Variables ***
-${PREFIX}= Execution passed with message:\n
+${PREFIX}= Execution passed with message:\n
*** Test Cases ***
Message is required
@@ -11,11 +11,11 @@ Message is required
With message
${tc}= Check Test Tags ${TESTNAME} force1 force2
- Check Log Message ${tc.kws[0].msgs[0]} ${PREFIX}My message
+ Check Log Message ${tc[0, 0]} ${PREFIX}My message
With HTML message
${tc}= Check Test Tags ${TESTNAME} force1 force2
- Check Log Message ${tc.kws[0].msgs[0]} ${PREFIX}Message HTML
+ Check Log Message ${tc[0, 0]} ${PREFIX}Message HTML
Empty message is not allowed
Check Test Case ${TESTNAME}
@@ -40,17 +40,17 @@ Used in template keyword
Used in for loop
${tc}= Check Test Case ${TESTNAME}
- Check Log Message ${tc.kws[0].kws[0].kws[0].msgs[0]} ${PREFIX}Message with 'foo'
+ Check Log Message ${tc[0, 0, 0, 0]} ${PREFIX}Message with 'foo'
Used in setup
${tc} = Check Test Case ${TESTNAME}
- Keyword Should Have Been Executed ${tc.kws[0]}
+ Keyword Should Have Been Executed ${tc[0]}
Keyword Should Have Been Executed ${tc.teardown}
Used in teardown
${tc}= Check Test Case ${TESTNAME}
Should Be Equal ${tc.teardown.status} PASS
- Check Log Message ${tc.teardown.kws[0].msgs[0]} ${PREFIX}This message is used.
+ Check Log Message ${tc.teardown[0, 0]} ${PREFIX}This message is used.
Before failing teardown
Check Test Case ${TESTNAME}
@@ -60,14 +60,14 @@ After continuable failure
After continuable failure in user keyword
${tc}= Check Test Case ${TESTNAME}
- Should Be Equal ${tc.kws[0].status} FAIL
+ Should Be Equal ${tc[0].status} FAIL
After continuable failure in FOR loop
${tc}= Check Test Case ${TESTNAME}
- Should Be Equal ${tc.kws[0].status} FAIL
- Should Be Equal ${tc.kws[0].kws[0].status} FAIL
- Should Be Equal ${tc.kws[0].kws[0].kws[0].status} FAIL
- Should Be Equal ${tc.kws[0].kws[0].kws[1].status} PASS
+ Should Be Equal ${tc[0].status} FAIL
+ Should Be Equal ${tc[0, 0].status} FAIL
+ Should Be Equal ${tc[0, 0, 0].status} FAIL
+ Should Be Equal ${tc[0, 0, 1].status} PASS
After continuable failure and before failing teardown
Check Test Case ${TESTNAME}
@@ -86,57 +86,57 @@ After continuable failure in keyword teardown
Remove one tag
${tc}= Check Test Tags ${TESTNAME} force2
- Check Log Message ${tc.kws[0].msgs[0]} Removed tag 'force1'.
- Check Log Message ${tc.kws[0].msgs[1]} ${PREFIX}Message
+ Check Log Message ${tc[0, 0]} Removed tag 'force1'.
+ Check Log Message ${tc[0, 1]} ${PREFIX}Message
Remove multiple tags
${tc}= Check Test Tags ${TESTNAME}
- Check Log Message ${tc.kws[0].msgs[0]} Removed tags 'force1' and 'force2'.
- Check Log Message ${tc.kws[0].msgs[1]} ${PREFIX}Message
+ Check Log Message ${tc[0, 0]} Removed tags 'force1' and 'force2'.
+ Check Log Message ${tc[0, 1]} ${PREFIX}Message
Remove tags with pattern
${tc}= Check Test Tags ${TESTNAME}
- Check Log Message ${tc.kws[0].msgs[0]} Removed tag 'force?'.
- Check Log Message ${tc.kws[0].msgs[1]} ${PREFIX}Message
+ Check Log Message ${tc[0, 0]} Removed tag 'force?'.
+ Check Log Message ${tc[0, 1]} ${PREFIX}Message
Set one tag
${tc}= Check Test Tags ${TESTNAME} force1 force2 tag
- Check Log Message ${tc.kws[0].msgs[0]} Set tag 'tag'.
- Check Log Message ${tc.kws[0].msgs[1]} ${PREFIX}Message
+ Check Log Message ${tc[0, 0]} Set tag 'tag'.
+ Check Log Message ${tc[0, 1]} ${PREFIX}Message
Set multiple tags
${tc}= Check Test Tags ${TESTNAME} force1 force2 tag1 tag2
- Check Log Message ${tc.kws[0].msgs[0]} Set tags 'tag1' and 'tag2'.
- Check Log Message ${tc.kws[0].msgs[1]} ${PREFIX}Message
+ Check Log Message ${tc[0, 0]} Set tags 'tag1' and 'tag2'.
+ Check Log Message ${tc[0, 1]} ${PREFIX}Message
Set and remove tags
${tc}= Check Test Tags ${TESTNAME} tag1 tag2
- Check Log Message ${tc.kws[0].msgs[0]} Removed tag 'force?'.
- Check Log Message ${tc.kws[0].msgs[1]} Set tags 'tag1' and 'tag2'.
- Check Log Message ${tc.kws[0].msgs[2]} ${PREFIX}Message
+ Check Log Message ${tc[0, 0]} Removed tag 'force?'.
+ Check Log Message ${tc[0, 1]} Set tags 'tag1' and 'tag2'.
+ Check Log Message ${tc[0, 2]} ${PREFIX}Message
Set tags are not removed
${tc}= Check Test Tags ${TESTNAME} force1 force2 tag1 tag2
- Check Log Message ${tc.kws[0].msgs[0]} Removed tag 'tag?'.
- Check Log Message ${tc.kws[0].msgs[1]} Set tags 'tag1' and 'tag2'.
- Check Log Message ${tc.kws[0].msgs[2]} ${PREFIX}Message
+ Check Log Message ${tc[0, 0]} Removed tag 'tag?'.
+ Check Log Message ${tc[0, 1]} Set tags 'tag1' and 'tag2'.
+ Check Log Message ${tc[0, 2]} ${PREFIX}Message
Set tags in teardown
${tc}= Check Test Tags ${TESTNAME} tag1 tag2
- Check Log Message ${tc.teardown.msgs[0]} Removed tag 'force?'.
- Check Log Message ${tc.teardown.msgs[1]} Set tags 'tag1' and 'tag2'.
- Check Log Message ${tc.teardown.msgs[2]} ${PREFIX}Message
+ Check Log Message ${tc.teardown[0]} Removed tag 'force?'.
+ Check Log Message ${tc.teardown[1]} Set tags 'tag1' and 'tag2'.
+ Check Log Message ${tc.teardown[2]} ${PREFIX}Message
Pass Execution If when condition is true
Check Test Case ${TESTNAME}
Pass Execution If when condition is false
${tc} = Check Test Case ${TESTNAME}
- Keyword Should Have Been Executed ${tc.kws[1]}
+ Keyword Should Have Been Executed ${tc[1]}
Pass Execution If resolves variables only condition is true
${tc} = Check Test Case ${TESTNAME}
- Keyword Should Have Been Executed ${tc.kws[1]}
+ Keyword Should Have Been Executed ${tc[1]}
Pass Execution If with multiple variables
Check Test Tags ${TESTNAME} force1 force2 my tags
diff --git a/atest/robot/running/prevent_recursion.robot b/atest/robot/running/prevent_recursion.robot
deleted file mode 100644
index 8641c4e476d..00000000000
--- a/atest/robot/running/prevent_recursion.robot
+++ /dev/null
@@ -1,21 +0,0 @@
-*** Settings ***
-Suite Setup Run Tests ${EMPTY} running/prevent_recursion.robot
-Resource atest_resource.robot
-
-*** Test Cases ***
-Infinite recursion
- Check Test Case ${TESTNAME}
-
-Infinite cyclic recursion
- Check Test Case ${TESTNAME}
-
-Infinite recursion with Run Keyword
- Check Test Case ${TESTNAME}
-
-Infinitely recursive for loop
- Check Test Case ${TESTNAME}
-
-Recursion below the recursion limit is ok
- [Documentation] Also verifies that recursion limit blown earlier doesn't affect subsequent tests
- Check Test Case ${TESTNAME}
-
diff --git a/atest/robot/running/return.robot b/atest/robot/running/return.robot
index 6937411cab8..a94de10b833 100644
--- a/atest/robot/running/return.robot
+++ b/atest/robot/running/return.robot
@@ -5,42 +5,51 @@ Resource atest_resource.robot
*** Test Cases ***
Simple
${tc} = Check Test Case ${TESTNAME}
- Should Be Equal ${tc.body[0].body[1].type} RETURN
- Should Be Equal ${tc.body[0].body[1].status} PASS
- Should Be Equal ${tc.body[0].body[2].status} NOT RUN
+ Should Be Equal ${tc[0, 1].type} RETURN
+ Should Be Equal ${tc[0, 1].values} ${{()}}
+ Should Be Equal ${tc[0, 1].status} PASS
+ Should Be Equal ${tc[0, 1].message} ${EMPTY}
+ Should Be Equal ${tc[0, 2].status} NOT RUN
+ Should Be Equal ${tc[0].message} ${EMPTY}
Return value
- Check Test Case ${TESTNAME}
+ ${tc} = Check Test Case ${TESTNAME}
+ Should Be Equal ${tc[0, 0].type} RETURN
+ Should Be Equal ${tc[0, 0].values} ${{('value',)}}
Return value as variable
- Check Test Case ${TESTNAME}
+ ${tc} = Check Test Case ${TESTNAME}
+ Should Be Equal ${tc[0, 0].type} RETURN
+ Should Be Equal ${tc[0, 0].values} ${{('\${42}',)}}
Return multiple values
- Check Test Case ${TESTNAME}
+ ${tc} = Check Test Case ${TESTNAME}
+ Should Be Equal ${tc[0, 0].type} RETURN
+ Should Be Equal ${tc[0, 0].values} ${{('first', '\${2}', 'third')}}
In nested keyword
Check Test Case ${TESTNAME}
In IF
${tc} = Check Test Case ${TESTNAME}
- Should Be Equal ${tc.body[0].body[0].body[0].body[0].type} RETURN
- Should Be Equal ${tc.body[0].body[0].body[0].body[0].status} PASS
- Should Be Equal ${tc.body[0].body[0].body[0].body[1].status} NOT RUN
- Should Be Equal ${tc.body[0].body[1].status} NOT RUN
- Should Be Equal ${tc.body[2].body[0].body[1].body[0].type} RETURN
- Should Be Equal ${tc.body[2].body[0].body[1].body[0].status} PASS
- Should Be Equal ${tc.body[2].body[0].body[1].body[1].status} NOT RUN
- Should Be Equal ${tc.body[2].body[1].status} NOT RUN
+ Should Be Equal ${tc[0, 0, 0, 0].type} RETURN
+ Should Be Equal ${tc[0, 0, 0, 0].status} PASS
+ Should Be Equal ${tc[0, 0, 0, 1].status} NOT RUN
+ Should Be Equal ${tc[0, 1].status} NOT RUN
+ Should Be Equal ${tc[2, 0, 1, 0].type} RETURN
+ Should Be Equal ${tc[2, 0, 1, 0].status} PASS
+ Should Be Equal ${tc[2, 0, 1, 1].status} NOT RUN
+ Should Be Equal ${tc[2, 1].status} NOT RUN
In inline IF
Check Test Case ${TESTNAME}
In FOR
${tc} = Check Test Case ${TESTNAME}
- Should Be Equal ${tc.body[0].body[0].body[0].body[0].type} RETURN
- Should Be Equal ${tc.body[0].body[0].body[0].body[0].status} PASS
- Should Be Equal ${tc.body[0].body[0].body[0].body[1].status} NOT RUN
- Should Be Equal ${tc.body[0].body[1].status} NOT RUN
+ Should Be Equal ${tc[0, 0, 0, 0].type} RETURN
+ Should Be Equal ${tc[0, 0, 0, 0].status} PASS
+ Should Be Equal ${tc[0, 0, 0, 1].status} NOT RUN
+ Should Be Equal ${tc[0, 1].status} NOT RUN
In nested FOR/IF structure
Check Test Case ${TESTNAME}
diff --git a/atest/robot/running/return_from_keyword.robot b/atest/robot/running/return_from_keyword.robot
index 4c9c2fa72b6..34b905cfa3c 100644
--- a/atest/robot/running/return_from_keyword.robot
+++ b/atest/robot/running/return_from_keyword.robot
@@ -56,5 +56,5 @@ Return From Keyword If does not evaluate bogus arguments if condition is untrue
Logs Info
${tc} = Check Test Case Without Return Value
- Check Log Message ${tc.kws[0].kws[0].msgs[0]}
+ Check Log Message ${tc[0, 0, 0]}
... Returning from the enclosing user keyword.
diff --git a/atest/robot/running/setup_and_teardown_using_embedded_arguments.robot b/atest/robot/running/setup_and_teardown_using_embedded_arguments.robot
new file mode 100644
index 00000000000..49d2660f2d4
--- /dev/null
+++ b/atest/robot/running/setup_and_teardown_using_embedded_arguments.robot
@@ -0,0 +1,14 @@
+*** Settings ***
+Suite Setup Run Tests ${EMPTY} running/setup_and_teardown_using_embedded_arguments.robot
+Resource atest_resource.robot
+
+*** Test Cases ***
+Suite setup and teardown
+ Should Be Equal ${SUITE.setup.status} PASS
+ Should Be Equal ${SUITE.teardown.status} PASS
+
+Test setup and teardown
+ Check Test Case ${TESTNAME}
+
+Keyword setup and teardown
+ Check Test Case ${TESTNAME}
diff --git a/atest/robot/running/skip.robot b/atest/robot/running/skip.robot
index e6f8726acf1..1af592e235f 100644
--- a/atest/robot/running/skip.robot
+++ b/atest/robot/running/skip.robot
@@ -44,7 +44,7 @@ Skip in Teardown After Failure In Body
Teardown is executed after skip
${tc} = Check Test Case ${TEST NAME}
- Check log message ${tc.teardown.msgs[0]} Teardown is executed!
+ Check log message ${tc.teardown[0]} Teardown is executed!
Fail in Teardown After Skip In Body
Check Test Case ${TEST NAME}
@@ -91,6 +91,10 @@ Skip In Suite Teardown After Fail In Setup
Skip In Directory Suite Teardown
Check Test Case ${TEST NAME}
+Tests have correct status if suite has nothing to run and directory suite setup uses skip
+ Check Test Case `robot:skip` with skip in directory suite setup
+ Check Test Case `--skip` with skip in directory suite setup
+
Skip with Run Keyword and Ignore Error
Check Test Case ${TEST NAME}
@@ -106,24 +110,58 @@ Skip with Wait Until Keyword Succeeds
Skipped with --skip
Check Test Case ${TEST NAME}
-Skipped when test is tagged with robot:skip
+Skipped with --skip when tag uses variable
+ Check Test Case ${TEST NAME}
+
+Skipped with robot:skip
+ Check Test Case ${TEST NAME}
+
+Skipped with robot:skip when tag uses variable
Check Test Case ${TEST NAME}
Skipped with --SkipOnFailure
Check Test Case ${TEST NAME}
-Skipped with --SkipOnFailure when Failure in Test Setup
+Skipped with --SkipOnFailure when tag uses variable
Check Test Case ${TEST NAME}
-Skipped with --SkipOnFailure when Failure in Test Teardown
+Skipped with --SkipOnFailure when failure in setup
Check Test Case ${TEST NAME}
-Skipped with --SkipOnFailure when Set Tags Used in Teardown
- Check Test Case Skipped with --SkipOnFailure when Set Tags Used in Teardown
+Skipped with --SkipOnFailure when failure in teardown
+ Check Test Case ${TEST NAME}
-Skipped although test fails since test is tagged with robot:skip-on-failure
+Skipped with --SkipOnFailure when Set Tags used in teardown
Check Test Case ${TEST NAME}
-Using Skip Does Not Affect Passing And Failing Tests
- Check Test Case Passing Test
- Check Test Case Failing Test
+Skipped with robot:skip-on-failure
+ Check Test Case ${TEST NAME}
+
+Skipped with robot:skip-on-failure when tag uses variable
+ Check Test Case ${TEST NAME}
+
+Skipping does not affect passing and failing tests
+ Check Test Case Passing
+ Check Test Case Failing
+
+Suite setup and teardown are not run if all tests are unconditionally skipped or excluded
+ ${suite} = Get Test Suite All Skipped
+ Should Be True not ($suite.setup or $suite.teardown)
+ Should Be True not ($suite.suites[0].setup or $suite.suites[0].teardown)
+ Check Test Case Skip using robot:skip
+ Check Test Case Skip using --skip
+ Length Should Be ${suite.suites[0].tests} 2
+
+--skip and --skip-on-failure used multiple times
+ Run Tests --skip skip-this --skip no-match --SkipOnFailure skip-on-failure --skip-on-failure xxx running/skip/skip.robot
+ Check Test Case Skipped with --skip
+ ... message=Test skipped using 'no-match' and 'skip-this' tags.
+ Check Test Case Skipped with --SkipOnFailure
+ ... message=Failed test skipped using 'skip-on-failure' and 'xxx' tags.\n\nOriginal failure:\nOoops, we fail!
+
+--skip and --skip-on-failure with patterns
+ Run Tests --skip skip-t*s --skip no-match --SkipOnFailure xxxORskip-on-failure running/skip/skip.robot
+ Check Test Case Skipped with --skip
+ ... message=Test skipped using 'no-match' and 'skip-t*s' tag patterns.
+ Check Test Case Skipped with --SkipOnFailure
+ ... message=Failed test skipped using 'xxx OR skip-on-failure' tag pattern.\n\nOriginal failure:\nOoops, we fail!
diff --git a/atest/robot/running/skip_in_rpa_mode.robot b/atest/robot/running/skip_in_rpa_mode.robot
index e5370328d05..6da7ef27bfd 100644
--- a/atest/robot/running/skip_in_rpa_mode.robot
+++ b/atest/robot/running/skip_in_rpa_mode.robot
@@ -1,5 +1,5 @@
*** Settings ***
-Suite Setup Run Tests --rpa --skip skip-this --SkipOnFailure skip-on-failure --variable test_or_task:Task running/skip/
+Suite Setup Run Tests --rpa --skip skip-this --SkipOnFailure skip-on-failure --variable test_or_task:task running/skip/
Resource atest_resource.robot
*** Test Cases ***
@@ -8,4 +8,3 @@ Skipped with --skip
Skipped with --SkipOnFailure
Check Test Case ${TEST NAME}
-
diff --git a/atest/robot/running/skip_with_template.robot b/atest/robot/running/skip_with_template.robot
new file mode 100644
index 00000000000..a642c665146
--- /dev/null
+++ b/atest/robot/running/skip_with_template.robot
@@ -0,0 +1,71 @@
+*** Settings ***
+Suite Setup Run Tests ${EMPTY} running/skip_with_template.robot
+Resource atest_resource.robot
+
+*** Test Cases ***
+SKIP + PASS -> PASS
+ ${tc} = Check Test Case ${TEST NAME}
+ Status Should Be ${tc[0]} SKIP Skipped
+ Status Should Be ${tc[1]} PASS
+
+FAIL + ANY -> FAIL
+ ${tc} = Check Test Case ${TEST NAME}
+ Status Should Be ${tc[0]} PASS
+ Status Should Be ${tc[1]} SKIP Skipped
+ Status Should Be ${tc[2]} PASS
+ Status Should Be ${tc[3]} FAIL Failed
+ Status Should Be ${tc[4]} SKIP Skipped
+
+Only SKIP -> SKIP
+ ${tc} = Check Test Case ${TEST NAME}
+ Status Should Be ${tc[0]} SKIP Skipped
+ Status Should Be ${tc[1]} SKIP Skipped
+
+IF w/ SKIP + PASS -> PASS
+ ${tc} = Check Test Case ${TEST NAME}
+ Status Should Be ${tc[0]} PASS
+ Status Should Be ${tc[1]} SKIP Skipped
+ Status Should Be ${tc[2]} PASS
+
+IF w/ FAIL + ANY -> FAIL
+ ${tc} = Check Test Case ${TEST NAME}
+ Status Should Be ${tc[0]} FAIL Failed
+ Status Should Be ${tc[1]} SKIP Skipped
+ Status Should Be ${tc[2]} PASS
+
+IF w/ only SKIP -> SKIP
+ ${tc} = Check Test Case ${TEST NAME}
+ Status Should Be ${tc[0]} SKIP All iterations skipped.
+ Status Should Be ${tc[1]} SKIP Skip 3
+ Status Should Be ${tc[2]} SKIP Skip 4
+
+FOR w/ SKIP + PASS -> PASS
+ ${tc} = Check Test Case ${TEST NAME}
+ Status Should Be ${tc[0]} PASS
+ Status Should Be ${tc[1]} SKIP just once
+ Status Should Be ${tc[2]} PASS
+
+FOR w/ FAIL + ANY -> FAIL
+ ${tc} = Check Test Case ${TEST NAME}
+ Status Should Be ${tc[0]} FAIL Several failures occurred:\n\n1) a\n\n2) b
+ Status Should Be ${tc[1]} SKIP just once
+ Status Should Be ${tc[2]} PASS
+
+FOR w/ only SKIP -> SKIP
+ ${tc} = Check Test Case ${TEST NAME}
+ Status Should Be ${tc[0]} SKIP All iterations skipped.
+ Status Should Be ${tc[1]} SKIP just once
+
+Messages in test body are ignored
+ ${tc} = Check Test Case ${TEST NAME}
+ Check Log Message ${tc[0]} Hello 'Messages in test body are ignored', says listener!
+ Check Log Message ${tc[1, 0, 0]} Library listener adds messages to body of this test.
+ Check Log Message ${tc[2, 0, 0]} This iteration is skipped! SKIP
+ Check Log Message ${tc[3, 0, 0]} This iteration passes!
+ Check Log Message ${tc[4]} Bye 'Messages in test body are ignored', says listener!
+
+*** Keywords ***
+Status Should Be
+ [Arguments] ${item} ${status} ${message}=
+ Should Be Equal ${item.status} ${status}
+ Should Be Equal ${item.message} ${message}
diff --git a/atest/robot/running/steps_after_failure.robot b/atest/robot/running/steps_after_failure.robot
index 30a1de2dc96..602f40d3001 100644
--- a/atest/robot/running/steps_after_failure.robot
+++ b/atest/robot/running/steps_after_failure.robot
@@ -5,143 +5,159 @@ Resource atest_resource.robot
*** Test Cases ***
Library keyword after failure
${tc} = Check Test Case ${TESTNAME}
- Should Not Be Run ${tc.body[2:]} 5
- Check Log Message ${tc.teardown.msgs[0]} This is run
+ Should Not Be Run ${tc[2:]} 5
+ Check Log Message ${tc.teardown[0]} This is run
User keyword after failure
${tc} = Check Test Case ${TESTNAME}
- Should Not Be Run ${tc.body[1:]}
+ Should Not Be Run ${tc[1:]}
Non-existing keyword after failure
${tc} = Check Test Case ${TESTNAME}
- Should Not Be Run ${tc.body[1:]}
+ Should Not Be Run ${tc[1:]}
Invalid keyword usage after failure
${tc} = Check Test Case ${TESTNAME}
- Should Not Be Run ${tc.body[1:]}
+ Should Not Be Run ${tc[1:]}
Assignment after failure
${tc} = Check Test Case ${TESTNAME}
- Should Not Be Run ${tc.body[1:]} 4
- Check Keyword Data ${tc.body[1]} Not run assign=\${x} status=NOT RUN
- Check Keyword Data ${tc.body[2]} Not run assign=\${x} status=NOT RUN
- Check Keyword Data ${tc.body[3]} Not run assign=\${x}, \${y} status=NOT RUN
- Check Keyword Data ${tc.body[4]} Not run assign=\${x}, \${y} status=NOT RUN
+ Should Not Be Run ${tc[1:]} 4
+ Check Keyword Data ${tc[1]} Not run assign=\${x} status=NOT RUN
+ Check Keyword Data ${tc[2]} Not run assign=\${x} status=NOT RUN
+ Check Keyword Data ${tc[3]} Not run assign=\${x}, \${y} status=NOT RUN
+ Check Keyword Data ${tc[4]} Not run assign=\${x}, \${y} status=NOT RUN
IF after failure
${tc} = Check Test Case ${TESTNAME}
- Should Not Be Run ${tc.body[1:]}
- Should Not Be Run ${tc.body[1].body[0].body}
- Should Not Be Run ${tc.body[1].body[1].body}
- Check Keyword Data ${tc.body[1].body[1].body[0]}
+ Should Not Be Run ${tc[1:]}
+ Should Not Be Run ${tc[1, 0].body}
+ Should Not Be Run ${tc[1, 1].body}
+ Check Keyword Data ${tc[1, 1, 0]}
+ ... BuiltIn.Fail assign=\${x} args=This should not be run status=NOT RUN
+
+GROUP after failure
+ ${tc} = Check Test Case ${TESTNAME}
+ Should Not Be Run ${tc[1:]}
+ Should Not Be Run ${tc[1].body} 2
+ Check Keyword Data ${tc[1, 1]}
... BuiltIn.Fail assign=\${x} args=This should not be run status=NOT RUN
FOR after failure
${tc} = Check Test Case ${TESTNAME}
- Should Not Be Run ${tc.body[1:]}
- Should Not Be Run ${tc.body[1].body}
- Should Not Be Run ${tc.body[1].body[0].body} 2
- Check Keyword Data ${tc.body[1].body[0].body[1]}
+ Should Not Be Run ${tc[1:]}
+ Should Not Be Run ${tc[1].body}
+ Should Not Be Run ${tc[1, 0].body} 2
+ Check Keyword Data ${tc[1, 0, 1]}
... BuiltIn.Fail assign=\${x} args=This should not be run either status=NOT RUN
TRY after failure
${tc} = Check Test Case ${TESTNAME}
- Should Not Be Run ${tc.body[1:]}
- Should Not Be Run ${tc.body[1].body} 4
- FOR ${step} IN @{tc.body[1].body}
+ Should Not Be Run ${tc[1:]}
+ Should Not Be Run ${tc[1].body} 4
+ FOR ${step} IN @{tc[1].body}
Should Not Be Run ${step.body}
END
WHILE after failure
${tc} = Check Test Case ${TESTNAME}
- Should Not Be Run ${tc.body[1:]} 3
- Should Not Be Run ${tc.body[1].body}
- Should Not Be Run ${tc.body[1].body[0].body} 3
- Should Not Be Run ${tc.body[2].body}
- Should Not Be Run ${tc.body[2].body[0].body} 2
- Should Not Be Run ${tc.body[3].body}
- Should Not Be Run ${tc.body[3].body[0].body} 1
+ Should Not Be Run ${tc[1:]} 3
+ Should Not Be Run ${tc[1].body}
+ Should Not Be Run ${tc[1, 0].body} 3
+ Should Not Be Run ${tc[2].body}
+ Should Not Be Run ${tc[2, 0].body} 2
+ Should Not Be Run ${tc[3].body}
+ Should Not Be Run ${tc[3, 0].body} 1
RETURN after failure
${tc} = Check Test Case ${TESTNAME}
- Should Not Be Run ${tc.body[1:]}
- Should Not Be Run ${tc.body[0].body[1:]} 2
- Should Be Equal ${tc.body[0].body[1].type} RETURN
+ Should Not Be Run ${tc[1:]}
+ Should Not Be Run ${tc[0][1:]} 2
+ Should Be Equal ${tc[0, 1].type} RETURN
BREAK and CONTINUE after failure
${tc} = Check Test Case ${TESTNAME}
- Should Not Be Run ${tc.body[1:]} 1
- Should Not Be Run ${tc.body[0].body[0].body[1:]} 2
- Should Not Be Run ${tc.body[1].body}
- Should Not Be Run ${tc.body[1].body[0].body} 2
+ Should Not Be Run ${tc[1:]} 1
+ Should Not Be Run ${tc[0, 0][1:]} 2
+ Should Not Be Run ${tc[1].body}
+ Should Not Be Run ${tc[1, 0].body} 2
Nested control structure after failure
${tc} = Check Test Case ${TESTNAME}
- Should Not Be Run ${tc.body[1:]} 2
- Should Be Equal ${tc.body[1].type} FOR
- Should Not Be Run ${tc.body[1].body} 1
- Should Be Equal ${tc.body[1].body[0].type} ITERATION
- Should Not Be Run ${tc.body[1].body[0].body} 2
- Should Be Equal ${tc.body[1].body[0].body[0].type} IF/ELSE ROOT
- Should Not Be Run ${tc.body[1].body[0].body[0].body} 2
- Should Be Equal ${tc.body[1].body[0].body[0].body[0].type} IF
- Should Not Be Run ${tc.body[1].body[0].body[0].body[0].body} 2
- Should Be Equal ${tc.body[1].body[0].body[0].body[0].body[0].type} FOR
- Should Not Be Run ${tc.body[1].body[0].body[0].body[0].body[0].body} 1
- Should Be Equal ${tc.body[1].body[0].body[0].body[0].body[0].body[0].type} ITERATION
- Should Not Be Run ${tc.body[1].body[0].body[0].body[0].body[0].body[0].body} 3
- Should Be Equal ${tc.body[1].body[0].body[0].body[0].body[0].body[0].body[0].type} KEYWORD
- Should Be Equal ${tc.body[1].body[0].body[0].body[0].body[0].body[0].body[1].type} KEYWORD
- Should Be Equal ${tc.body[1].body[0].body[0].body[0].body[0].body[0].body[2].type} KEYWORD
- Should Be Equal ${tc.body[1].body[0].body[0].body[0].body[1].type} KEYWORD
- Should Be Equal ${tc.body[1].body[0].body[0].body[1].type} ELSE
- Should Not Be Run ${tc.body[1].body[0].body[0].body[1].body} 2
- Should Be Equal ${tc.body[1].body[0].body[0].body[1].body[0].type} WHILE
- Should Not Be Run ${tc.body[1].body[0].body[0].body[1].body[0].body} 1
- Should Be Equal ${tc.body[1].body[0].body[0].body[1].body[0].body[0].type} ITERATION
- Should Not Be Run ${tc.body[1].body[0].body[0].body[1].body[0].body[0].body} 2
- Should Be Equal ${tc.body[1].body[0].body[0].body[1].body[0].body[0].body[0].type} KEYWORD
- Should Be Equal ${tc.body[1].body[0].body[0].body[1].body[0].body[0].body[1].type} KEYWORD
- Should Be Equal ${tc.body[1].body[0].body[0].body[1].body[1].type} TRY/EXCEPT ROOT
- Should Not Be Run ${tc.body[1].body[0].body[0].body[1].body[1].body} 2
- Should Be Equal ${tc.body[1].body[0].body[0].body[1].body[1].body[0].type} TRY
- Should Not Be Run ${tc.body[1].body[0].body[0].body[1].body[1].body[0].body} 1
- Should Be Equal ${tc.body[1].body[0].body[0].body[1].body[1].body[0].body[0].type} KEYWORD
- Should Be Equal ${tc.body[1].body[0].body[0].body[1].body[1].body[1].type} EXCEPT
- Should Not Be Run ${tc.body[1].body[0].body[0].body[1].body[1].body[1].body} 1
- Should Be Equal ${tc.body[1].body[0].body[0].body[1].body[1].body[1].body[0].type} BREAK
- Should Be Equal ${tc.body[1].body[0].body[1].type} KEYWORD
- Should Be Equal ${tc.body[2].type} KEYWORD
+ Should Not Be Run ${tc[1:]} 2
+ Should Be Equal ${tc[1].type} FOR
+ Should Not Be Run ${tc[1].body} 1
+ Should Be Equal ${tc[1, 0].type} ITERATION
+ Should Not Be Run ${tc[1, 0].body} 2
+ Should Be Equal ${tc[1, 0, 0].type} IF/ELSE ROOT
+ Should Not Be Run ${tc[1, 0, 0].body} 2
+ Should Be Equal ${tc[1, 0, 0, 0].type} IF
+ Should Not Be Run ${tc[1, 0, 0, 0].body} 2
+ Should Be Equal ${tc[1, 0, 0, 0, 0].type} FOR
+ Should Not Be Run ${tc[1, 0, 0, 0, 0].body} 1
+ Should Be Equal ${tc[1, 0, 0, 0, 0, 0].type} ITERATION
+ Should Not Be Run ${tc[1, 0, 0, 0, 0, 0].body} 2
+ Should Be Equal ${tc[1, 0, 0, 0, 0, 0, 0].type} KEYWORD
+ Should Be Equal ${tc[1, 0, 0, 0, 0, 0, 1].type} GROUP
+ Should Not Be Run ${tc[1, 0, 0, 0, 0, 0, 1].body} 2
+ Should Be Equal ${tc[1, 0, 0, 0, 0, 0, 1, 0].type} KEYWORD
+ Should Be Equal ${tc[1, 0, 0, 0, 0, 0, 1, 1].type} KEYWORD
+ Should Be Equal ${tc[1, 0, 0, 0, 1].type} KEYWORD
+ Should Be Equal ${tc[1, 0, 0, 1].type} ELSE
+ Should Not Be Run ${tc[1, 0, 0, 1].body} 2
+ Should Be Equal ${tc[1, 0, 0, 1, 0].type} WHILE
+ Should Not Be Run ${tc[1, 0, 0, 1, 0].body} 1
+ Should Be Equal ${tc[1, 0, 0, 1, 0, 0].type} ITERATION
+ Should Not Be Run ${tc[1, 0, 0, 1, 0, 0].body} 2
+ Should Be Equal ${tc[1, 0, 0, 1, 0, 0, 0].type} KEYWORD
+ Should Be Equal ${tc[1, 0, 0, 1, 0, 0, 1].type} KEYWORD
+ Should Be Equal ${tc[1, 0, 0, 1, 1].type} TRY/EXCEPT ROOT
+ Should Not Be Run ${tc[1, 0, 0, 1, 1].body} 2
+ Should Be Equal ${tc[1, 0, 0, 1, 1, 0].type} TRY
+ Should Not Be Run ${tc[1, 0, 0, 1, 1, 0].body} 1
+ Should Be Equal ${tc[1, 0, 0, 1, 1, 0, 0].type} KEYWORD
+ Should Be Equal ${tc[1, 0, 0, 1, 1, 1].type} EXCEPT
+ Should Not Be Run ${tc[1, 0, 0, 1, 1, 1].body} 1
+ Should Be Equal ${tc[1, 0, 0, 1, 1, 1, 0].type} BREAK
+ Should Be Equal ${tc[1, 0, 1].type} KEYWORD
+ Should Be Equal ${tc[2].type} KEYWORD
Failure in user keyword
${tc} = Check Test Case ${TESTNAME}
- Should Not Be Run ${tc.body[1:]}
- Should Not Be Run ${tc.body[0].body[1:]} 2
+ Should Not Be Run ${tc[1:]}
+ Should Not Be Run ${tc[0][1:]} 2
Failure in IF branch
${tc} = Check Test Case ${TESTNAME}
- Should Not Be Run ${tc.body[0].body[0].body[1:]}
- Should Not Be Run ${tc.body[0].body[1].body}
- Should Not Be Run ${tc.body[1:]}
+ Should Not Be Run ${tc[0, 0][1:]}
+ Should Not Be Run ${tc[0, 1].body}
+ Should Not Be Run ${tc[1:]}
Failure in ELSE IF branch
${tc} = Check Test Case ${TESTNAME}
- Should Not Be Run ${tc.body[0].body[0].body}
- Should Not Be Run ${tc.body[0].body[1].body[1:]}
- Should Not Be Run ${tc.body[0].body[2].body}
- Should Not Be Run ${tc.body[1:]}
+ Should Not Be Run ${tc[0, 0].body}
+ Should Not Be Run ${tc[0, 1][1:]}
+ Should Not Be Run ${tc[0, 2].body}
+ Should Not Be Run ${tc[1:]}
Failure in ELSE branch
${tc} = Check Test Case ${TESTNAME}
- Should Not Be Run ${tc.body[0].body[0].body}
- Should Not Be Run ${tc.body[0].body[1].body[1:]}
- Should Not Be Run ${tc.body[1:]}
+ Should Not Be Run ${tc[0, 0].body}
+ Should Not Be Run ${tc[0, 1][1:]}
+ Should Not Be Run ${tc[1:]}
+
+Failure in GROUP
+ ${tc} = Check Test Case ${TESTNAME}
+ Should Not Be Run ${tc[0, 0][1:]}
+ Should Not Be Run ${tc[0][1:]} 2
+ Should Not Be Run ${tc[0, 2].body}
+ Should Not Be Run ${tc[1:]}
Failure in FOR iteration
${tc} = Check Test Case ${TESTNAME}
- Should Not Be Run ${tc.body[1:]}
- Length Should Be ${tc.body[0].body} 1
- Should Not Be Run ${tc.body[0].body[0].body[1:]}
+ Should Not Be Run ${tc[1:]}
+ Length Should Be ${tc[0].body} 1
+ Should Not Be Run ${tc[0, 0][1:]}
*** Keywords ***
Should Not Be Run
diff --git a/atest/robot/running/stopping_with_signal.robot b/atest/robot/running/stopping_with_signal.robot
index a1a70c1d154..f93f2eb4348 100644
--- a/atest/robot/running/stopping_with_signal.robot
+++ b/atest/robot/running/stopping_with_signal.robot
@@ -59,8 +59,8 @@ One Signal Should Stop Test Execution Gracefully And Test Case And Suite Teardow
Start And Send Signal with_teardown.robot One SIGINT
Check Test Cases Have Failed Correctly
${tc} = Get Test Case Test
- Check Log Message ${tc.teardown.msgs[0]} Logging Test Case Teardown
- Check Log Message ${SUITE.teardown.kws[0].msgs[0]} Logging Suite Teardown
+ Check Log Message ${tc.teardown[0]} Logging Test Case Teardown
+ Check Log Message ${SUITE.teardown[0, 0]} Logging Suite Teardown
Skip Teardowns After Stopping Gracefully
Start And Send Signal with_teardown.robot One SIGINT 0s --SkipTeardownOnExit
@@ -69,6 +69,32 @@ Skip Teardowns After Stopping Gracefully
Teardown Should Not Be Defined ${tc}
Teardown Should Not Be Defined ${SUITE}
+SIGINT Signal Should Stop Async Test Execution Gracefully
+ Start And Send Signal async_stop.robot One SIGINT 5
+ Check Test Cases Have Failed Correctly
+ ${tc} = Get Test Case Test
+ Length Should Be ${tc[1].body} 1
+ Check Log Message ${tc[1, 0]} Start Sleep
+ Length Should Be ${SUITE.teardown.body} 0
+
+Two SIGINT Signals Should Stop Async Test Execution Forcefully
+ Start And Send Signal async_stop.robot Two SIGINTs 5
+ Check Tests Have Been Forced To Shutdown
+
+SIGTERM Signal Should Stop Async Test Execution Gracefully
+ [Tags] no-windows
+ Start And Send Signal async_stop.robot One SIGTERM 5
+ Check Test Cases Have Failed Correctly
+ ${tc} = Get Test Case Test
+ Length Should Be ${tc[1].body} 1
+ Check Log Message ${tc[1, 0]} Start Sleep
+ Length Should Be ${SUITE.teardown.body} 0
+
+Two SIGTERM Signals Should Stop Async Test Execution Forcefully
+ [Tags] no-windows
+ Start And Send Signal async_stop.robot Two SIGTERMs 5
+ Check Tests Have Been Forced To Shutdown
+
*** Keywords ***
Start And Send Signal
[Arguments] ${datasource} ${signals} ${sleep}=0s @{extra options}
diff --git a/atest/robot/running/test_case_status.robot b/atest/robot/running/test_case_status.robot
index 842747f7f03..55e4ae27b1a 100644
--- a/atest/robot/running/test_case_status.robot
+++ b/atest/robot/running/test_case_status.robot
@@ -1,11 +1,11 @@
-*** Setting ***
+*** Settings ***
Documentation Tests for setting test case status correctly when test passes
... and when a failure or error occurs. Also includes test cases
... for running test setup and teardown in different situations.
Suite Setup Run Tests ${EMPTY} running/test_case_status.robot
Resource atest_resource.robot
-*** Test Case ***
+*** Test Cases ***
Test Passes
Check Test Case ${TEST NAME}
@@ -41,12 +41,10 @@ Test Setup And Teardown Pass
Check Test Case ${TEST NAME}
Test Teardown is Run When Setup Fails
- ${test} Check Test Case ${TEST NAME}
- ${td} = Set Variable ${test.teardown}
- Should Not Be Equal ${td} ${None} Teardown not run No values
- Length Should Be ${td.msgs} 1
- Check Log Message ${td.msgs[0]} Hello from teardown!
- Length Should Be ${td.kws} 0
+ ${tc} = Check Test Case ${TEST NAME}
+ Should Not Be Equal ${tc.teardown} ${None} Teardown not run No values
+ Length Should Be ${tc.teardown.body} 1
+ Check Log Message ${tc.teardown[0]} Hello from teardown!
Test Setup And Teardown Fails
Check Test Case ${TEST NAME}
diff --git a/atest/robot/running/test_template.robot b/atest/robot/running/test_template.robot
index c6306bc90d2..ba2a2e22941 100644
--- a/atest/robot/running/test_template.robot
+++ b/atest/robot/running/test_template.robot
@@ -59,32 +59,32 @@ Invalid FOR
Template With IF
${tc} = Check Test Case ${TESTNAME}
- Should Be Equal ${tc.body[0].status} PASS
- Should Be Equal ${tc.body[0].body[0].type} IF
- Should Be Equal ${tc.body[0].body[0].status} NOT RUN
- Should Be Equal ${tc.body[0].body[1].type} ELSE IF
- Should Be Equal ${tc.body[0].body[1].status} NOT RUN
- Should Be Equal ${tc.body[0].body[2].type} ELSE
- Should Be Equal ${tc.body[0].body[2].status} PASS
+ Should Be Equal ${tc[0].status} PASS
+ Should Be Equal ${tc[0, 0].type} IF
+ Should Be Equal ${tc[0, 0].status} NOT RUN
+ Should Be Equal ${tc[0, 1].type} ELSE IF
+ Should Be Equal ${tc[0, 1].status} NOT RUN
+ Should Be Equal ${tc[0, 2].type} ELSE
+ Should Be Equal ${tc[0, 2].status} PASS
Template With IF Failing
${tc} = Check Test Case ${TESTNAME}
- Should Be Equal ${tc.body[0].status} FAIL
- Should Be Equal ${tc.body[0].body[0].type} IF
- Should Be Equal ${tc.body[0].body[0].status} FAIL
- Should Be Equal ${tc.body[1].status} FAIL
- Should Be Equal ${tc.body[1].body[0].type} IF
- Should Be Equal ${tc.body[1].body[0].status} NOT RUN
- Should Be Equal ${tc.body[1].body[1].type} ELSE IF
- Should Be Equal ${tc.body[1].body[1].status} FAIL
- Should Be Equal ${tc.body[1].body[2].type} ELSE
- Should Be Equal ${tc.body[1].body[2].status} NOT RUN
+ Should Be Equal ${tc[0].status} FAIL
+ Should Be Equal ${tc[0, 0].type} IF
+ Should Be Equal ${tc[0, 0].status} FAIL
+ Should Be Equal ${tc[1].status} FAIL
+ Should Be Equal ${tc[1, 0].type} IF
+ Should Be Equal ${tc[1, 0].status} NOT RUN
+ Should Be Equal ${tc[1, 1].type} ELSE IF
+ Should Be Equal ${tc[1, 1].status} FAIL
+ Should Be Equal ${tc[1, 2].type} ELSE
+ Should Be Equal ${tc[1, 2].status} NOT RUN
Invalid IF
${tc} = Check Test Case ${TESTNAME}
- Should Be Equal ${tc.body[0].status} FAIL
- Should Be Equal ${tc.body[0].body[0].type} IF
- Should Be Equal ${tc.body[0].body[0].status} FAIL
+ Should Be Equal ${tc[0].status} FAIL
+ Should Be Equal ${tc[0, 0].type} IF
+ Should Be Equal ${tc[0, 0].status} FAIL
FOR and IF
Check Test Case ${TESTNAME}
@@ -116,7 +116,7 @@ Templated test with for loop continues after keyword timeout
Templated test ends after syntax errors
Check Test Case ${TESTNAME}
-Templated test continues after variable error
+Templated test continues after non-syntax errors
Check Test Case ${TESTNAME}
Templates and fatal errors
diff --git a/atest/robot/running/test_template_with_embeded_args.robot b/atest/robot/running/test_template_with_embeded_args.robot
index 680406bce21..65b3d427abe 100644
--- a/atest/robot/running/test_template_with_embeded_args.robot
+++ b/atest/robot/running/test_template_with_embeded_args.robot
@@ -5,39 +5,39 @@ Resource atest_resource.robot
*** Test Cases ***
Matching arguments
${tc} = Check Test Case ${TESTNAME}
- Keyword should be ${tc.kws[0]} The result of 1 + 1 should be 2
- Keyword should be ${tc.kws[1]} The result of 1 + 2 should be 3
- Keyword should be ${tc.kws[2]} The result of 1 + 3 should be 5
+ Keyword should be ${tc[0]} The result of 1 + 1 should be 2
+ Keyword should be ${tc[1]} The result of 1 + 2 should be 3
+ Keyword should be ${tc[2]} The result of 1 + 3 should be 5
Argument names do not need to be same as in definition
${tc} = Check Test Case ${TESTNAME}
- Keyword should be ${tc.kws[0]} The result of 1 + 1 should be 2
- Keyword should be ${tc.kws[1]} The result of 1 + 2 should be 3
- Keyword should be ${tc.kws[2]} The result of 1 + 3 should be 5
+ Keyword should be ${tc[0]} The result of 1 + 1 should be 2
+ Keyword should be ${tc[1]} The result of 1 + 2 should be 3
+ Keyword should be ${tc[2]} The result of 1 + 3 should be 5
Some arguments can be hard-coded
${tc} = Check Test Case ${TESTNAME}
- Keyword should be ${tc.kws[0]} The result of 1 + 1 should be 3
- Keyword should be ${tc.kws[1]} The result of 1 + 2 should be 3
- Keyword should be ${tc.kws[2]} The result of 1 + 3 should be 3
+ Keyword should be ${tc[0]} The result of 1 + 1 should be 3
+ Keyword should be ${tc[1]} The result of 1 + 2 should be 3
+ Keyword should be ${tc[2]} The result of 1 + 3 should be 3
Can have different arguments than definition
${tc} = Check Test Case ${TESTNAME}
- Keyword should be ${tc.kws[0]} The result of 38 + 3 + 1 should be 42
- Keyword should be ${tc.kws[1]} The non-existing of 666 should be 42
+ Keyword should be ${tc[0]} The result of 38 + 3 + 1 should be 42
+ Keyword should be ${tc[1]} The non-existing of 666 should be 42
Can use variables
${tc} = Check Test Case ${TESTNAME}
- Keyword should be ${tc.kws[0]} The result of \${1} + \${2} should be \${3}
+ Keyword should be ${tc[0]} The result of \${1} + \${2} should be \${3}
Cannot have more arguments than variables
${tc} = Check Test Case ${TESTNAME}
- Keyword should be ${tc.kws[0]} The result of \${calc} should be 3
+ Keyword should be ${tc[0]} The result of \${calc} should be 3
... 1 + 2 extra
Cannot have less arguments than variables
${tc} = Check Test Case ${TESTNAME}
- Keyword should be ${tc.kws[0]} The result of \${calc} should be \${extra}
+ Keyword should be ${tc[0]} The result of \${calc} should be \${extra}
... 1 + 2
*** Keywords ***
diff --git a/atest/robot/running/timeouts.robot b/atest/robot/running/timeouts.robot
index 6a0b6edbb57..020db103dd5 100644
--- a/atest/robot/running/timeouts.robot
+++ b/atest/robot/running/timeouts.robot
@@ -16,14 +16,14 @@ Timeouted Test Passes
Timeouted Test Fails Before Timeout
Check Test Case Failing Before Timeout
-Show Correct Trace Back When Failing Before Timeout
+Show Correct Traceback When Failing Before Timeout
${tc} = Check Test Case ${TEST NAME}
${expected} = Catenate SEPARATOR=\n
... Traceback (most recent call last):
... ${SPACE*2}File "*", line *, in exception
... ${SPACE*4}raise exception(msg)
... RuntimeError: Failure before timeout
- Check Log Message ${tc.kws[0].msgs[-1]} ${expected} pattern=yes level=DEBUG
+ Check Log Message ${tc[0, -1]} ${expected} DEBUG pattern=True traceback=True
Timeouted Test Timeouts
Check Test Case Sleeping And Timeouting
@@ -63,17 +63,17 @@ Test Timeouts When Also Keywords Are Timeouted
Keyword Timeout From Variable
${tc} = Check Test Case ${TEST NAME}
- Should Be Equal ${tc.kws[0].timeout} 1 millisecond
+ Should Be Equal ${tc[0].timeout} 1 millisecond
Keyword Timeout From Argument
${tc} = Check Test Case ${TEST NAME}
- Should Be Equal ${tc.kws[0].timeout} 1 second
- Should Be Equal ${tc.kws[1].timeout} 2 milliseconds
+ Should Be Equal ${tc[0].timeout} 1 second
+ Should Be Equal ${tc[1].timeout} 2 milliseconds
Embedded Arguments Timeout From Argument
${tc} = Check Test Case ${TEST NAME}
- Should Be Equal ${tc.kws[0].timeout} 1 second
- Should Be Equal ${tc.kws[1].timeout} 3 milliseconds
+ Should Be Equal ${tc[0].timeout} 1 second
+ Should Be Equal ${tc[1].timeout} 3 milliseconds
Local Variables Are Not Visible In Child Keyword Timeout
Check Test Case ${TEST NAME}
@@ -88,9 +88,9 @@ Test Timeout During Setup
Teardown After Test Timeout
[Documentation] Test that teardown is executed after a test has timed out
${tc} = Check Test Case ${TEST NAME}
- Check Log Message ${tc.teardown.msgs[0]} Teardown executed
+ Check Log Message ${tc.teardown[0]} Teardown executed
${tc} = Check Test Case Teardown With Sleep After Test Timeout
- Check Log Message ${tc.teardown.kws[1].msgs[0]} Teardown executed
+ Check Log Message ${tc.teardown[1, 0]} Teardown executed
Failing Teardown After Test Timeout
Check Test Case ${TEST NAME}
@@ -98,7 +98,7 @@ Failing Teardown After Test Timeout
Test Timeout During Teardown
[Documentation] Test timeout should not interrupt teardown but test should be failed afterwards
${tc} = Check Test Case ${TEST NAME}
- Check Log Message ${tc.teardown.kws[1].msgs[0]} Teardown executed
+ Check Log Message ${tc.teardown[1, 0]} Teardown executed
Timeouted Setup Passes
Check Test Case ${TEST NAME}
@@ -133,8 +133,8 @@ Keyword Timeout Should Not Be Active For Run Keyword Variants But To Keywords Th
Logging With Timeouts
[Documentation] Testing that logging works with timeouts
${tc} = Check Test Case Timeouted Keyword Passes
- Check Log Message ${tc.kws[0].msgs[1]} Testing logging in timeouted test
- Check Log Message ${tc.kws[1].kws[0].msgs[1]} Testing logging in timeouted keyword
+ Check Log Message ${tc[0, 1]} Testing logging in timeouted test
+ Check Log Message ${tc[1, 0, 1]} Testing logging in timeouted keyword
Timeouted Keyword Called With Wrong Number of Arguments
Check Test Case ${TEST NAME}
@@ -144,31 +144,31 @@ Timeouted Keyword Called With Wrong Number of Arguments with Run Keyword
Test Timeout Logging
${tc} = Check Test Case Passing
- Timeout should have been active ${tc.kws[0]} 1 second 1
+ Timeout should have been active ${tc[0]} 1 second 1
${tc} = Check Test Case Failing Before Timeout
- Timeout should have been active ${tc.kws[0]} 2 seconds 3
+ Timeout should have been active ${tc[0]} 2 seconds 3
${tc} = Check Test Case Sleeping And Timeouting
- Timeout should have been active ${tc.kws[0]} 1 second 2 exceeded=True
+ Timeout should have been active ${tc[0]} 1 second 2 exceeded=True
Keyword Timeout Logging
${tc} = Check Test Case Timeouted Keyword Passes
- Keyword timeout should have been active ${tc.kws[1].kws[0]} 5 seconds 2
+ Keyword timeout should have been active ${tc[1, 0]} 5 seconds 2
${tc} = Check Test Case Timeouted Keyword Fails Before Timeout
- Keyword timeout should have been active ${tc.kws[0].kws[0]} 2 hours 30 minutes 3
+ Keyword timeout should have been active ${tc[0, 0]} 2 hours 30 minutes 3
${tc} = Check Test Case Timeouted Keyword Timeouts
- Keyword timeout should have been active ${tc.kws[0].kws[0]} 99 milliseconds 2 exceeded=True
+ Keyword timeout should have been active ${tc[0, 0]} 99 milliseconds 2 exceeded=True
Zero timeout is ignored
${tc} = Check Test Case ${TEST NAME}
Should Be Equal ${tc.timeout} 0 seconds
- Should Be Equal ${tc.kws[0].timeout} 0 seconds
- Should Be True ${tc.kws[0].elapsedtime} > 99
+ Should Be Equal ${tc[0].timeout} 0 seconds
+ Elapsed Time Should Be Valid ${tc[0].elapsed_time} minimum=0.099
Negative timeout is ignored
${tc} = Check Test Case ${TEST NAME}
- Should Be Equal ${tc.kws[0].timeout} - 1 second
- Should Be Equal ${tc.kws[0].timeout} - 1 second
- Should Be True ${tc.kws[0].elapsedtime} > 99
+ Should Be Equal ${tc[0].timeout} - 1 second
+ Should Be Equal ${tc[0].timeout} - 1 second
+ Elapsed Time Should Be Valid ${tc[0].elapsed_time} minimum=0.099
Invalid test timeout
Check Test Case ${TEST NAME}
@@ -179,10 +179,9 @@ Invalid keyword timeout
*** Keywords ***
Timeout should have been active
[Arguments] ${kw} ${timeout} ${msg count} ${exceeded}=False ${type}=Test
- Check Log Message ${kw.msgs[0]} ${type} timeout ${timeout} active. * left. DEBUG pattern=True
- Length Should Be ${kw.msgs} ${msg count}
- Run Keyword If ${exceeded}
- ... Timeout should have exceeded ${kw} ${timeout} ${type}
+ Check Log Message ${kw[0]} ${type} timeout ${timeout} active. * left. DEBUG pattern=True
+ Length Should Be ${kw.body} ${msg count}
+ IF ${exceeded} Timeout should have exceeded ${kw} ${timeout} ${type}
Keyword timeout should have been active
[Arguments] ${kw} ${timeout} ${msg count} ${exceeded}=False
@@ -190,4 +189,4 @@ Keyword timeout should have been active
Timeout should have exceeded
[Arguments] ${kw} ${timeout} ${type}=Test
- Check Log Message ${kw.msgs[1]} ${type} timeout ${timeout} exceeded. FAIL
+ Check Log Message ${kw[1]} ${type} timeout ${timeout} exceeded. FAIL
diff --git a/atest/robot/running/timeouts_with_custom_messages.robot b/atest/robot/running/timeouts_with_custom_messages.robot
index 46127b1ae28..4f21b34fb55 100644
--- a/atest/robot/running/timeouts_with_custom_messages.robot
+++ b/atest/robot/running/timeouts_with_custom_messages.robot
@@ -9,19 +9,15 @@ Default Test Timeout Message
Test Timeout Message
Check Test Case ${TEST NAME}
- Using more than one value with timeout should error 1 9 2
Test Timeout Message In Multiple Columns
Check Test Case ${TEST NAME}
- Using more than one value with timeout should error 2 13 7
Keyword Timeout Message
Check Test Case ${TEST NAME}
- Using more than one value with timeout should error 3 26 2
Keyword Timeout Message In Multiple Columns
Check Test Case ${TEST NAME}
- Using more than one value with timeout should error 4 30 7
*** Keywords ***
Using more than one value with timeout should error
diff --git a/atest/robot/running/try_except/except_behaviour.robot b/atest/robot/running/try_except/except_behaviour.robot
index 74a8adcce74..1ede85342a5 100644
--- a/atest/robot/running/try_except/except_behaviour.robot
+++ b/atest/robot/running/try_except/except_behaviour.robot
@@ -5,19 +5,19 @@ Test Template Verify try except and block statuses
*** Test Cases ***
Equals is the default matcher
- FAIL PASS
+ FAIL PASS pattern_types=[None]
Equals with whitespace
FAIL PASS
Glob matcher
- FAIL NOT RUN PASS
+ FAIL NOT RUN PASS pattern_types=['GloB', 'gloB']
Startswith matcher
- FAIL PASS
+ FAIL PASS pattern_types=['start']
Regexp matcher
- FAIL NOT RUN PASS
+ FAIL NOT RUN PASS pattern_types=['REGEXP', 'REGEXP']
Regexp escapes
FAIL PASS
@@ -35,16 +35,22 @@ Non-string pattern
FAIL NOT RUN NOT RUN NOT RUN NOT RUN
Variable in pattern type
- FAIL PASS
+ FAIL PASS pattern_types=['\${regexp}']
Invalid variable in pattern type
- FAIL FAIL PASS
+ FAIL FAIL PASS pattern_types=['\${does not exist}']
Invalid pattern type
- FAIL FAIL
+ FAIL NOT RUN NOT RUN pattern_types=['glob', 'invalid']
+
+Invalid pattern type from variable
+ FAIL FAIL pattern_types=["\${{'invalid'}}"]
Non-string pattern type
- FAIL FAIL
+ FAIL FAIL pattern_types=['\${42}']
+
+Pattern type multiple times
+ FAIL PASS NOT RUN pattern_types=['start']
Pattern type without patterns
FAIL PASS
@@ -53,7 +59,7 @@ Skip cannot be caught
SKIP NOT RUN PASS tc_status=SKIP
Return cannot be caught
- PASS NOT RUN PASS path=body[0].body[0]
+ PASS NOT RUN PASS path=body[0].body[0]
AS gets the message
FAIL PASS
diff --git a/atest/robot/running/try_except/invalid_try_except.robot b/atest/robot/running/try_except/invalid_try_except.robot
index e69a824b577..45ad4c198fb 100644
--- a/atest/robot/running/try_except/invalid_try_except.robot
+++ b/atest/robot/running/try_except/invalid_try_except.robot
@@ -81,3 +81,10 @@ CONTINUE in FINALLY
RETURN in FINALLY
TRY:PASS FINALLY:FAIL path=body[0].body[0]
+
+Invalid TRY/EXCEPT causes syntax error that cannot be caught
+ TRY:FAIL EXCEPT:NOT RUN ELSE:NOT RUN
+
+Dangling FINALLY
+ [Template] Check Test Case
+ ${TEST NAME}
diff --git a/atest/robot/running/try_except/try_except.robot b/atest/robot/running/try_except/try_except.robot
index aca14fc3d85..8d074cd6446 100644
--- a/atest/robot/running/try_except/try_except.robot
+++ b/atest/robot/running/try_except/try_except.robot
@@ -35,21 +35,21 @@ Default except pattern
FAIL PASS
Syntax errors cannot be caught
- FAIL
+ FAIL NOT RUN NOT RUN
Finally block executed when no failures
[Template] None
${tc}= Verify try except and block statuses PASS NOT RUN PASS PASS
- Check Log Message ${tc.body[0].body[0].body[0].msgs[0]} all good
- Check Log Message ${tc.body[0].body[2].body[0].msgs[0]} in the else
- Check Log Message ${tc.body[0].body[3].body[0].msgs[0]} Hello from finally!
+ Check Log Message ${tc[0, 0, 0, 0]} all good
+ Check Log Message ${tc[0, 2, 0, 0]} in the else
+ Check Log Message ${tc[0, 3, 0, 0]} Hello from finally!
Finally block executed after catch
[Template] None
${tc}= Verify try except and block statuses FAIL PASS PASS
- Check Log Message ${tc.body[0].body[0].body[0].msgs[0]} all not good FAIL
- Check Log Message ${tc.body[0].body[1].body[0].msgs[0]} we are safe now
- Check Log Message ${tc.body[0].body[2].body[0].msgs[0]} Hello from finally!
+ Check Log Message ${tc[0, 0, 0, 0]} all not good FAIL
+ Check Log Message ${tc[0, 1, 0, 0]} we are safe now
+ Check Log Message ${tc[0, 2, 0, 0]} Hello from finally!
Finally block executed after failure in except
FAIL FAIL NOT RUN PASS
diff --git a/atest/robot/running/try_except/try_except_resource.robot b/atest/robot/running/try_except/try_except_resource.robot
index 6f557b433e4..590cc5ffd60 100644
--- a/atest/robot/running/try_except/try_except_resource.robot
+++ b/atest/robot/running/try_except/try_except_resource.robot
@@ -4,9 +4,10 @@ Library Collections
*** Keywords ***
Verify try except and block statuses
- [Arguments] @{types_and_statuses} ${tc_status}= ${path}=body[0]
+ [Arguments] @{types_and_statuses} ${tc_status}= ${path}=body[0] ${pattern_types}=[]
${tc}= Check test status @{{[s.split(':')[-1] for s in $types_and_statuses]}} tc_status=${tc_status}
Block statuses should be ${tc.${path}} @{types_and_statuses}
+ Pattern types should be ${tc.${path}} ${pattern_types}
RETURN ${tc}
Check Test Status
@@ -34,3 +35,11 @@ Block statuses should be
Should Be Equal ${block.status} ${type_and_status}
END
END
+
+Pattern types should be
+ [Arguments] ${try_except} ${pattern_types}
+ @{pattern_types} = Evaluate ${pattern_types}
+ FOR ${except} ${expected} IN ZIP ${try_except.body[1:]} ${pattern_types} mode=shortest
+ Should Be Equal ${except.type} EXCEPT
+ Should Be Equal ${except.pattern_type} ${expected}
+ END
diff --git a/atest/robot/running/while/break_and_continue.robot b/atest/robot/running/while/break_and_continue.robot
index 8fb4aac3eb5..bc932d39d12 100644
--- a/atest/robot/running/while/break_and_continue.robot
+++ b/atest/robot/running/while/break_and_continue.robot
@@ -1,7 +1,7 @@
*** Settings ***
Resource while.resource
Suite Setup Run Tests ${EMPTY} running/while/break_and_continue.robot
-Test Template Check while loop
+Test Template Check WHILE loop
*** Test Cases ***
With CONTINUE
@@ -31,6 +31,24 @@ With BREAK inside EXCEPT
With BREAK inside TRY-ELSE
PASS 1
+BREAK with continuable failures
+ FAIL 1
+
+CONTINUE with continuable failures
+ FAIL 2
+
+Invalid BREAK
+ FAIL 1
+
+Invalid CONTINUE
+ FAIL 1
+
+Invalid BREAK not executed
+ PASS 1
+
+Invalid CONTINUE not executed
+ NOT RUN 1
+
With CONTINUE in UK
PASS 5 body[0].body[0]
diff --git a/atest/robot/running/while/invalid_while.robot b/atest/robot/running/while/invalid_while.robot
index c6687e9b247..e2526405943 100644
--- a/atest/robot/running/while/invalid_while.robot
+++ b/atest/robot/running/while/invalid_while.robot
@@ -1,23 +1,61 @@
*** Settings ***
Resource while.resource
-Suite Setup Run Tests ${EMPTY} running/while/invalid_while.robot
+Suite Setup Run Tests --log test_result_model_as_well running/while/invalid_while.robot
*** Test Cases ***
-No condition
- Check Test Case ${TESTNAME}
-
Multiple conditions
- ${tc} = Check Test Case ${TESTNAME}
- Should Be Equal ${tc.body[0].condition} Too, many, !
+ ${tc} = Check Invalid WHILE Test Case
+ Should Be Equal ${tc[0].condition} Too, many, conditions, !
Invalid condition
- Check Test Case ${TESTNAME}
+ Check Invalid WHILE Test Case
+
+Non-existing ${variable} in condition
+ Check Invalid WHILE Test Case
-Non-existing variable in condition
- Check Test Case ${TESTNAME}
+Non-existing $variable in condition
+ Check Invalid WHILE Test Case
+
+Recommend $var syntax if invalid condition contains ${var}
+ Check Test Case ${TEST NAME}
+
+Invalid condition on second round
+ Check Test Case ${TEST NAME}
No body
- Check Test Case ${TESTNAME}
+ Check Invalid WHILE Test Case body=False
No END
- Check Test Case ${TESTNAME}
+ Check Invalid WHILE Test Case
+
+Invalid data causes syntax error
+ Check Test Case ${TEST NAME}
+
+Invalid condition causes normal error
+ Check Test Case ${TEST NAME}
+
+Non-existing variable in condition causes normal error
+ Check Test Case ${TEST NAME}
+
+Templatest are not supported
+ ${tc} = Check Test Case ${TEST NAME}
+ Should Be Equal ${tc[0].type} WHILE
+ Should Be Equal ${tc[0].status} FAIL
+ Should Be Equal ${tc[0, 0].type} ITERATION
+ Should Be Equal ${tc[0, 0].status} NOT RUN
+ Check Keyword Data ${tc[0, 0, 0]} ${EMPTY} args=1 status=NOT RUN
+ Check Keyword Data ${tc[0, 0, 1]} ${EMPTY} args=2 status=NOT RUN
+
+*** Keywords ***
+Check Invalid WHILE Test Case
+ [Arguments] ${body}=True
+ ${tc} = Check Test Case ${TESTNAME}
+ Should Be Equal ${tc[0].type} WHILE
+ Should Be Equal ${tc[0].status} FAIL
+ Should Be Equal ${tc[0, 0].type} ITERATION
+ Should Be Equal ${tc[0, 0].status} NOT RUN
+ IF ${body}
+ Should Be Equal ${tc[0, 0, 0].full_name} BuiltIn.Fail
+ Should Be Equal ${tc[0, 0, 0].status} NOT RUN
+ END
+ RETURN ${tc}
diff --git a/atest/robot/running/while/nested_while.robot b/atest/robot/running/while/nested_while.robot
index 96e08c15726..e745c002619 100644
--- a/atest/robot/running/while/nested_while.robot
+++ b/atest/robot/running/while/nested_while.robot
@@ -5,24 +5,24 @@ Suite Setup Run Tests ${EMPTY} running/while/nested_while.robot
*** Test Cases ***
Inside FOR
${tc}= Check test case ${TEST NAME}
- Check loop attributes ${tc.body[0].body[0].body[0]} PASS 4
- Check loop attributes ${tc.body[0].body[1].body[0]} PASS 3
- Check loop attributes ${tc.body[0].body[2].body[0]} PASS 2
- Length should be ${tc.body[0].body} 3
+ Check loop attributes ${tc[0, 0, 0]} PASS 4
+ Check loop attributes ${tc[0, 1, 0]} PASS 3
+ Check loop attributes ${tc[0, 2, 0]} PASS 2
+ Length should be ${tc[0].body} 3
Failing inside FOR
${tc}= Check test case ${TEST NAME}
- Check loop attributes ${tc.body[0].body[0].body[0]} FAIL 2
- Length should be ${tc.body[0].body} 1
+ Check loop attributes ${tc[0, 0, 0]} FAIL 2
+ Length should be ${tc[0].body} 1
Inside IF
${tc}= Check test case ${TEST NAME}
- Check loop attributes ${tc.body[0].body[0].body[1]} PASS 4
+ Check loop attributes ${tc[0, 0, 1]} PASS 4
In suite setup
${suite}= Get Test Suite Nested While
- Check loop attributes ${suite.setup.body[1]} PASS 4
+ Check loop attributes ${suite.setup[1]} PASS 4
In suite teardown
${suite}= Get Test Suite Nested While
- Check loop attributes ${suite.teardown.body[1]} PASS 4
+ Check loop attributes ${suite.teardown[1]} PASS 4
diff --git a/atest/robot/running/while/on_limit.robot b/atest/robot/running/while/on_limit.robot
new file mode 100644
index 00000000000..91415d27cb9
--- /dev/null
+++ b/atest/robot/running/while/on_limit.robot
@@ -0,0 +1,59 @@
+*** Settings ***
+Suite Setup Run Tests ${EMPTY} running/while/on_limit.robot
+Resource while.resource
+
+*** Test Cases ***
+On limit pass with time limit defined
+ Check WHILE Loop PASS not known
+
+On limit pass with iteration limit defined
+ Check WHILE loop PASS 5
+
+On limit fail
+ Check WHILE Loop FAIL 5
+
+On limit pass with failures in loop
+ Check WHILE Loop FAIL 1
+
+On limit pass with continuable failure
+ Check WHILE Loop FAIL 2
+
+On limit fail with continuable failure
+ Check WHILE Loop FAIL 2
+
+Invalid on_limit
+ Check WHILE Loop FAIL 1 not_run=True
+
+Invalid on_limit from variable
+ Check WHILE Loop FAIL 1 not_run=True
+
+On limit without limit
+ Check WHILE Loop FAIL 1 not_run=True
+
+On limit with invalid variable
+ Check WHILE Loop FAIL 1 not_run=True
+
+On limit message
+ Check WHILE Loop FAIL 11
+
+On limit message without limit
+ Check WHILE Loop FAIL 10000
+
+On limit message from variable
+ Check WHILE Loop FAIL 5
+
+Part of on limit message from variable
+ Check WHILE Loop FAIL 5
+
+On limit message is not used if limit is not hit
+ Check WHILE Loop PASS 2
+
+Nested while on limit message
+ Check WHILE Loop FAIL 1 path=body[0]
+ Check WHILE Loop FAIL 5 path=body[0].body[0].body[0]
+
+On limit message before limit
+ Check WHILE Loop FAIL 5
+
+On limit message with invalid variable
+ Check WHILE Loop FAIL 1 not_run=True
diff --git a/atest/robot/running/while/while.resource b/atest/robot/running/while/while.resource
index 4ee25df6dc0..392399f1bb6 100644
--- a/atest/robot/running/while/while.resource
+++ b/atest/robot/running/while/while.resource
@@ -1,17 +1,21 @@
*** Settings ***
-Resource atest_resource.robot
-
+Resource atest_resource.robot
*** Keywords ***
-Check while loop
- [Arguments] ${status} ${iterations} ${path}=body[0]
- ${tc}= Check test case ${TEST NAME}
- ${loop}= Check loop attributes ${tc.${path}} ${status} ${iterations}
+Check WHILE loop
+ [Arguments] ${status} ${iterations} ${path}=body[0] ${not_run}=False
+ ${tc}= Check Test Case ${TEST NAME}
+ ${loop}= Check Loop Attributes ${tc.${path}} ${status} ${iterations}
+ IF ${not_run}
+ Should Be Equal ${loop.body[0].status} NOT RUN
+ END
RETURN ${loop}
-Check loop attributes
+Check Loop Attributes
[Arguments] ${loop} ${status} ${iterations}
- Should be equal ${loop.type} WHILE
- Should be equal ${loop.status} ${status}
- Length Should Be ${loop.kws} ${iterations}
+ Should Be Equal ${loop.type} WHILE
+ Should Be Equal ${loop.status} ${status}
+ IF '${iterations}' != 'not known'
+ Length Should Be ${loop.non_messages} ${iterations}
+ END
RETURN ${loop}
diff --git a/atest/robot/running/while/while.robot b/atest/robot/running/while/while.robot
index 91ff24732f1..76580a70e04 100644
--- a/atest/robot/running/while/while.robot
+++ b/atest/robot/running/while/while.robot
@@ -5,7 +5,7 @@ Suite Setup Run Tests ${EMPTY} running/while/while.robot
*** Test Cases ***
Loop executed once
${loop}= Check While Loop PASS 1
- Check Log Message ${loop.body[0].body[0].msgs[0]} 1
+ Check Log Message ${loop[0, 0, 0]} 1
Loop executed multiple times
Check While Loop PASS 5
@@ -17,12 +17,24 @@ Loop not executed
Should Be Equal ${item.status} NOT RUN
END
+No Condition
+ Check While Loop PASS 5
+
Execution fails on the first loop
Check While Loop FAIL 1
Execution fails after some loops
Check While Loop FAIL 3
+Continuable failure in loop
+ Check While Loop FAIL 3
+
+Normal failure after continuable failure in loop
+ Check While Loop FAIL 2
+
+Normal failure outside loop after continuable failures in loop
+ Check While Loop FAIL 2
+
Loop in loop
Check While Loop PASS 5
Check While Loop PASS 3 path=body[0].body[0].body[2]
@@ -35,3 +47,8 @@ Loop fails in keyword
With RETURN
Check While Loop PASS 1 path=body[0].body[0]
+
+Condition evaluation time is included in elapsed time
+ ${loop} = Check WHILE loop PASS 1
+ Elapsed Time Should Be Valid ${loop.elapsed_time} minimum=0.2
+ Elapsed Time Should Be Valid ${loop.body[0].elapsed_time} minimum=0.1
diff --git a/atest/robot/running/while/while_limit.robot b/atest/robot/running/while/while_limit.robot
index e1d29da9799..22185673eee 100644
--- a/atest/robot/running/while/while_limit.robot
+++ b/atest/robot/running/while/while_limit.robot
@@ -4,40 +4,63 @@ Resource while.resource
*** Test Cases ***
Default limit is 10000 iterations
- Check Test Case ${TESTNAME}
+ Check WHILE Loop FAIL 10000
Limit with iteration count
- Check while loop FAIL 5
+ Check WHILE Loop FAIL 5
-Limit with iteration count with spaces
- ${tc}= Check Test Case ${TESTNAME}
- Should Be Equal ${tc.body[0].limit} 3 0
+Iteration count with 'times' suffix
+ Check WHILE Loop FAIL 3
-Limit with iteration count with underscore
- ${tc}= Check Test Case ${TESTNAME}
- Should Be Equal ${tc.body[0].limit} 1_0
+Iteration count with 'x' suffix
+ Check WHILE Loop FAIL 4
+
+Iteration count normalization
+ ${loop}= Check WHILE Loop PASS 1 body[0]
+ Should Be Equal ${loop.limit} 1_000
+ ${loop}= Check WHILE Loop FAIL 30 body[1]
+ Should Be Equal ${loop.limit} 3 0 T i m e S
Limit as timestr
- Check Test Case ${TESTNAME}
+ Check WHILE Loop FAIL not known
Limit from variable
- Check Test Case ${TESTNAME}
+ Check WHILE Loop FAIL 11
Part of limit from variable
- Check Test Case ${TESTNAME}
+ Check WHILE Loop FAIL not known
Limit can be disabled
- Check Test Case ${TESTNAME}
+ Check WHILE Loop PASS 10041
+
+No condition with limit
+ Check WHILE Loop FAIL 2
+
+Limit exceeds in teardown
+ Check WHILE Loop FAIL not known teardown.body[0]
+
+Limit exceeds after failures in teardown
+ Check WHILE Loop FAIL 2 teardown.body[0]
+
+Continue after limit in teardown
+ Check WHILE Loop PASS not known teardown.body[0]
Invalid limit invalid suffix
- Check Test Case ${TESTNAME}
+ Check WHILE Loop FAIL 1 not_run=True
Invalid limit invalid value
- Check Test Case ${TESTNAME}
+ Check WHILE Loop FAIL 1 not_run=True
Invalid limit mistyped prefix
- Check Test Case ${TESTNAME}
+ Check WHILE Loop FAIL 1 not_run=True
+
+Limit with non-existing variable
+ Check WHILE Loop FAIL 1 not_run=True
+
+Limit used multiple times
+ ${loop}= Check WHILE Loop FAIL 1 not_run=True
+ Should Be Equal ${loop.limit} 2
Invalid values after limit
- ${tc} = Check Test Case ${TESTNAME}
- Should Be Equal ${tc.body[0].condition} $variable < 2, limit=-1x, invalid, values
+ ${loop}= Check WHILE Loop FAIL 1 not_run=True
+ Should Be Equal ${loop.condition} $variable < 2, limit=2, invalid
diff --git a/atest/robot/standard_libraries/builtin/builtin_propertys.robot b/atest/robot/standard_libraries/builtin/builtin_propertys.robot
new file mode 100644
index 00000000000..623baaf6a52
--- /dev/null
+++ b/atest/robot/standard_libraries/builtin/builtin_propertys.robot
@@ -0,0 +1,11 @@
+*** Settings ***
+Resource atest_resource.robot
+
+*** Test Cases ***
+Normal run
+ Run Tests ${EMPTY} standard_libraries/builtin/builtin_propertys.robot
+ Check Test Case Test propertys
+
+Dry-run
+ Run Tests --dryrun --variable DRYRUN:True standard_libraries/builtin/builtin_propertys.robot
+ Check Test Case Test propertys
diff --git a/atest/robot/standard_libraries/builtin/call_method.robot b/atest/robot/standard_libraries/builtin/call_method.robot
index e6980cf031a..b32218f47d5 100644
--- a/atest/robot/standard_libraries/builtin/call_method.robot
+++ b/atest/robot/standard_libraries/builtin/call_method.robot
@@ -1,25 +1,35 @@
*** Settings ***
-Suite Setup Run Tests ${EMPTY} standard_libraries/builtin/call_method.robot
+Suite Setup Run Tests --loglevel DEBUG standard_libraries/builtin/call_method.robot
Resource atest_resource.robot
*** Test Cases ***
Call Method
- Check Test Case ${TEST NAME}
+ Check Test Case ${TEST NAME}
Call Method Returns
- Check Test Case ${TEST NAME}
+ Check Test Case ${TEST NAME}
Called Method Fails
- Check Test Case ${TEST NAME}
+ ${tc} = Check Test Case ${TEST NAME}
+ Check Log Message ${tc[0, 0]} Calling method 'my_method' failed: Expected failure FAIL
+ ${error} = Catenate SEPARATOR=\n
+ ... RuntimeError: Expected failure
+ ...
+ ... The above exception was the direct cause of the following exception:
+ ...
+ ... RuntimeError: Calling method 'my_method' failed: Expected failure
+ Traceback Should Be ${tc[0, 1]}
+ ... standard_libraries/builtin/objects_for_call_method.py my_method raise RuntimeError('Expected failure')
+ ... error=${error}
Call Method With Kwargs
- Check Test Case ${TEST NAME}
+ Check Test Case ${TEST NAME}
Equals in non-kwargs must be escaped
- Check Test Case ${TEST NAME}
+ Check Test Case ${TEST NAME}
Call Method From Module
- Check Test Case ${TEST NAME}
+ Check Test Case ${TEST NAME}
Call Non Existing Method
- Check Test Case ${TEST NAME}
+ Check Test Case ${TEST NAME}
diff --git a/atest/robot/standard_libraries/builtin/converter.robot b/atest/robot/standard_libraries/builtin/converter.robot
index 632e676d759..4911659cec2 100644
--- a/atest/robot/standard_libraries/builtin/converter.robot
+++ b/atest/robot/standard_libraries/builtin/converter.robot
@@ -5,7 +5,7 @@ Resource atest_resource.robot
*** Test Cases ***
Convert To Integer
${tc}= Check Test Case ${TEST NAME}
- Verify argument type message ${tc.kws[0].kws[0].msgs[0]}
+ Verify argument type message ${tc[0, 0, 0]}
Convert To Integer With Base
Check Test Case ${TEST NAME}
@@ -18,19 +18,19 @@ Convert To Integer With Embedded Base
Convert To Binary
${tc}= Check Test Case ${TEST NAME}
- Verify argument type message ${tc.kws[0].kws[0].msgs[0]}
+ Verify argument type message ${tc[0, 0, 0]}
Convert To Octal
${tc}= Check Test Case ${TEST NAME}
- Verify argument type message ${tc.kws[0].kws[0].msgs[0]}
+ Verify argument type message ${tc[0, 0, 0]}
Convert To Hex
${tc}= Check Test Case ${TEST NAME}
- Verify argument type message ${tc.kws[0].kws[0].msgs[0]}
+ Verify argument type message ${tc[0, 0, 0]}
Convert To Number
${tc}= Check Test Case ${TEST NAME}
- Verify argument type message ${tc.kws[0].kws[0].msgs[0]}
+ Verify argument type message ${tc[0, 0, 0]}
Convert To Number With Precision
Check Test Case ${TEST NAME}
@@ -40,11 +40,11 @@ Numeric conversions with long types
Convert To String
${tc}= Check Test Case ${TEST NAME}
- Verify argument type message ${tc.kws[0].msgs[0]}
+ Verify argument type message ${tc[0, 0]}
Convert To Boolean
${tc}= Check Test Case ${TEST NAME}
- Verify argument type message ${tc.kws[0].msgs[0]}
+ Verify argument type message ${tc[0, 0]}
Create List
Check Test Case ${TEST NAME}
diff --git a/atest/robot/standard_libraries/builtin/count.robot b/atest/robot/standard_libraries/builtin/count.robot
index 91acc6c8fe5..43122824093 100644
--- a/atest/robot/standard_libraries/builtin/count.robot
+++ b/atest/robot/standard_libraries/builtin/count.robot
@@ -6,24 +6,24 @@ Resource builtin_resource.robot
Get Count
[Documentation] Tested also by Should Contain X Times keyword that uses this intenally.
${tc} = Check test case ${TESTNAME}
- Check Log Message ${tc.kws[0].kws[0].msgs[0]} Item found from container 2 times.
- Check Log Message ${tc.kws[1].kws[0].msgs[0]} Item found from container 2 times.
- Check Log Message ${tc.kws[2].kws[0].msgs[0]} Item found from container 1 time.
- Check Log Message ${tc.kws[3].kws[0].msgs[0]} Item found from container 1 time.
- Check Log Message ${tc.kws[4].kws[0].msgs[0]} Item found from container 50 times.
- Check Log Message ${tc.kws[5].kws[0].msgs[0]} Item found from container 0 times.
+ Check Log Message ${tc[0, 0, 0]} Item found from container 2 times.
+ Check Log Message ${tc[1, 0, 0]} Item found from container 2 times.
+ Check Log Message ${tc[2, 0, 0]} Item found from container 1 time.
+ Check Log Message ${tc[3, 0, 0]} Item found from container 1 time.
+ Check Log Message ${tc[4, 0, 0]} Item found from container 50 times.
+ Check Log Message ${tc[5, 0, 0]} Item found from container 0 times.
Should Contain X Times with strings
${tc} = Check test case ${TESTNAME}
- Check Log Message ${tc.kws[0].msgs[0]} Item found from container 2 times.
- Check Log Message ${tc.kws[1].msgs[0]} Item found from container 1 time.
- Check Log Message ${tc.kws[3].msgs[0]} Item found from container 0 times.
+ Check Log Message ${tc[0, 0]} Item found from container 2 times.
+ Check Log Message ${tc[1, 0]} Item found from container 1 time.
+ Check Log Message ${tc[3, 0]} Item found from container 0 times.
Should Contain X Times with containers
${tc} = Check test case ${TESTNAME}
- Check Log Message ${tc.kws[0].msgs[0]} Item found from container 1 time.
- Check Log Message ${tc.kws[1].msgs[0]} Item found from container 2 times.
- Check Log Message ${tc.kws[3].msgs[0]} Item found from container 0 times.
+ Check Log Message ${tc[0, 0]} Item found from container 1 time.
+ Check Log Message ${tc[1, 0]} Item found from container 2 times.
+ Check Log Message ${tc[3, 0]} Item found from container 0 times.
Should Contain X Times failing
Check test case ${TESTNAME}
diff --git a/atest/robot/standard_libraries/builtin/evaluate.robot b/atest/robot/standard_libraries/builtin/evaluate.robot
index d43a7f03d7d..02696b60849 100644
--- a/atest/robot/standard_libraries/builtin/evaluate.robot
+++ b/atest/robot/standard_libraries/builtin/evaluate.robot
@@ -9,6 +9,9 @@ Resource atest_resource.robot
Evaluate
Check Test Case ${TESTNAME}
+Custom additions to builtins are supported
+ Check Test Case ${TESTNAME}
+
Modules are imported automatically
Check Test Case ${TESTNAME}
@@ -36,6 +39,9 @@ Explicit modules can override builtins
Explicit modules used in lambda
Check Test Case ${TESTNAME}
+Evaluation namespace is mutable
+ Check Test Case ${TESTNAME}
+
Custom namespace
Check Test Case ${TESTNAME}
@@ -108,5 +114,11 @@ Evaluate Nonstring
Evaluate doesn't see module globals
Check Test Case ${TESTNAME}
+Automatic variables are seen in expression part of comprehensions only with Python 3.12+
+ Check Test Case ${TESTNAME}
+
+Automatic variables are not seen inside lambdas
+ Check Test Case ${TESTNAME}
+
Evaluation errors can be caught
Check Test Case ${TESTNAME}
diff --git a/atest/robot/standard_libraries/builtin/fail.robot b/atest/robot/standard_libraries/builtin/fail.robot
index 5076fb2fe28..c78ede1bbe6 100644
--- a/atest/robot/standard_libraries/builtin/fail.robot
+++ b/atest/robot/standard_libraries/builtin/fail.robot
@@ -5,45 +5,51 @@ Resource atest_resource.robot
*** Test Cases ***
Fail
${tc}= Check Test Tags ${TESTNAME} force1 force2
- Length Should Be ${tc.kws[0].msgs} 1
+ Length Should Be ${tc[0].body} 1
-Fail With Message
+Fail with message
${tc}= Check Test Tags ${TESTNAME} force1 force2
- Length Should Be ${tc.kws[0].msgs} 1
+ Length Should Be ${tc[0].body} 1
+
+Fail with non-string message
+ Check Test Case ${TESTNAME}
+
+Fail with non-true message having non-empty string representation
+ Check Test Case ${TESTNAME}
Set one tag
${tc}= Check Test Tags ${TESTNAME} force1 force2 tag
- Length Should Be ${tc.kws[0].msgs} 2
- Check Log Message ${tc.kws[0].msgs[0]} Set tag 'tag'.
+ Length Should Be ${tc[0].body} 2
+ Check Log Message ${tc[0, 0]} Set tag 'tag'.
Set multiple tags
${tc}= Check Test Tags ${TESTNAME} force1 force2 tag1 tag2
- Length Should Be ${tc.kws[0].msgs} 2
- Check Log Message ${tc.kws[0].msgs[0]} Set tags 'tag1' and 'tag2'.
+ Length Should Be ${tc[0].body} 2
+ Check Log Message ${tc[0, 0]} Set tags 'tag1' and 'tag2'.
Remove one tag
${tc}= Check Test Tags ${TESTNAME} force2
- Length Should Be ${tc.kws[0].msgs} 2
- Check Log Message ${tc.kws[0].msgs[0]} Removed tag 'force1'.
+ Length Should Be ${tc[0].body} 2
+ Check Log Message ${tc[0, 0]} Removed tag 'force1'.
Remove multiple tags
${tc}= Check Test Tags ${TESTNAME}
- Length Should Be ${tc.kws[0].msgs} 2
- Check Log Message ${tc.kws[0].msgs[0]} Removed tags 'force1' and 'force2'.
+ Length Should Be ${tc[0].body} 2
+ Check Log Message ${tc[0, 0]} Removed tags 'force1' and 'force2'.
Remove multiple tags with pattern
${tc}= Check Test Tags ${TESTNAME}
- Length Should Be ${tc.kws[0].msgs} 2
- Check Log Message ${tc.kws[0].msgs[0]} Removed tag 'force?'.
+ Length Should Be ${tc[0].body} 2
+ Check Log Message ${tc[0, 0]} Removed tag 'force?'.
Set and remove tags
${tc}= Check Test Tags ${TESTNAME} force2 tag1 tag2
- Length Should Be ${tc.kws[0].msgs} 3
- Check Log Message ${tc.kws[0].msgs[0]} Removed tags 'force1' and 'nonEx'.
- Check Log Message ${tc.kws[0].msgs[1]} Set tags 'tag1' and 'tag2'.
+ Length Should Be ${tc[0].body} 3
+ Check Log Message ${tc[0, 0]} Removed tags 'force1' and 'nonEx'.
+ Check Log Message ${tc[0, 1]} Set tags 'tag1' and 'tag2'.
Set tags should not be removed
${tc}= Check Test Tags ${TESTNAME} fii foo
- Length Should Be ${tc.kws[0].msgs} 3
- Check Log Message ${tc.kws[0].msgs[0]} Removed tag 'f*'.
- Check Log Message ${tc.kws[0].msgs[1]} Set tags 'foo' and 'fii'.
+ Length Should Be ${tc[0].body} 3
+ Check Log Message ${tc[0, 0]} Removed tag 'f*'.
+ Check Log Message ${tc[0, 1]} Set tags 'foo' and 'fii'.
diff --git a/atest/robot/standard_libraries/builtin/fatal_error.robot b/atest/robot/standard_libraries/builtin/fatal_error.robot
index a840ddb8f20..009dcd947f3 100644
--- a/atest/robot/standard_libraries/builtin/fatal_error.robot
+++ b/atest/robot/standard_libraries/builtin/fatal_error.robot
@@ -10,4 +10,4 @@ Subsequent tests are not executed after `Fatal Error` keyword has been used
Check Test Case ${TESTNAME}
Suite teardown is executed after `Fatal Error` keyword
- Check Log Message ${SUITE.teardown.msgs[0]} AssertionError FAIL
+ Check Log Message ${SUITE.teardown[0]} AssertionError FAIL
diff --git a/atest/robot/standard_libraries/builtin/keyword_should_exist.robot b/atest/robot/standard_libraries/builtin/keyword_should_exist.robot
index 481c5491150..1cd444746e5 100644
--- a/atest/robot/standard_libraries/builtin/keyword_should_exist.robot
+++ b/atest/robot/standard_libraries/builtin/keyword_should_exist.robot
@@ -31,6 +31,9 @@ Keyword does not exist
Keyword does not exist with custom message
Check Test Case ${TESTNAME}
+Recommendations not shown if keyword does not exist
+ Check Test Case ${TESTNAME}
+
Duplicate keywords
Check Test Case ${TESTNAME}
diff --git a/atest/robot/standard_libraries/builtin/length.robot b/atest/robot/standard_libraries/builtin/length.robot
index ce90b08a30c..7825b0c8ff7 100644
--- a/atest/robot/standard_libraries/builtin/length.robot
+++ b/atest/robot/standard_libraries/builtin/length.robot
@@ -5,19 +5,19 @@ Resource builtin_resource.robot
*** Test Cases ***
Get Length
${tc} = Check Test Case ${TESTNAME}
- Check Log Message ${tc.kws[0].kws[0].msgs[0]} Length is 0
- Check Log Message ${tc.kws[1].kws[0].msgs[0]} Length is 1
- Check Log Message ${tc.kws[2].kws[0].msgs[0]} Length is 2
- Check Log Message ${tc.kws[3].kws[0].msgs[0]} Length is 3
- Check Log Message ${tc.kws[4].kws[0].msgs[0]} Length is 11
- Check Log Message ${tc.kws[5].kws[0].msgs[0]} Length is 0
+ Check Log Message ${tc[0, 0, 0]} Length is 0.
+ Check Log Message ${tc[1, 0, 0]} Length is 1.
+ Check Log Message ${tc[2, 0, 0]} Length is 2.
+ Check Log Message ${tc[3, 0, 0]} Length is 3.
+ Check Log Message ${tc[4, 0, 0]} Length is 11.
+ Check Log Message ${tc[5, 0, 0]} Length is 0.
Length Should Be
${tc} = Check Test Case ${TESTNAME}
- Check Log Message ${tc.kws[-1].msgs[0]} Length is 2
- Check Log Message ${tc.kws[-1].msgs[1]} Length of '*' should be 3 but is 2. FAIL pattern=yep
- Check Log Message ${tc.kws[-1].msgs[2]} Traceback* DEBUG pattern=yep
- Length Should Be ${tc.kws[-1].msgs} 3
+ Check Log Message ${tc[-1, 0]} Length is 2.
+ Check Log Message ${tc[-1, 1]} Length of '*' should be 3 but is 2. FAIL pattern=yep
+ Check Log Message ${tc[-1, 2]} Traceback* DEBUG pattern=yep
+ Length Should Be ${tc[-1].body} 3
Length Should Be with custom message
Check Test Case ${TESTNAME}
@@ -26,25 +26,25 @@ Length Should Be with invalid length
Check Test Case ${TESTNAME}
Should Be Empty
- Check test case ${TESTNAME} 1
- Check test case ${TESTNAME} 2
- Check test case ${TESTNAME} 3
+ Check Test Case ${TESTNAME} 1
+ Check Test Case ${TESTNAME} 2
+ Check Test Case ${TESTNAME} 3
Should Be Empty with custom message
- Check test case ${TESTNAME}
+ Check Test Case ${TESTNAME}
Should Not Be Empty
- Check test case ${TESTNAME} 1
- Check test case ${TESTNAME} 2
+ Check Test Case ${TESTNAME} 1
+ Check Test Case ${TESTNAME} 2
Should Not Be Empty with custom message
- Check test case ${TESTNAME}
+ Check Test Case ${TESTNAME}
Getting length with `length` method
- Check test case ${TESTNAME}
+ Check Test Case ${TESTNAME}
Getting length with `size` method
- Check test case ${TESTNAME}
+ Check Test Case ${TESTNAME}
Getting length with `length` attribute
- Check test case ${TESTNAME}
+ Check Test Case ${TESTNAME}
diff --git a/atest/robot/standard_libraries/builtin/log.robot b/atest/robot/standard_libraries/builtin/log.robot
index 10ae859f4f1..d714149f292 100644
--- a/atest/robot/standard_libraries/builtin/log.robot
+++ b/atest/robot/standard_libraries/builtin/log.robot
@@ -8,182 +8,191 @@ ${HTML} Robot Framework
*** Test Cases ***
Log
${tc} = Check Test Case ${TEST NAME}
- Check Log Message ${tc.kws[0].msgs[0]} Hello, world!
- Check Log Message ${tc.kws[1].msgs[0]} 42
- Check Log Message ${tc.kws[2].msgs[0]} None
- Check Log Message ${tc.kws[3].msgs[0]} String presentation of MyObject
+ Check Log Message ${tc[0, 0]} Hello, world!
+ Check Log Message ${tc[1, 0]} 42
+ Check Log Message ${tc[2, 0]} None
+ Check Log Message ${tc[3, 0]} String presentation of MyObject
Log with different levels
${tc} = Check Test Case ${TEST NAME}
- Check Log Message ${tc.kws[0].msgs[1]} Log says: Hello from tests! INFO
- Check Log Message ${tc.kws[1].msgs[1]} Trace level TRACE
- Check Log Message ${tc.kws[2].msgs[1]} Debug level DEBUG
- Check Log Message ${tc.kws[3].msgs[1]} Info level INFO
- Check Log Message ${tc.kws[4].msgs[1]} Warn level WARN
- Check Log Message ${tc.kws[5].msgs[1]} Error level ERROR
- Check Log Message ${ERRORS[0]} Warn level WARN
- Check Log Message ${ERRORS[1]} Error level ERROR
- Length Should Be ${ERRORS} 4 # Two deprecation warnings from `repr`.
+ Check Log Message ${tc[0, 1]} Log says: Hello from tests! INFO
+ Check Log Message ${tc[1, 1]} Trace level TRACE
+ Check Log Message ${tc[2, 1]} Debug level DEBUG
+ Check Log Message ${tc[3, 1]} Info level INFO
+ Check Log Message ${tc[4, 1]} Warn level WARN
+ Check Log Message ${tc[5, 1]} Error level ERROR
+ Check Log Message ${ERRORS[0]} Warn level WARN
+ Check Log Message ${ERRORS[1]} Error level ERROR
+ Length Should Be ${ERRORS} 4 # Two deprecation warnings from `repr`.
Invalid log level failure is catchable
Check Test Case ${TEST NAME}
HTML is escaped by default
${tc} = Check Test Case ${TEST NAME}
- Check Log Message ${tc.kws[0].msgs[0]} not bold
- Check Log Message ${tc.kws[1].msgs[0]} ${HTML}
+ Check Log Message ${tc[0, 0]} not bold
+ Check Log Message ${tc[1, 0]} ${HTML}
HTML pseudo level
${tc} = Check Test Case ${TEST NAME}
- Check Log Message ${tc.kws[0].msgs[0]} bold html=True
- Check Log Message ${tc.kws[1].msgs[0]} ${HTML} html=True
+ Check Log Message ${tc[0, 0]} bold html=True
+ Check Log Message ${tc[1, 0]} ${HTML} html=True
Explicit HTML
${tc} = Check Test Case ${TEST NAME}
- Check Log Message ${tc.kws[0].msgs[0]} bold html=True
- Check Log Message ${tc.kws[1].msgs[0]} ${HTML} DEBUG html=True
- Check Log Message ${tc.kws[2].msgs[0]} ${HTML} DEBUG
+ Check Log Message ${tc[0, 0]} bold html=True
+ Check Log Message ${tc[1, 0]} ${HTML} DEBUG html=True
+ Check Log Message ${tc[2, 0]} ${HTML} DEBUG
FAIL is not valid log level
Check Test Case ${TEST NAME}
Log also to console
${tc} = Check Test Case ${TEST NAME}
- Check Log Message ${tc.kws[0].msgs[0]} Hello, console!
- Check Log Message ${tc.kws[1].msgs[0]} ${HTML} DEBUG html=True
+ Check Log Message ${tc[0, 0]} Hello, console!
+ Check Log Message ${tc[1, 0]} ${HTML} DEBUG html=True
Stdout Should Contain Hello, console!\n
Stdout Should Contain ${HTML}\n
+CONSOLE pseudo level
+ ${tc} = Check Test Case ${TEST NAME}
+ Check Log Message ${tc[0, 0]} Hello, info and console!
+ Stdout Should Contain Hello, info and console!\n
+
repr=True
${tc} = Check Test Case ${TEST NAME}
- Check Log Message ${tc.kws[0].msgs[0]} The 'repr' argument of 'BuiltIn.Log' is deprecated. Use 'formatter=repr' instead. WARN
- Check Log Message ${tc.kws[0].msgs[1]} Nothing special here
- Check Log Message ${tc.kws[1].msgs[0]} The 'repr' argument of 'BuiltIn.Log' is deprecated. Use 'formatter=repr' instead. WARN
- Check Log Message ${tc.kws[1].msgs[1]} 'Hyvää yötä ☃!'
+ Check Log Message ${tc[0, 0]} The 'repr' argument of 'BuiltIn.Log' is deprecated. Use 'formatter=repr' instead. WARN
+ Check Log Message ${tc[0, 1]} Nothing special here
+ Check Log Message ${tc[1, 0]} The 'repr' argument of 'BuiltIn.Log' is deprecated. Use 'formatter=repr' instead. WARN
+ Check Log Message ${tc[1, 1]} 'Hyvää yötä ☃!'
formatter=repr
${tc} = Check Test Case ${TEST NAME}
- Check Log Message ${tc.kws[0].msgs[0]} 'Nothing special here'
- Check Log Message ${tc.kws[1].msgs[0]} 'Hyvää yötä ☃!'
- Check Log Message ${tc.kws[2].msgs[0]} 42 DEBUG
- Check Log Message ${tc.kws[4].msgs[0]} b'\\x00abc\\xff (formatter=repr)'
- Check Log Message ${tc.kws[6].msgs[0]} 'hyvä'
+ Check Log Message ${tc[0, 0]} 'Nothing special here'
+ Check Log Message ${tc[1, 0]} 'Hyvää yötä ☃!'
+ Check Log Message ${tc[2, 0]} 42 DEBUG
+ Check Log Message ${tc[4, 0]} b'\\x00abc\\xff (formatter=repr)'
+ Check Log Message ${tc[6, 0]} 'hyvä'
Stdout Should Contain b'\\x00abc\\xff (formatter=repr)'
formatter=ascii
${tc} = Check Test Case ${TEST NAME}
- Check Log Message ${tc.kws[0].msgs[0]} 'Nothing special here'
- Check Log Message ${tc.kws[1].msgs[0]} 'Hyv\\xe4\\xe4 y\\xf6t\\xe4 \\u2603!'
- Check Log Message ${tc.kws[2].msgs[0]} 42 DEBUG
- Check Log Message ${tc.kws[4].msgs[0]} b'\\x00abc\\xff (formatter=ascii)'
- Check Log Message ${tc.kws[6].msgs[0]} 'hyva\\u0308'
+ Check Log Message ${tc[0, 0]} 'Nothing special here'
+ Check Log Message ${tc[1, 0]} 'Hyv\\xe4\\xe4 y\\xf6t\\xe4 \\u2603!'
+ Check Log Message ${tc[2, 0]} 42 DEBUG
+ Check Log Message ${tc[4, 0]} b'\\x00abc\\xff (formatter=ascii)'
+ Check Log Message ${tc[6, 0]} 'hyva\\u0308'
Stdout Should Contain b'\\x00abc\\xff (formatter=ascii)'
formatter=str
${tc} = Check Test Case ${TEST NAME}
- Check Log Message ${tc.kws[0].msgs[0]} Nothing special here
- Check Log Message ${tc.kws[1].msgs[0]} Hyvää yötä ☃!
- Check Log Message ${tc.kws[2].msgs[0]} 42 DEBUG
- Check Log Message ${tc.kws[4].msgs[0]} abc\\xff (formatter=str)
- Check Log Message ${tc.kws[6].msgs[0]} hyvä
- Stdout Should Contain abc\\xff (formatter=str)
+ Check Log Message ${tc[0, 0]} Nothing special here
+ Check Log Message ${tc[1, 0]} Hyvää yötä ☃!
+ Check Log Message ${tc[2, 0]} 42 DEBUG
+ Check Log Message ${tc[4, 0]} abc\xff (formatter=str)
+ Check Log Message ${tc[6, 0]} hyvä
+ Stdout Should Contain abc\xff (formatter=str)
formatter=repr pretty prints
${tc} = Check Test Case ${TEST NAME}
${long string} = Evaluate ' '.join(['Robot Framework'] * 1000)
- ${small dict} = Set Variable {3: b'items', 'a': 'sorted', 'small': 'dict'}
+ ${small dict} = Set Variable {'small': 'dict', 3: b'items', 'NOT': 'sorted'}
${small list} = Set Variable ['small', b'list', 'not sorted', 4]
- Check Log Message ${tc.kws[1].msgs[0]} '${long string}'
- Check Log Message ${tc.kws[3].msgs[0]} ${small dict}
- Check Log Message ${tc.kws[5].msgs[0]} {'big': 'dict',\n\ 'list': [1, 2, 3],\n\ 'long': '${long string}',\n\ 'nested': ${small dict}}
- Check Log Message ${tc.kws[7].msgs[0]} ${small list}
- Check Log Message ${tc.kws[9].msgs[0]} ['big',\n\ 'list',\n\ '${long string}',\n\ b'${long string}',\n\ ['nested', ('tuple', 2)],\n\ ${small dict}]
- Check Log Message ${tc.kws[11].msgs[0]} ['hyvä', b'hyv\\xe4', {'☃': b'\\x00\\xff'}]
+ Check Log Message ${tc[1, 0]} '${long string}'
+ Check Log Message ${tc[3, 0]} ${small dict}
+ Check Log Message ${tc[5, 0]} {'big': 'dict',\n 'long': '${long string}',\n 'nested': ${small dict},\n 'list': [1, 2, 3],\n 'sorted': False}
+ Check Log Message ${tc[7, 0]} ${small list}
+ Check Log Message ${tc[9, 0]} ['big',\n 'list',\n '${long string}',\n b'${long string}',\n ['nested', ('tuple', 2)],\n ${small dict}]
+ Check Log Message ${tc[11, 0]} ['hyvä', b'hyv\\xe4', {'☃': b'\\x00\\xff'}]
Stdout Should Contain ${small dict}
Stdout Should Contain ${small list}
formatter=len
${tc} = Check Test Case ${TEST NAME}
- Check Log Message ${tc.kws[0].msgs[0]} 20
- Check Log Message ${tc.kws[1].msgs[0]} 13 DEBUG
- Check Log Message ${tc.kws[3].msgs[0]} 21
- Check Log Message ${tc.kws[5].msgs[0]} 5
+ Check Log Message ${tc[0, 0]} 20
+ Check Log Message ${tc[1, 0]} 13 DEBUG
+ Check Log Message ${tc[3, 0]} 21
+ Check Log Message ${tc[5, 0]} 5
formatter=type
${tc} = Check Test Case ${TEST NAME}
- Check Log Message ${tc.kws[0].msgs[0]} str
- Check Log Message ${tc.kws[1].msgs[0]} str
- Check Log Message ${tc.kws[2].msgs[0]} int DEBUG
- Check Log Message ${tc.kws[4].msgs[0]} bytes
- Check Log Message ${tc.kws[6].msgs[0]} datetime
+ Check Log Message ${tc[0, 0]} str
+ Check Log Message ${tc[1, 0]} str
+ Check Log Message ${tc[2, 0]} int DEBUG
+ Check Log Message ${tc[4, 0]} bytes
+ Check Log Message ${tc[6, 0]} datetime
formatter=invalid
Check Test Case ${TEST NAME}
Log callable
${tc} = Check Test Case ${TEST NAME}
- Check Log Message ${tc.kws[0].msgs[0]} *objects_for_call_method.MyObject* pattern=yes
- Check Log Message ${tc.kws[2].msgs[0]}
${FLATTENED}
+ Should Be Equal ${TC[6].doc} Doc of flat keyword.
+ Check Counts ${TC[6]} 1
Match full name
- Should Be Equal ${TC.kws[3].doc} Logs the given message with the given level.\n\n${FLAT TEXT}
- Length Should Be ${TC.kws[3].kws} 0
- Length Should Be ${TC.kws[3].msgs} 1
- Check Log Message ${TC.kws[3].msgs[0]} Flatten me too!!
+ Should Be Equal ${TC[3].message} *HTML* ${FLATTENED}
+ Should Be Equal ${TC[3].doc} Logs the given message with the given level.
+ Check Counts ${TC[3]} 1
+ Check Log Message ${TC[3, 0]} Flatten me too!!
Flattened in log after execution
- Should Contain X Times ${LOG} Doc of keyword 3 1
- Should Contain X Times ${LOG} Doc of keyword 2 1
- Should Contain X Times ${LOG} Doc of keyword 1 1
- Should Contain X Times ${LOG} ${FLAT HTML} 6
- Should Contain ${LOG} *
Original message:\nNot <b>HTML</b> fail
+ ... Pass=PASS:${prefix}\nTest skipped using 'NOT skip' tag pattern.
+ ... Fail=FAIL:${prefix}\nTest skipped using 'NOT skip' tag pattern.
Original message:\nNot <b>HTML</b> fail
... Skip=SKIP:${prefix}\nHTML skip
Original message:\nHTML skip
*** Keywords ***
Run original tests
- Create Output With Robot ${ORIGINAL} --variable FAIL:YES --variable LEVEL:WARN ${SUITES}
+ ${options} = Catenate
+ ... --variable FAIL:YES
+ ... --variable LEVEL:WARN
+ ... --doc "Doc for original run"
+ ... --metadata Original:True
+ Create Output With Robot ${ORIGINAL} ${options} ${SUITES}
Verify original tests
+ VAR ${ORIGINAL ELAPSED} ${SUITE.elapsed_time} scope=SUITE
Verify original tests
Should Be Equal ${SUITE.name} Suites
@@ -98,12 +112,15 @@ Verify original tests
Re-run tests
[Arguments] ${options}=
${options} = Catenate
+ ... --doc "Doc for re-run"
+ ... --metadata ReRun:True
... --variable SUITE_SETUP:NoOperation # Affects misc/suites/__init__.robot
... --variable SUITE_TEARDOWN:NONE # -- ;; --
... --variable SETUP_MSG:Rerun! # Affects misc/suites/fourth.robot
... --variable TEARDOWN_MSG:New! # -- ;; --
... --variable SETUP:NONE # Affects misc/suites/subsuites/sub1.robot
... --variable TEARDOWN:NONE # -- ;; --
+ ... --variable SLEEP:0.5 # -- ;; --
... --rerunfailed ${ORIGINAL} ${options}
Create Output With Robot ${MERGE 1} ${options} ${SUITES}
Should Be Equal ${SUITE.name} Suites
@@ -164,15 +181,20 @@ Test merge should have been successful
... ${SUITE.suites[7]}
Suite setup and teardown should have been merged
- Should Be Equal ${SUITE.setup.name} BuiltIn.No Operation
+ Should Be Equal ${SUITE.setup.full_name} BuiltIn.No Operation
Should Be Equal ${SUITE.teardown.name} ${NONE}
Should Be Equal ${SUITE.suites[1].name} Fourth
- Check Log Message ${SUITE.suites[1].setup.msgs[0]} Rerun!
- Check Log Message ${SUITE.suites[1].teardown.msgs[0]} New!
+ Check Log Message ${SUITE.suites[1].setup[0]} Rerun!
+ Check Log Message ${SUITE.suites[1].teardown[0]} New!
Should Be Equal ${SUITE.suites[2].suites[0].name} Sub1
Should Be Equal ${SUITE.suites[2].suites[0].setup.name} ${NONE}
Should Be Equal ${SUITE.suites[2].suites[0].teardown.name} ${NONE}
+Suite documentation and metadata should have been merged
+ Should Be Equal ${SUITE.doc} Doc for re-run
+ Should Be Equal ${SUITE.metadata}[ReRun] True
+ Should Be Equal ${SUITE.metadata}[Original] True
+
Test add should have been successful
Should Be Equal ${SUITE.name} Suites
Should Contain Suites ${SUITE} @{ALL SUITES}
@@ -227,7 +249,7 @@ Warnings should have been merged
Check Log Message ${ERRORS[0]} Original message WARN
Check Log Message ${ERRORS[1]} Override WARN
${tc} = Check Test Case SubSuite1 First
- Check Log Message ${tc.kws[0].msgs[0]} Override WARN
+ Check Log Message ${tc[0, 0]} Override WARN
Merge should have failed
Stderr Should Be Equal To
@@ -237,15 +259,15 @@ Merge should have failed
Timestamps should be cleared
[Arguments] @{suites}
FOR ${suite} IN @{suites}
- Should Be Equal ${suite.starttime} ${None}
- Should Be Equal ${suite.endtime} ${None}
+ Should Be Equal ${suite.start_time} ${None}
+ Should Be Equal ${suite.end_time} ${None}
END
Timestamps should be set
[Arguments] @{suites}
FOR ${suite} IN @{suites}
- Timestamp Should Be Valid ${suite.starttime}
- Timestamp Should Be Valid ${suite.endtime}
+ Timestamp Should Be Valid ${suite.start_time}
+ Timestamp Should Be Valid ${suite.end_time}
END
Create expected merge message header
@@ -300,7 +322,6 @@ Create expected multi-merge message
... ${message 1}
...
${message 2}
-Log should have been created with all Log keywords flattened
+Log should have been created with Fail keywords flattened
${log} = Get File ${OUTDIR}/log.html
- Should Not Contain ${log} "*