Skip to content
This repository has been archived by the owner on Dec 8, 2022. It is now read-only.

Add CLI input mode to test runner #2955

Merged
merged 15 commits into from
Jan 27, 2021

Conversation

aggarw13
Copy link
Contributor

@aggarw13 aggarw13 commented Jan 26, 2021

This PR adds a another solution ( separate from the configurable delay config added in PR #2950 ) for the problem of race condition between AWS IoT Device Tester (that runs on host machine) getting serial console access to device and the test runner executing tests on the device.

This PR adds support to the test runner to wait for accepting a "start" command from the serial console (based on FreeRTOS+CLI library) before beginning execution of tests. This can allow the AWS IoT Device Tester to ensure that the device is serially accessible before tests begin executing on the device.
This feature can be enabled with the AWS_TEST_RUNNER_ENABLE_CLI_INPUT config in the aws_test_runner_config.h file.

Here is the log that shows interacting with the test runner when the feature is enable. Note that after the start command is provided on the serial prompt, the tests starts executing.

Welcome to the Test Runner console.
[Type the "start" command to beging executing tests]
>help

help:
 Lists all the registered commands

start : Instructs test runner to start executing tests on the board.
[Press Enter to run the previous command]
>start
---------STARTING TESTS---------
TEST(Full_PKCS11_StartFinish, AFQP_StartFinish_FirstTest) PASS
TEST(Full_PKCS11_StartFinish, AFQP_GetFunctionList) PASS

Note: This feature uses FreeRTOS+CLI library along with Common I/O interface for UART communication. Thus, this feature is only supported by boards that provide ports for Common I/O interface.

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

@aggarw13 aggarw13 requested a review from a team as a code owner January 26, 2021 23:49
size_t commandBufferLength,
char * pOutputBuffer,
size_t outpuBufferLength )
void FreeRTOS_CLI_ProcessInputBuffer( xConsoleIO_t consoleIO,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This duplicates some code in FreeRTOS_CLI_ProcessInputBuffer to handle the exiting the CLI loop. Can we keep code same and instead add a new command such as exit for the loop to exit gracefully.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We discussed offline and decided that we can add a exit command support to the FreeRTOS_CLIEnterConsoleLoop function to allow the console loop to be terminated in a separate PR.

UNITY_BEGIN();
UNITY_BEGIN();

RunTests();
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It seems the memory leak test wont start before the test begin
testrunnerFULL_MEMORYLEAK_ENABLED

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed by moving all the test setup, execution of tests and teardown logic in common function.


UNITY_BEGIN();

RunTests();
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Needs to handle UNITY_END() ?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Have moved all the test logic (of setup, executing tests and tear down) to common function, which is called from the handler now.


/* Set flag to indicate reception of command from serial console to start */
/* executing tests. */
xReceivedCommand = pdTRUE;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

An "OK" response back to the peer here will help in debugging a broken CLI connection to the actual test issues.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added the response to acknowledge processing of command.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Per offline discussion, removed the response for now as the response would be visible only at the end of the test execution. We can consider adding the response string later if there are architectural changes of running the CLI console in a different task than the test runner.

@aggarw13 aggarw13 merged commit 441e02e into aws:master Jan 27, 2021
@aggarw13 aggarw13 deleted the poc/support-prompt-input-for-test-runner branch January 27, 2021 23:45
ifyall pushed a commit to ifyall/amazon-freertos that referenced this pull request May 19, 2021
mingyue86010 pushed a commit that referenced this pull request May 26, 2021
cobusve pushed a commit that referenced this pull request May 27, 2021
* Add cypress internal tests and 062 boards from cypress-202007.00

* Add 062 projects

* Sync aws demo/test from 064 board to 062 boards. 064 got its changes from Amazon in #2544.

* Sync aws demo/test from 064 board to 062 boards. 064 got its changes from Amazon in #2544.

* Update 062 boards for PKCS11 changes from Amazon master. Update demo defaults for 062 boards from Amazon master. Regenerate GeneratedSource.

* Regenerate GeneratedSource, remove iot_pkcs11_pal.c to align with Amazon PKCS11 implementation from master

* Update Cy8CKIT-062-WIFI-BT with regenerated GeneratedSource and removed iot_pkcs11_pal.c to align with Amazon master updated to PKCS11.

* Align aws_test config files to new Amazon master templates. Regenerate 062-WIFI-BT GeneratedSource with tools_2.2

* Regeneate 062S2-43012 aws_tests GeneratedSource with tools_2.2

* Regenerate 062S2S0-4343W aws_test GeneratedSource with tools_2.2

* Regenerate CY8CPROTO-062-4343W aws_test GeneratedSource with tools_2.2

* Updated IDT to 4.0.1

* Copy .gitlab-ci.yml from cypress-202007.00 branch

* Align iot_config.h to set BLE/WIFI_SUPPORTED as on master. Add definitions for configCOMMAND_INT_MAX_OUTPUT_SIZE and configINCLUDE_DEMO_DEBUG_STATS to FreeRTOSConfig.h as on master

* Add script for cleaning up old IDT OTA buckets

* Align use of BLE_SUPPORT to BLE_SUPPORTED for consistency across CMake vs Make

* Align test scripts to use BLE_SUPPORTED

* Add Celluar definition to device.json for IDT tests as required

* Move definition of BLE_SUPPORTED DEFINE from project Makefiles to the common mtb_global_settings.mk. Aligned aws_tests main.c to use #if BLE_SUPPORTED as seen in the aws_demos main.c

* Add keys from cypress-2020007.00 branch back, as they were removed by Amazon on master.

* Update userdata.json for IDT to 4.0 standard

* Minor fixes to aws-tests to compile successfully

* Update yml file to version IDT version 4.0.1

* Allow IAR build/compile/funcation CI to fail due to SWINTEGRATION-341 until a fix/workaround is available.

* Remove tinycrypt from CMake configuration and re-enable IAR-based tests. Fixes SWINTEGRATION-341.

* CY8CKIT-064S0S2-4343W: Fix aws_demo

aws_demo is no longer connecting to AP. It looks like a merge error
resulted in the removal of the needed function call.

Signed-off-by: Raymond Ngun <raymond.ngun@infineon.com>

* Sync 062 main.c from 064 main.c

* Update policy for swap+copy

Signed-off-by: Raymond Ngun <raymond.ngun@infineon.com>

* Sync all boards to 064

Signed-off-by: Raymond Ngun <raymond.ngun@infineon.com>

* Revert "Add CLI input mode to test runner (#2955)"

This reverts commit 441e02e.

* Update cysecuretools to use 3.1.0

Signed-off-by: Raymond Ngun <raymond.ngun@infineon.com>

* Update TFM policies

Policies are out of date and also needs adjustments for swap support.

Signed-off-by: Raymond Ngun <raymond.ngun@infineon.com>

* Fix OTA_SUPPORT build for aws_tests

Signed-off-by: Raymond Ngun <raymond.ngun@infineon.com>

* SWAP_WITH_STATUS bootloader support

SWINTEGRATION-270

* Sync boards to 064

* fix to remove configASSERT() in Release build

* Add PSOC6 Component to fix WDT enable problem

* Add watchdog timer to 064S0

Create a thread to start and kick WDT before prvWifiConnect()
Return WiFI status from prvWifiConnect()
Stop after prvWifiConnect() returns.
If we fail to connect in prvWifiConnect(), reboot

* Update SoftAP creation sequence for aws demos

* MIDDLEWARE-3942

Update AFR OTA docs

* Update OTA block size and OTA thread priority

- 4k block size speeds up HTTP
- OTA agent can run at lowest priority

Signed-off-by: Raymond Ngun <raymond.ngun@infineon.com>

* Sync OTA changes from 064 to other boards

Signed-off-by: Raymond Ngun <raymond.ngun@infineon.com>

* Revert "fix to remove configASSERT() in Release build"

This reverts commit 5c7c72a.
This changed caused FullWiFi and FullSecureSockets IDT tests
to fail.

Signed-off-by: Raymond Ngun <raymond.ngun@infineon.com>

* fix for SWINTEGRATION-386, use -j4 in make

* Sync 064 main.c to 062 boards.

* Correct version control conflict marker

* Add subscrition container to afr source.

* Add missing MQTT source files to Make build system

* More missing MQTT files from Make build system

* Remove keys for upstreaming to AWS github.

* Remove 062 boards for upstream to AWS GitHub

* Remove internal Cypress testing files.

* Checkout the files from main branch which should not be changed by this PR.

Co-authored-by: Fyall Ian (CYSC CSS ICW SW T 1) <Ian.Fyall@infineon.com>
Co-authored-by: midd <midd@cypress.com>
Co-authored-by: Raymond Ngun <raymond.ngun@cypress.com>
Co-authored-by: RaymondNgun <raymond.ngun@infineon.com>
Co-authored-by: stde <SteveO.DeFrisco>
Co-authored-by: Chandran Praveen Babu (CYSC CSS ICW SW PSW) <PraveenBabu.Chandran@infineon.com>
Co-authored-by: Bill Waters <bill.waters@infineon.com>
Co-authored-by: Ming Yue <mingyue86010@gmail.com>
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants