RTOS threads test: Handle out of memory cases #7744

davidsaada · 2018-08-09T14:37:16Z

Description

The RTOS threads test crashes if it fails to allocate memory on low memory boards. Fix this problem.
Resolves #7535

Pull request type

[x] Fix
[ ] Refactor
[ ] Target update
[ ] Feature
[ ] Breaking change

cmonr · 2018-08-14T01:53:42Z

@c1728p9 Mind taking a look before we start CI?

c1728p9

It seems a bit concerning to skip tests because the device doesn't have enough memory to run them. If the TMPM3H6 doesn't have enough memory to run tests should it be enabled for mbed 5? What is the impact of leaving this target out?

c1728p9 · 2018-08-15T20:20:55Z

TESTS/mbedmicro-rtos-mbed/threads/main.cpp

+
+ // Don't fail test due to lack of memory. Call function directly instead
+ if (!child || !dummy) {
+ increment(counter);


If this isn't actually getting called on a new thread doesn't this defeat the purpose of this test?

This continues our discussion from below, under the assumption we want to allow low memory boards to still pass the test. If we don't have enough memory to launch a new thread, but still need this test to pass under these conditions (which is what this condition checks), then the only way to make this test pass is to manually call the increment function.

aashishc1988

@davidsaada Can you please take a look at my comment?

aashishc1988 · 2018-08-15T22:03:34Z

TESTS/mbedmicro-rtos-mbed/threads/main.cpp

@@ -174,6 +196,10 @@ void test_single_thread()
 template <int N, void (*F)(counter_t *)>
 void test_parallel_threads()
 {
+ char *dummy = new (std::nothrow) char[PARALLEL_THREAD_STACK_SIZE * N];
+ delete[] dummy;
+ TEST_SKIP_UNLESS_MESSAGE(dummy, "Not enough memory to run test");


@davidsaada This doesnt seem right. We are trying to allocate the space and then deleting it right away, and then comparing it using the macro. It will always skip the test. Unless I am missing something here. Can you please check?

The delete[] command has no effect on the value of dummy variable. The test will be skipped only if memory allocation fails. I could switch these two lines, but it won't change anything.

davidsaada · 2018-08-16T09:24:10Z

It seems a bit concerning to skip tests because the device doesn't have enough memory to run them. If the TMPM3H6 doesn't have enough memory to run tests should it be enabled for mbed 5? What is the impact of leaving this target out?

I have no problem leaving this (or any other low memory) target out of our CI or from mbed-os. This PR was based on the assumption that these boards were included in our CI. Under this assumption, one would need these tests to pass. Diluting the tests to reduce memory consumption is a lose-lose attitude, as it harms the test quality on the one hand, and may not be sufficient for even lower memory boards when these are added. Now, we could have this board specifically black listed in the test code (e.g. have an #ifdef excluding specific targets from the test). These exclusions are pretty ugly IMO and shouldn't be part of any code. What I tried to accomplish here was a "dynamic black list" - exclude boards that haven't got enough memory by simply testing whether memory allocation succeeds. This prevents the need for explicit black or white lists in the code. But again - if one decides to exclude this board (or other boards) from CI, I'm fine with it, other than the fact that it's out of this PR's scope.

davidsaada · 2018-08-16T09:47:34Z

@OPpuolitaival @bulislaw

OPpuolitaival · 2018-08-16T12:31:35Z

This looks good for me. It is better to run tests in boards which have enough memory than leave out the test. We will run more boards in continuous integration in future.

c1728p9 · 2018-08-17T14:59:09Z

Hi @davidsaada, I agree, something like a black list is preferable to modifying each test case. Even that is not ideal though.

I would like to see some minimum requirements for Mbed 5 devices and then require that tests fit in that. Otherwise there will be pressure to keep shrinking/bypassing tests to fit on even smaller devices. I proposed a minimum requirement of 2K for heap and 2K for data in #5285 but even that caused too many devices to fail.

I'm not sure how usable these smaller devices are with Mbed 5 if they can't pass our already space optimized tests. It seems wrong to allow a device which doesn't have enough space to create a second thread (with an already minimized stack size) to be Mbed 5 enabled.

cmonr · 2018-08-18T01:16:36Z

@c1728p9 It sounds like you're amicable to this PR for now. Would you mind OKing or dismissing your review?
@aashishc1988 Any thoughts?

@c1728p9 @aashishc1988 @davidsaada @OPpuolitaival Does it make sense to create a new issue/internal ticket to track down and enable minimum testing requirements?

@davidsaada

I'm leaving how to proceed with this PR up to @davidsaada

davidsaada · 2018-08-19T07:52:46Z

Does it make sense to create a new issue/internal ticket to track down and enable minimum testing requirements?

Sure does. We also had some internal discussions about it (following this PR and ones alike). Guess a new issue may be a good place to discuss it.

cmonr · 2018-08-21T14:03:16Z

@davidsaada @c1728p9 Could one of y'all open an issue (if one isn't already), and link it here? In the meanwhile, we'll get this PR on its way.

/morph build

davidsaada · 2018-08-21T16:12:11Z

Opened #7845 to follow up on our discussion regarding low resource boards.

mbed-ci · 2018-08-21T16:14:57Z

Build : SUCCESS

Build number : 2853
Build artifacts/logs : http://mbed-os.s3-website-eu-west-1.amazonaws.com/?prefix=builds/7744/

Triggering tests

/morph test
/morph uvisor-test
/morph export-build
/morph mbed2-build

mbed-ci · 2018-08-21T18:05:28Z

Exporter Build : SUCCESS

Build number : 2484
Build artifacts/logs : http://mbed-os.s3-website-eu-west-1.amazonaws.com/?prefix=builds/exporter/7744/

mbed-ci · 2018-08-22T11:39:48Z

Test : FAILURE

Build number : 2613
Test logs :http://mbed-os-logs.s3-website-us-west-1.amazonaws.com/?prefix=logs/7744/2613

davidsaada · 2018-08-22T11:45:29Z

Test needs to be rerun. Failure is on a single board (NRF51_DK) on two other tests.

0xc0170 · 2018-08-22T11:46:00Z

/morph test

cmonr · 2018-08-22T15:40:40Z

/morph test

mbed-ci · 2018-08-23T10:03:59Z

Test : SUCCESS

Build number : 2625
Test logs :http://mbed-os-logs.s3-website-us-west-1.amazonaws.com/?prefix=logs/7744/2625

0xc0170 · 2018-08-23T11:17:36Z

I would like to see some minimum requirements for Mbed 5 devices and then require that tests fit in that. Otherwise there will be pressure to keep shrinking/bypassing tests to fit on even smaller devices. I proposed a minimum requirement of 2K for heap and 2K for data in #5285 but even that caused too many devices to fail.

I'm not sure how usable these smaller devices are with Mbed 5 if they can't pass our already space optimized tests. It seems wrong to allow a device which doesn't have enough space to create a second thread (with an already minimized stack size) to be Mbed 5 enabled.

👍 Would be useful to have the minimum req.

@c1728p9 You happy with this as it is ?

cmonr · 2018-08-23T15:08:32Z

Opened #7845 to follow up on our discussion regarding low resource boards.

Going to merge since discussion will be moved to referenced issue.

RTOS threads test: Handle out of memory cases

75bc3d3

davidsaada mentioned this pull request Aug 9, 2018

test: some cases are failing with GCC_ARM for Toshiba's TMPM3H6 #7535

Closed

cmonr requested review from 0xc0170 and c1728p9 August 9, 2018 16:23

cmonr added the needs: review label Aug 9, 2018

cmonr approved these changes Aug 14, 2018

View reviewed changes

c1728p9 previously requested changes Aug 15, 2018

View reviewed changes

aashishc1988 suggested changes Aug 15, 2018

View reviewed changes

cmonr mentioned this pull request Aug 16, 2018

When stack stats enabled, prevent exceptions if memory allocations fail #7730

Merged

aashishc1988 approved these changes Aug 20, 2018

View reviewed changes

cmonr added needs: CI and removed needs: review labels Aug 21, 2018

0xc0170 requested a review from bulislaw August 22, 2018 09:13

cmonr added ready for merge release-version: 5.9.6 and removed needs: CI labels Aug 23, 2018

cmonr merged commit e530939 into ARMmbed:master Aug 23, 2018

0xc0170 removed the ready for merge label Aug 23, 2018

davidsaada deleted the david_threads_test_fix branch August 23, 2018 15:10

jeromecoutant mentioned this pull request Oct 19, 2018

Component SD test: skip multiple thread test depending on RAM #8484

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

RTOS threads test: Handle out of memory cases #7744

RTOS threads test: Handle out of memory cases #7744

davidsaada commented Aug 9, 2018

cmonr commented Aug 14, 2018

c1728p9 left a comment

c1728p9 Aug 15, 2018

davidsaada Aug 16, 2018

aashishc1988 left a comment

aashishc1988 Aug 15, 2018

davidsaada Aug 16, 2018

davidsaada commented Aug 16, 2018 •

edited

Loading

davidsaada commented Aug 16, 2018

OPpuolitaival commented Aug 16, 2018

c1728p9 commented Aug 17, 2018

cmonr commented Aug 18, 2018

davidsaada commented Aug 19, 2018

cmonr commented Aug 21, 2018

davidsaada commented Aug 21, 2018

mbed-ci commented Aug 21, 2018

mbed-ci commented Aug 21, 2018

mbed-ci commented Aug 22, 2018

davidsaada commented Aug 22, 2018

0xc0170 commented Aug 22, 2018

cmonr commented Aug 22, 2018

mbed-ci commented Aug 23, 2018

0xc0170 commented Aug 23, 2018

cmonr commented Aug 23, 2018

RTOS threads test: Handle out of memory cases #7744

RTOS threads test: Handle out of memory cases #7744

Conversation

davidsaada commented Aug 9, 2018

Description

Pull request type

cmonr commented Aug 14, 2018

c1728p9 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

aashishc1988 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

davidsaada commented Aug 16, 2018 • edited Loading

davidsaada commented Aug 16, 2018

OPpuolitaival commented Aug 16, 2018

c1728p9 commented Aug 17, 2018

cmonr commented Aug 18, 2018

davidsaada commented Aug 19, 2018

cmonr commented Aug 21, 2018

davidsaada commented Aug 21, 2018

mbed-ci commented Aug 21, 2018

Triggering tests

mbed-ci commented Aug 21, 2018

mbed-ci commented Aug 22, 2018

davidsaada commented Aug 22, 2018

0xc0170 commented Aug 22, 2018

cmonr commented Aug 22, 2018

mbed-ci commented Aug 23, 2018

0xc0170 commented Aug 23, 2018

cmonr commented Aug 23, 2018

davidsaada commented Aug 16, 2018 •

edited

Loading