Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmake: allow running the tests in parallel like in the Makefile #102

Closed
isaac-io opened this issue Aug 2, 2022 · 2 comments · Fixed by #103
Closed

cmake: allow running the tests in parallel like in the Makefile #102

isaac-io opened this issue Aug 2, 2022 · 2 comments · Fixed by #103
Assignees
Labels
build Build related enhancement New feature or request Upstreamable can be upstreamed to RocksDB
Milestone

Comments

@isaac-io
Copy link
Contributor

isaac-io commented Aug 2, 2022

The Makefile supports running tests in parallel using GNU Parallel. CMake runs the tests through ctest, and is also supposed to run on Windows, so GNU Parallel can't be used.

However, CTest natively supports running tests in parallel with the appropriate environment variable set (CTEST_PARALLEL_LEVEL). The problem is that users don't usually have it set, and they might also be used to the J environment variable as that's what the Makefile uses.

Add a CMake script in order to be able to set the CTEST_PARALLEL_LEVEL (if not set by the user) according to the J environment variable if it was set, or fall back to the CPU core count, as the Makefile does.

While at it, allow speeding up the tests on Linux by setting TEST_TMPDIR to /dev/shm (if exists) according to the Makefile logic (only when the sticky bit is set), otherwise falling back on /tmp if exists.

@isaac-io isaac-io added enhancement New feature or request build Build related labels Aug 2, 2022
@isaac-io isaac-io self-assigned this Aug 2, 2022
isaac-io added a commit that referenced this issue Aug 2, 2022
Allow running tests in parallel by default by setting the CTEST_PARALLEL_LEVEL
environment variable. The functionality is added using a CMake script in
order to determine options when running the tests (rather than when configuring
CMake) in compatibility with the Makefile (in particular -- the `J` and
`TEST_TMPDIR` environment variables).

Test Plan: run the check target using different generators (ninja, make)
to see the improvement in test runtime.
isaac-io added a commit that referenced this issue Aug 2, 2022
Allow running tests in parallel by default by setting the CTEST_PARALLEL_LEVEL
environment variable. The functionality is added using a CMake script in
order to determine options when running the tests (rather than when configuring
CMake) in compatibility with the Makefile (in particular -- the `J` and
`TEST_TMPDIR` environment variables).

Test Plan: run the check target using different generators (ninja, make)
to see the improvement in test runtime.
isaac-io added a commit that referenced this issue Aug 2, 2022
Allow running tests in parallel by default by setting the CTEST_PARALLEL_LEVEL
environment variable. The functionality is added using a CMake script in
order to determine options when running the tests (rather than when configuring
CMake) in compatibility with the Makefile (in particular -- the `J` and
`TEST_TMPDIR` environment variables).

Test Plan: run the check target using different generators (ninja, make)
to see the improvement in test runtime.
isaac-io added a commit that referenced this issue Aug 2, 2022
Allow running tests in parallel by default by setting the CTEST_PARALLEL_LEVEL
environment variable. The functionality is added using a CMake script in
order to determine options when running the tests (rather than when configuring
CMake) in compatibility with the Makefile (in particular -- the `J` and
`TEST_TMPDIR` environment variables).

Test Plan: run the check target using different generators (ninja, make)
to see the improvement in test runtime.
isaac-io added a commit that referenced this issue Aug 2, 2022
Allow running tests in parallel by default by setting the CTEST_PARALLEL_LEVEL
environment variable. The functionality is added using a CMake script in
order to determine options when running the tests (rather than when configuring
CMake) in compatibility with the Makefile (in particular -- the `J` and
`TEST_TMPDIR` environment variables).

Test Plan: run the check target using different generators (ninja, make)
to see the improvement in test runtime.
isaac-io added a commit that referenced this issue Aug 19, 2022
Allow running tests in parallel by default by setting the CTEST_PARALLEL_LEVEL
environment variable. The functionality is added using a CMake script in
order to determine options when running the tests (rather than when configuring
CMake) in compatibility with the Makefile (in particular -- the `J` and
`TEST_TMPDIR` environment variables).

Test Plan: run the check target using different generators (ninja, make)
to see the improvement in test runtime.
isaac-io added a commit that referenced this issue Aug 19, 2022
Allow running tests in parallel by default by setting the CTEST_PARALLEL_LEVEL
environment variable. The functionality is added using a CMake script in
order to determine options when running the tests (rather than when configuring
CMake) in compatibility with the Makefile (in particular -- the `J` and
`TEST_TMPDIR` environment variables).

Test Plan: run the check target using different generators (ninja, make)
to see the improvement in test runtime.
isaac-io added a commit that referenced this issue Aug 19, 2022
Allow running tests in parallel by default by setting the CTEST_PARALLEL_LEVEL
environment variable. The functionality is added using a CMake script in
order to determine options when running the tests (rather than when configuring
CMake) in compatibility with the Makefile (in particular -- the `J` and
`TEST_TMPDIR` environment variables).

Test Plan: run the check target using different generators (ninja, make)
to see the improvement in test runtime.
isaac-io added a commit that referenced this issue Aug 21, 2022
Allow running tests in parallel by default by setting the CTEST_PARALLEL_LEVEL
environment variable. The functionality is added using a CMake script in
order to determine options when running the tests (rather than when configuring
CMake) in compatibility with the Makefile (in particular -- the `J` and
`TEST_TMPDIR` environment variables).

Test Plan: run the check target using different generators (ninja, make)
to see the improvement in test runtime.
isaac-io added a commit that referenced this issue Aug 24, 2022
Allow running tests in parallel by default by setting the CTEST_PARALLEL_LEVEL
environment variable. The functionality is added using a CMake script in
order to determine options when running the tests (rather than when configuring
CMake) in compatibility with the Makefile (in particular -- the `J` and
`TEST_TMPDIR` environment variables).

Test Plan: run the check target using different generators (ninja, make)
to see the improvement in test runtime.
isaac-io added a commit that referenced this issue Aug 25, 2022
Allow running tests in parallel by default by setting the CTEST_PARALLEL_LEVEL
environment variable. The functionality is added using a CMake script in
order to determine options when running the tests (rather than when configuring
CMake) in compatibility with the Makefile (in particular -- the `J` and
`TEST_TMPDIR` environment variables).

While at it, add a test timeout parameter in order to kill stuck tests,
and set the default timeout to 10 minutes (we don't currently have tests
that are running longer than that).

Test Plan: run the check target using different generators (ninja, make)
to see the improvement in test runtime.
isaac-io added a commit that referenced this issue Aug 28, 2022
Allow running tests in parallel by default by setting the CTEST_PARALLEL_LEVEL
environment variable. The functionality is added using a CMake script in
order to determine options when running the tests (rather than when configuring
CMake) in compatibility with the Makefile (in particular -- the `J` and
`TEST_TMPDIR` environment variables).

While at it, add a test timeout parameter in order to kill stuck tests,
and set the default timeout to 10 minutes (we don't currently have tests
that are running longer than that).

Test Plan: run the check target using different generators (ninja, make)
to see the improvement in test runtime.
isaac-io added a commit that referenced this issue Aug 28, 2022
Allow running tests in parallel by default by setting the CTEST_PARALLEL_LEVEL
environment variable. The functionality is added using a CMake script in
order to determine options when running the tests (rather than when configuring
CMake) in compatibility with the Makefile (in particular -- the `J` and
`TEST_TMPDIR` environment variables).

While at it, add a test timeout parameter in order to kill stuck tests,
and set the default timeout to 10 minutes (we don't currently have tests
that are running longer than that).

Test Plan: run the check target using different generators (ninja, make)
to see the improvement in test runtime.
isaac-io added a commit that referenced this issue Sep 13, 2022
Allow running tests in parallel by default by setting the CTEST_PARALLEL_LEVEL
environment variable. The functionality is added using a CMake script in
order to determine options when running the tests (rather than when configuring
CMake) in compatibility with the Makefile (in particular -- the `J` and
`TEST_TMPDIR` environment variables).

While at it, add a test timeout parameter in order to kill stuck tests,
and set the default timeout to 10 minutes (we don't currently have tests
that are running longer than that).

Test Plan: run the check target using different generators (ninja, make)
to see the improvement in test runtime.
@isaac-io isaac-io added this to the v2.1.0 milestone Sep 21, 2022
isaac-io added a commit that referenced this issue Sep 23, 2022
Allow running tests in parallel by default by setting the CTEST_PARALLEL_LEVEL
environment variable. The functionality is added using a CMake script in
order to determine options when running the tests (rather than when configuring
CMake) in compatibility with the Makefile (in particular -- the `J` and
`TEST_TMPDIR` environment variables).

While at it, add a test timeout parameter in order to kill stuck tests,
and set the default timeout to 10 minutes (we don't currently have tests
that are running longer than that).

Test Plan: run the check target using different generators (ninja, make)
to see the improvement in test runtime.
@isaac-io
Copy link
Contributor Author

isaac-io commented Oct 3, 2022

Tests in the CMake configuration are executed using ctest. The default ctest output format is two lines for each test, a start line and an end line.

How to test this:

  1. Generate a CMake target configuration using your favourite generator (in the example below I'm using the Makefile generator, which is the default on Linux):
    rm -rf build && mkdir build && cd build
    cmake ..
    make -j$(nproc) check
    
  2. When ctest starts running you're seeing $(nproc) start lines before you start seeing end lines. This is the indication that tests are running in parallel (you'll also see total runtime reduced, but you can only compare before and after).

@Yuval-Ariel
Copy link
Contributor

QA passed:
image

isaac-io added a commit that referenced this issue Oct 3, 2022
Allow running tests in parallel by default by setting the CTEST_PARALLEL_LEVEL
environment variable. The functionality is added using a CMake script in
order to determine options when running the tests (rather than when configuring
CMake) in compatibility with the Makefile (in particular -- the `J` and
`TEST_TMPDIR` environment variables).

While at it, add a test timeout parameter in order to kill stuck tests,
and set the default timeout to 10 minutes (we don't currently have tests
that are running longer than that).

Test Plan: run the check target using different generators (ninja, make)
to see the improvement in test runtime.
isaac-io added a commit that referenced this issue Oct 3, 2022
Allow running tests in parallel by default by setting the CTEST_PARALLEL_LEVEL
environment variable. The functionality is added using a CMake script in
order to determine options when running the tests (rather than when configuring
CMake) in compatibility with the Makefile (in particular -- the `J` and
`TEST_TMPDIR` environment variables).

While at it, add a test timeout parameter in order to kill stuck tests,
and set the default timeout to 10 minutes (we don't currently have tests
that are running longer than that).

Test Plan: run the check target using different generators (ninja, make)
to see the improvement in test runtime.
isaac-io added a commit that referenced this issue Oct 19, 2022
Allow running tests in parallel by default by setting the CTEST_PARALLEL_LEVEL
environment variable. The functionality is added using a CMake script in
order to determine options when running the tests (rather than when configuring
CMake) in compatibility with the Makefile (in particular -- the `J` and
`TEST_TMPDIR` environment variables).

While at it, add a test timeout parameter in order to kill stuck tests,
and set the default timeout to 10 minutes (we don't currently have tests
that are running longer than that).

Test Plan: run the check target using different generators (ninja, make)
to see the improvement in test runtime.
isaac-io added a commit that referenced this issue Oct 19, 2022
Allow running tests in parallel by default by setting the CTEST_PARALLEL_LEVEL
environment variable. The functionality is added using a CMake script in
order to determine options when running the tests (rather than when configuring
CMake) in compatibility with the Makefile (in particular -- the `J` and
`TEST_TMPDIR` environment variables).

While at it, add a test timeout parameter in order to kill stuck tests,
and set the default timeout to 10 minutes (we don't currently have tests
that are running longer than that).

Test Plan: run the check target using different generators (ninja, make)
to see the improvement in test runtime.
isaac-io added a commit that referenced this issue Oct 19, 2022
Allow running tests in parallel by default by setting the CTEST_PARALLEL_LEVEL
environment variable. The functionality is added using a CMake script in
order to determine options when running the tests (rather than when configuring
CMake) in compatibility with the Makefile (in particular -- the `J` and
`TEST_TMPDIR` environment variables).

While at it, add a test timeout parameter in order to kill stuck tests,
and set the default timeout to 10 minutes (we don't currently have tests
that are running longer than that).

Test Plan: run the check target using different generators (ninja, make)
to see the improvement in test runtime.
isaac-io added a commit that referenced this issue Oct 19, 2022
Allow running tests in parallel by default by setting the CTEST_PARALLEL_LEVEL
environment variable. The functionality is added using a CMake script in
order to determine options when running the tests (rather than when configuring
CMake) in compatibility with the Makefile (in particular -- the `J` and
`TEST_TMPDIR` environment variables).

While at it, add a test timeout parameter in order to kill stuck tests,
and set the default timeout to 10 minutes (we don't currently have tests
that are running longer than that).

Test Plan: run the check target using different generators (ninja, make)
to see the improvement in test runtime.
isaac-io added a commit that referenced this issue Oct 19, 2022
Allow running tests in parallel by default by setting the CTEST_PARALLEL_LEVEL
environment variable. The functionality is added using a CMake script in
order to determine options when running the tests (rather than when configuring
CMake) in compatibility with the Makefile (in particular -- the `J` and
`TEST_TMPDIR` environment variables).

While at it, add a test timeout parameter in order to kill stuck tests,
and set the default timeout to 10 minutes (we don't currently have tests
that are running longer than that).

Test Plan: run the check target using different generators (ninja, make)
to see the improvement in test runtime.
isaac-io added a commit that referenced this issue Oct 24, 2022
Allow running tests in parallel by default by setting the CTEST_PARALLEL_LEVEL
environment variable. The functionality is added using a CMake script in
order to determine options when running the tests (rather than when configuring
CMake) in compatibility with the Makefile (in particular -- the `J` and
`TEST_TMPDIR` environment variables).

While at it, add a test timeout parameter in order to kill stuck tests,
and set the default timeout to 10 minutes (we don't currently have tests
that are running longer than that).

Test Plan: run the check target using different generators (ninja, make)
to see the improvement in test runtime.
isaac-io added a commit that referenced this issue Oct 24, 2022
Allow running tests in parallel by default by setting the CTEST_PARALLEL_LEVEL
environment variable. The functionality is added using a CMake script in
order to determine options when running the tests (rather than when configuring
CMake) in compatibility with the Makefile (in particular -- the `J` and
`TEST_TMPDIR` environment variables).

While at it, add a test timeout parameter in order to kill stuck tests,
and set the default timeout to 10 minutes (we don't currently have tests
that are running longer than that).

Test Plan: run the check target using different generators (ninja, make)
to see the improvement in test runtime.
Yuval-Ariel pushed a commit that referenced this issue Nov 23, 2022
Allow running tests in parallel by default by setting the CTEST_PARALLEL_LEVEL
environment variable. The functionality is added using a CMake script in
order to determine options when running the tests (rather than when configuring
CMake) in compatibility with the Makefile (in particular -- the `J` and
`TEST_TMPDIR` environment variables).

While at it, add a test timeout parameter in order to kill stuck tests,
and set the default timeout to 10 minutes (we don't currently have tests
that are running longer than that).

Test Plan: run the check target using different generators (ninja, make)
to see the improvement in test runtime.
Yuval-Ariel pushed a commit that referenced this issue Nov 25, 2022
Allow running tests in parallel by default by setting the CTEST_PARALLEL_LEVEL
environment variable. The functionality is added using a CMake script in
order to determine options when running the tests (rather than when configuring
CMake) in compatibility with the Makefile (in particular -- the `J` and
`TEST_TMPDIR` environment variables).

While at it, add a test timeout parameter in order to kill stuck tests,
and set the default timeout to 10 minutes (we don't currently have tests
that are running longer than that).

Test Plan: run the check target using different generators (ninja, make)
to see the improvement in test runtime.
Yuval-Ariel pushed a commit that referenced this issue Apr 30, 2023
Allow running tests in parallel by default by setting the CTEST_PARALLEL_LEVEL
environment variable. The functionality is added using a CMake script in
order to determine options when running the tests (rather than when configuring
CMake) in compatibility with the Makefile (in particular -- the `J` and
`TEST_TMPDIR` environment variables).

While at it, add a test timeout parameter in order to kill stuck tests,
and set the default timeout to 10 minutes (we don't currently have tests
that are running longer than that).

Test Plan: run the check target using different generators (ninja, make)
to see the improvement in test runtime.
Yuval-Ariel pushed a commit that referenced this issue May 4, 2023
Allow running tests in parallel by default by setting the CTEST_PARALLEL_LEVEL
environment variable. The functionality is added using a CMake script in
order to determine options when running the tests (rather than when configuring
CMake) in compatibility with the Makefile (in particular -- the `J` and
`TEST_TMPDIR` environment variables).

While at it, add a test timeout parameter in order to kill stuck tests,
and set the default timeout to 10 minutes (we don't currently have tests
that are running longer than that).

Test Plan: run the check target using different generators (ninja, make)
to see the improvement in test runtime.
udi-speedb pushed a commit that referenced this issue Nov 13, 2023
Allow running tests in parallel by default by setting the CTEST_PARALLEL_LEVEL
environment variable. The functionality is added using a CMake script in
order to determine options when running the tests (rather than when configuring
CMake) in compatibility with the Makefile (in particular -- the `J` and
`TEST_TMPDIR` environment variables).

While at it, add a test timeout parameter in order to kill stuck tests,
and set the default timeout to 10 minutes (we don't currently have tests
that are running longer than that).

Test Plan: run the check target using different generators (ninja, make)
to see the improvement in test runtime.
udi-speedb pushed a commit that referenced this issue Dec 3, 2023
Allow running tests in parallel by default by setting the CTEST_PARALLEL_LEVEL
environment variable. The functionality is added using a CMake script in
order to determine options when running the tests (rather than when configuring
CMake) in compatibility with the Makefile (in particular -- the `J` and
`TEST_TMPDIR` environment variables).

While at it, add a test timeout parameter in order to kill stuck tests,
and set the default timeout to 10 minutes (we don't currently have tests
that are running longer than that).

Test Plan: run the check target using different generators (ninja, make)
to see the improvement in test runtime.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
build Build related enhancement New feature or request Upstreamable can be upstreamed to RocksDB
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants