Few issues running tests on dev machine

[gabhijit]

Following issues are observed while running tests on my dev machine (ubuntu 20.04/python 3.8.5)

1. When I am trying to run tests on my dev machine, the test cases wait for a very long time As shown below -

$ pytest-3 tests/test_run_reference_vmm.py 
======================================================================== test session starts =========================================================================
platform linux -- Python 3.8.5, pytest-4.6.9, py-1.8.1, pluggy-0.13.1
rootdir: /home/gabhijit/Work/hyphenOs/rust-vmm/vmm-reference
collecting ... 

This likely happens due to all the download happening when the test_run_reference_vmm.py module gets 'import'ed by pytest-3. Likely this can be fixed by moving the download code inside a setUp function (as discussed in one of the review comments.)

2. When I run cargo test --all Following test case fails

running 1 test
test resource_download::tests::test_error_cases ... FAILED

failures:

---- resource_download::tests::test_error_cases stdout ----
thread 'resource_download::tests::test_error_cases' panicked at 'assertion failed: matches!(s3_download("", None) . unwrap_err(), Error :: DownloadError(e) if e
         . contains("Missing required parameter"))', src/utils/src/resource_download.rs:51:9
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace


failures:
    resource_download::tests::test_error_cases

test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 0 filtered out

error: test failed, to rerun pass '-p utils --lib'

This failure happens due to python3-boto3 not been installed on my dev machine. I believe some updates to documentation should be able to fix this.

3. Documentation does not accurately reflect current state for the testing. Some updates (Just points, actual documentation may be different.)

• Document that tests require python3 and pytest and boto3 packages installed.
• Give quick instructions to download and install the above packages, both on distro and as a pip install
• Any other miscellaneous updates.

4. Also noticed a small issue - Inside the following file trying to debug one of the above issues -

#!/usr/bin/python3
# Copyright 2021 Amazon.com, Inc. or its affiliates. All Rights Reserved.
# SPDX-License-Identifier: Apache-2.0 OR BSD-3-Clause
"""Script for downloading the resources required for tests.
It returns the absolute path of the downloaded resource to stdout."""

from optparse import OptionParser
...

The first line should use /usr/bin/env python3 instead of having the hard-coded /usr/bin/python3. This will allow people to run tests with virtualenv or tools like pyenv and not necessarily system python3 that comes with the distro.

[andreeaflorescu]

Following issues are observed while running tests on my dev machine (ubuntu 20.04/python 3.8.5)

1. When I am trying to run tests on my dev machine, the test cases wait for a very long time As shown below -

$ pytest-3 tests/test_run_reference_vmm.py 
======================================================================== test session starts =========================================================================
platform linux -- Python 3.8.5, pytest-4.6.9, py-1.8.1, pluggy-0.13.1
rootdir: /home/gabhijit/Work/hyphenOs/rust-vmm/vmm-reference
collecting ... 

This likely happens due to all the download happening when the test_run_reference_vmm.py module gets 'import'ed by pytest-3. Likely this can be fixed by moving the download code inside a setUp function (as discussed in one of the review comments.)

1. When I run cargo test --all Following test case fails

running 1 test
test resource_download::tests::test_error_cases ... FAILED

failures:

---- resource_download::tests::test_error_cases stdout ----
thread 'resource_download::tests::test_error_cases' panicked at 'assertion failed: matches!(s3_download("", None) . unwrap_err(), Error :: DownloadError(e) if e
         . contains("Missing required parameter"))', src/utils/src/resource_download.rs:51:9
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace


failures:
    resource_download::tests::test_error_cases

test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 0 filtered out

error: test failed, to rerun pass '-p utils --lib'

This failure happens due to python3-boto3 not been installed on my dev machine. I believe some updates to documentation should be able to fix this.

1. Documentation does not accurately reflect current state for the testing. Some updates (Just points, actual documentation may be different.)

• Document that tests require python3 and pytest and boto3 packages installed.
• Give quick instructions to download and install the above packages, both on distro and as a pip install
• Any other miscellaneous updates.

@gabhijit we are optimizing for test runs using the rust-vmm container. To successfully run test, you need to run them with the container as follows:

docker run --device=/dev/kvm -it \
    --security-opt seccomp=unconfined \
    --volume $(pwd):/vmm-reference rustvmm/dev:v11
# in the container
pytest rust-vmm-ci tests/test_run_reference_vmm.py

This is unfortunately not documented, and we should add these steps to "Getting Started" document.

1. Also noticed a small issue - Inside the following file trying to debug one of the above issues -

#!/usr/bin/python3
# Copyright 2021 Amazon.com, Inc. or its affiliates. All Rights Reserved.
# SPDX-License-Identifier: Apache-2.0 OR BSD-3-Clause
"""Script for downloading the resources required for tests.
It returns the absolute path of the downloaded resource to stdout."""

from optparse import OptionParser
...

The first line should use /usr/bin/env python3 instead of having the hard-coded /usr/bin/python3. This will allow people to run tests with virtualenv or tools like pyenv and not necessarily system python3 that comes with the distro.

Do you want to submit a fix for this one?

[gabhijit]

I will send fixes to some of the points early next week.

[gabhijit]

#95 Fixes this.

[lauralt]

@gabhijit Is the first issue from your list fixed by #95? Should we keep the issue open and leave a comment with the current status?

[gabhijit]

@lauralt : Okay that makes sense, I am not sure though that there's an easy way of fixing the first issue - other than the one that I sent in the PR. So will it be fixed ever? Not sure, also this issue will be faced only first time running. But may be we can keep the issue open with that comment. I will add that as a separate comment.

[gabhijit]

Current Status - If you are running pytest tests/test_run_vmm_reference.py for the first time. it takes a while during the collect phase of the tests. This is currently a minor limitation, but this will not be faced on subsequent runs.