Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Bug] ./tools/devtool test fails on non-ec2 machines #3726

Closed
3 tasks done
roypat opened this issue May 24, 2023 · 3 comments · Fixed by #3771
Closed
3 tasks done

[Bug] ./tools/devtool test fails on non-ec2 machines #3726

roypat opened this issue May 24, 2023 · 3 comments · Fixed by #3771
Assignees
@roypat
Labels
Priority: Medium Indicates than an issue or pull request should be resolved ahead of issues or pull requests labelled

Comments

@roypat
Copy link
Contributor

roypat commented May 24, 2023

Describe the bug

Attempting to run our integration tests on a non-ec2 machine (e.g. local dev laptop) yields

roypat@ua2d7e1a6107c5b:~/Development/firecracker$ ./tools/devtool test
[Firecracker devtool] 2023-05-24 09:17:58 UTC
[Firecracker devtool] Kernel version: 5.19.0-41-generic
[Firecracker devtool] Starting test run ...
========================================== test session starts ==========================================
platform linux -- Python 3.10.6, pytest-7.2.2, pluggy-1.0.0
cachedir: ../build/pytest_cache
rootdir: /firecracker/tests, configfile: pytest.ini
plugins: xdist-3.2.1, json-report-1.5.0, metadata-2.0.4, timeout-2.1.0
timeout: 300.0s
timeout method: signal
timeout func_only: False
collected 423 items / 6 errors                                                                          

================================================ ERRORS =================================================
_______________________ ERROR collecting integration_tests/functional/test_rng.py _______________________
integration_tests/functional/test_rng.py:14: in <module>
    INSTANCE_TYPE = get_instance_type()
        NetIfaceConfig = <class 'framework.artifacts.NetIfaceConfig'>
        Path       = <class 'pathlib.Path'>
        __builtins__ = <builtins>
        __cached__ = '/firecracker/tests/integration_tests/functional/__pycache__/test_rng.cpython-310.pyc'
        __doc__    = 'Tests for the virtio-rng device'
        __file__   = '/firecracker/tests/integration_tests/functional/test_rng.py'
        __loader__ = <_pytest.assertion.rewrite.AssertionRewritingHook object at 0x7f3132830130>
        __name__   = 'test_rng'
        __package__ = ''
        __spec__   = ModuleSpec(name='test_rng', loader=<_pytest.assertion.rewrite.AssertionRewritingHook object at 0x7f3132830130>, origin='/firecracker/tests/integration_tests/functional/test_rng.py')
        check_entropy = <function check_entropy at 0x7f3131c5c430>
        get_instance_type = <function get_instance_type at 0x7f3131889c60>
        get_kernel_version = <function get_kernel_version at 0x7f31313b4940>
        pytest     = <module 'pytest' from '/usr/local/lib/python3.10/dist-packages/pytest/__init__.py'>
framework/utils_cpuid.py:79: in get_instance_type
    return imdsv2_get("/meta-data/instance-type")
framework/utils_imdsv2.py:59: in get
    raise Exception(f"IMDSv2 returned {res.status_code} for {url}")
E   Exception: IMDSv2 returned 403 for http://169.254.169.254/latest/meta-data/instance-type
        headers    = {'X-aws-ec2-metadata-token': b''}
        path       = '/meta-data/instance-type'
        res        = <Response [403]>
        self       = <framework.utils_imdsv2.IMDSv2Client object at 0x7f31324d23e0>
        url        = 'http://169.254.169.254/latest/meta-data/instance-type'
_______________ ERROR collecting integration_tests/performance/test_block_performance.py ________________
integration_tests/performance/test_block_performance.py:31: in <module>
    CONFIG = json.load(open(CONFIG_NAME_ABS, encoding="utf-8"))
E   FileNotFoundError: [Errno 2] No such file or directory: '/firecracker/tests/integration_tests/performance/configs/test_block_performance_config_5.19.json'
        BaselineProvider = <class 'framework.stats.baseline.Provider'>
        CONFIG_NAME_ABS = '/firecracker/tests/integration_tests/performance/configs/test_block_performance_config_5.19.json'
        CONFIG_NAME_REL = 'test_block_performance_config_5.19.json'
        CmdBuilder = <class 'framework.utils.CmdBuilder'>
        DictMetadataProvider = <class 'framework.stats.metadata.DictProvider'>
        DictQuery  = <class 'framework.utils.DictQuery'>
        Enum       = <enum 'Enum'>
        TEST_ID    = 'block_performance'
        __builtins__ = <builtins>
        __cached__ = '/firecracker/tests/integration_tests/performance/__pycache__/test_block_performance.cpython-310.pyc'
        __doc__    = 'Performance benchmark for block device emulation.'
        __file__   = '/firecracker/tests/integration_tests/performance/test_block_performance.py'
        __loader__ = <_pytest.assertion.rewrite.AssertionRewritingHook object at 0x7f3132830130>
        __name__   = 'test_block_performance'
        __package__ = ''
        __spec__   = ModuleSpec(name='test_block_performance', loader=<_pytest.assertion.rewrite.AssertionRewritingHook object at 0x7f3132830130>, origin='/firecracker/tests/integration_tests/performance/test_block_performance.py')
        concurrent = <module 'concurrent' from '/usr/lib/python3.10/concurrent/__init__.py'>
        defs       = <module 'integration_tests.performance.configs.defs' from '/firecracker/tests/integration_tests/performance/configs/defs.py'>
        drive_tools = <module 'host_tools.drive' from '/firecracker/tests/host_tools/drive.py'>
        get_cpu_model_name = <function get_cpu_model_name at 0x7f3131c5c700>
        get_cpu_percent = <function get_cpu_percent at 0x7f3131c139a0>
        get_instance_type = <function get_instance_type at 0x7f3131889c60>
        get_kernel_version = <function get_kernel_version at 0x7f3131c13eb0>
        json       = <module 'json' from '/usr/lib/python3.10/json/__init__.py'>
        kernel_version = '5.19'
        os         = <module 'os' from '/usr/lib/python3.10/os.py'>
        pytest     = <module 'pytest' from '/usr/local/lib/python3.10/dist-packages/pytest/__init__.py'>
        run_cmd    = <function run_cmd at 0x7f3131c137f0>
        shutil     = <module 'shutil' from '/usr/lib/python3.10/shutil.py'>
        st         = <module 'framework.stats' from '/firecracker/tests/framework/stats/__init__.py'>
________________ ERROR collecting integration_tests/performance/test_network_latency.py _________________
integration_tests/performance/test_network_latency.py:23: in <module>
    CONFIG_DICT = json.load(open(CONFIG_NAME_ABS, encoding="utf-8"))
E   FileNotFoundError: [Errno 2] No such file or directory: '/firecracker/tests/integration_tests/performance/configs/test_network_latency_config_5.19.json'
        BaselineProvider = <class 'framework.stats.baseline.Provider'>
        CONFIG_NAME_ABS = '/firecracker/tests/integration_tests/performance/configs/test_network_latency_config_5.19.json'
        CONFIG_NAME_REL = 'test_network_latency_config_5.19.json'
        CpuMap     = <class 'framework.utils.CpuMap'>
        DEFAULT_HOST_IP = '192.168.0.1'
        DictMetadataProvider = <class 'framework.stats.metadata.DictProvider'>
        DictQuery  = <class 'framework.utils.DictQuery'>
        TEST_ID    = 'network_latency'
        __builtins__ = <builtins>
        __cached__ = '/firecracker/tests/integration_tests/performance/__pycache__/test_network_latency.cpython-310.pyc'
        __doc__    = 'Tests the network latency of a Firecracker guest.'
        __file__   = '/firecracker/tests/integration_tests/performance/test_network_latency.py'
        __loader__ = <_pytest.assertion.rewrite.AssertionRewritingHook object at 0x7f3132830130>
        __name__   = 'test_network_latency'
        __package__ = ''
        __spec__   = ModuleSpec(name='test_network_latency', loader=<_pytest.assertion.rewrite.AssertionRewritingHook object at 0x7f3132830130>, origin='/firecracker/tests/integration_tests/performance/test_network_latency.py')
        consumer   = <module 'framework.stats.consumer' from '/firecracker/tests/framework/stats/consumer.py'>
        defs       = <module 'integration_tests.performance.configs.defs' from '/firecracker/tests/integration_tests/performance/configs/defs.py'>
        get_cpu_model_name = <function get_cpu_model_name at 0x7f3131c5c700>
        get_instance_type = <function get_instance_type at 0x7f3131889c60>
        get_kernel_version = <function get_kernel_version at 0x7f3131c13eb0>
        json       = <module 'json' from '/usr/lib/python3.10/json/__init__.py'>
        kernel_version = '5.19'
        os         = <module 'os' from '/usr/lib/python3.10/os.py'>
        producer   = <module 'framework.stats.producer' from '/firecracker/tests/framework/stats/producer.py'>
        pytest     = <module 'pytest' from '/usr/local/lib/python3.10/dist-packages/pytest/__init__.py'>
        re         = <module 're' from '/usr/lib/python3.10/re.py'>
_____________ ERROR collecting integration_tests/performance/test_network_tcp_throughput.py _____________
integration_tests/performance/test_network_tcp_throughput.py:31: in <module>
    CONFIG_DICT = json.load(open(CONFIG_NAME_ABS, encoding="utf-8"))
E   FileNotFoundError: [Errno 2] No such file or directory: '/firecracker/tests/integration_tests/performance/configs/test_network_tcp_throughput_config_5.19.json'
        BaselineProvider = <class 'framework.stats.baseline.Provider'>
        CONFIG_NAME_ABS = '/firecracker/tests/integration_tests/performance/configs/test_network_tcp_throughput_config_5.19.json'
        CONFIG_NAME_REL = 'test_network_tcp_throughput_config_5.19.json'
        CmdBuilder = <class 'framework.utils.CmdBuilder'>
        CpuMap     = <class 'framework.utils.CpuMap'>
        DEFAULT_HOST_IP = '192.168.0.1'
        DictMetadataProvider = <class 'framework.stats.metadata.DictProvider'>
        DictQuery  = <class 'framework.utils.DictQuery'>
        TEST_ID    = 'network_tcp_throughput'
        __builtins__ = <builtins>
        __cached__ = '/firecracker/tests/integration_tests/performance/__pycache__/test_network_tcp_throughput.cpython-310.pyc'
        __doc__    = 'Tests the network throughput of Firecracker uVMs.'
        __file__   = '/firecracker/tests/integration_tests/performance/test_network_tcp_throughput.py'
        __loader__ = <_pytest.assertion.rewrite.AssertionRewritingHook object at 0x7f3132830130>
        __name__   = 'test_network_tcp_throughput'
        __package__ = ''
        __spec__   = ModuleSpec(name='test_network_tcp_throughput', loader=<_pytest.assertion.rewrite.AssertionRewritingHook object at 0x7f3132830130>, origin='/firecracker/tests/integration_tests/performance/test_network_tcp_throughput.py')
        concurrent = <module 'concurrent' from '/usr/lib/python3.10/concurrent/__init__.py'>
        consumer   = <module 'framework.stats.consumer' from '/firecracker/tests/framework/stats/consumer.py'>
        defs       = <module 'integration_tests.performance.configs.defs' from '/firecracker/tests/integration_tests/performance/configs/defs.py'>
        get_cpu_model_name = <function get_cpu_model_name at 0x7f3131c5c700>
        get_cpu_percent = <function get_cpu_percent at 0x7f3131c139a0>
        get_instance_type = <function get_instance_type at 0x7f3131889c60>
        get_kernel_version = <function get_kernel_version at 0x7f3131c13eb0>
        json       = <module 'json' from '/usr/lib/python3.10/json/__init__.py'>
        kernel_version = '5.19'
        os         = <module 'os' from '/usr/lib/python3.10/os.py'>
        producer   = <module 'framework.stats.producer' from '/firecracker/tests/framework/stats/producer.py'>
        pytest     = <module 'pytest' from '/usr/local/lib/python3.10/dist-packages/pytest/__init__.py'>
        run_cmd    = <function run_cmd at 0x7f3131c137f0>
        time       = <module 'time' (built-in)>
__________ ERROR collecting integration_tests/performance/test_snapshot_restore_performance.py __________
integration_tests/performance/test_snapshot_restore_performance.py:26: in <module>
    CONFIG_DICT = json.load(open(CONFIG_NAME_ABS, encoding="utf-8"))
E   FileNotFoundError: [Errno 2] No such file or directory: '/firecracker/tests/integration_tests/performance/configs/test_snapshot_restore_performance_config_5.19.json'
        BaselineProvider = <class 'framework.stats.baseline.Provider'>
        CONFIG_NAME_ABS = '/firecracker/tests/integration_tests/performance/configs/test_snapshot_restore_performance_config_5.19.json'
        CONFIG_NAME_REL = 'test_snapshot_restore_performance_config_5.19.json'
        DictMetadataProvider = <class 'framework.stats.metadata.DictProvider'>
        DictQuery  = <class 'framework.utils.DictQuery'>
        MicrovmBuilder = <class 'framework.builder.MicrovmBuilder'>
        SnapshotBuilder = <class 'framework.builder.SnapshotBuilder'>
        SnapshotType = <enum 'SnapshotType'>
        TEST_ID    = 'snapshot_restore_performance'
        WORKLOAD   = 'restore'
        __builtins__ = <builtins>
        __cached__ = '/firecracker/tests/integration_tests/performance/__pycache__/test_snapshot_restore_performance.cpython-310.pyc'
        __doc__    = 'Performance benchmark for snapshot restore.'
        __file__   = '/firecracker/tests/integration_tests/performance/test_snapshot_restore_performance.py'
        __loader__ = <_pytest.assertion.rewrite.AssertionRewritingHook object at 0x7f3132830130>
        __name__   = 'test_snapshot_restore_performance'
        __package__ = ''
        __spec__   = ModuleSpec(name='test_snapshot_restore_performance', loader=<_pytest.assertion.rewrite.AssertionRewritingHook object at 0x7f3132830130>, origin='/firecracker/tests/integration_tests/performance/test_snapshot_restore_performance.py')
        create_net_devices_configuration = <function create_net_devices_configuration at 0x7f313132b130>
        defs       = <module 'integration_tests.performance.configs.defs' from '/firecracker/tests/integration_tests/performance/configs/defs.py'>
        drive_tools = <module 'host_tools.drive' from '/firecracker/tests/host_tools/drive.py'>
        get_cpu_model_name = <function get_cpu_model_name at 0x7f3131c5c700>
        get_instance_type = <function get_instance_type at 0x7f3131889c60>
        get_kernel_version = <function get_kernel_version at 0x7f3131c13eb0>
        json       = <module 'json' from '/usr/lib/python3.10/json/__init__.py'>
        lru_cache  = <function lru_cache at 0x7f3132f493f0>
        os         = <module 'os' from '/usr/lib/python3.10/os.py'>
        pytest     = <module 'pytest' from '/usr/local/lib/python3.10/dist-packages/pytest/__init__.py'>
        st         = <module 'framework.stats' from '/firecracker/tests/framework/stats/__init__.py'>
        tempfile   = <module 'tempfile' from '/usr/lib/python3.10/tempfile.py'>
________________ ERROR collecting integration_tests/performance/test_vsock_throughput.py ________________
integration_tests/performance/test_vsock_throughput.py:31: in <module>
    CONFIG_DICT = json.load(open(CONFIG_NAME_ABS, encoding="utf-8"))
E   FileNotFoundError: [Errno 2] No such file or directory: '/firecracker/tests/integration_tests/performance/configs/test_vsock_throughput_config_5.19.json'
        BaselineProvider = <class 'framework.stats.baseline.Provider'>
        CONFIG_NAME_ABS = '/firecracker/tests/integration_tests/performance/configs/test_vsock_throughput_config_5.19.json'
        CONFIG_NAME_REL = 'test_vsock_throughput_config_5.19.json'
        CmdBuilder = <class 'framework.utils.CmdBuilder'>
        CpuMap     = <class 'framework.utils.CpuMap'>
        DictMetadataProvider = <class 'framework.stats.metadata.DictProvider'>
        DictQuery  = <class 'framework.utils.DictQuery'>
        TEST_ID    = 'vsock_throughput'
        VSOCK_UDS_PATH = 'v.sock'
        __builtins__ = <builtins>
        __cached__ = '/firecracker/tests/integration_tests/performance/__pycache__/test_vsock_throughput.cpython-310.pyc'
        __doc__    = 'Tests the VSOCK throughput of Firecracker uVMs.'
        __file__   = '/firecracker/tests/integration_tests/performance/test_vsock_throughput.py'
        __loader__ = <_pytest.assertion.rewrite.AssertionRewritingHook object at 0x7f3132830130>
        __name__   = 'test_vsock_throughput'
        __package__ = ''
        __spec__   = ModuleSpec(name='test_vsock_throughput', loader=<_pytest.assertion.rewrite.AssertionRewritingHook object at 0x7f3132830130>, origin='/firecracker/tests/integration_tests/performance/test_vsock_throughput.py')
        concurrent = <module 'concurrent' from '/usr/lib/python3.10/concurrent/__init__.py'>
        consumer   = <module 'framework.stats.consumer' from '/firecracker/tests/framework/stats/consumer.py'>
        defs       = <module 'integration_tests.performance.configs.defs' from '/firecracker/tests/integration_tests/performance/configs/defs.py'>
        get_cpu_model_name = <function get_cpu_model_name at 0x7f3131c5c700>
        get_cpu_percent = <function get_cpu_percent at 0x7f3131c139a0>
        get_instance_type = <function get_instance_type at 0x7f3131889c60>
        get_kernel_version = <function get_kernel_version at 0x7f3131c13eb0>
        json       = <module 'json' from '/usr/lib/python3.10/json/__init__.py'>
        kernel_version = '5.19'
        make_host_port_path = <function make_host_port_path at 0x7f3119865e10>
        os         = <module 'os' from '/usr/lib/python3.10/os.py'>
        producer   = <module 'framework.stats.producer' from '/firecracker/tests/framework/stats/producer.py'>
        pytest     = <module 'pytest' from '/usr/local/lib/python3.10/dist-packages/pytest/__init__.py'>
        run_cmd    = <function run_cmd at 0x7f3131c137f0>
        time       = <module 'time' (built-in)>
======================================== short test summary info ========================================
ERROR integration_tests/functional/test_rng.py - Exception: IMDSv2 returned 403 for http://169.254.169.254/latest/meta-data/instance-type
ERROR integration_tests/performance/test_block_performance.py - FileNotFoundError: [Errno 2] No such file or directory: '/firecracker/tests/integration_tests/perfor...
ERROR integration_tests/performance/test_network_latency.py - FileNotFoundError: [Errno 2] No such file or directory: '/firecracker/tests/integration_tests/perfor...
ERROR integration_tests/performance/test_network_tcp_throughput.py - FileNotFoundError: [Errno 2] No such file or directory: '/firecracker/tests/integration_tests/perfor...
ERROR integration_tests/performance/test_snapshot_restore_performance.py - FileNotFoundError: [Errno 2] No such file or directory: '/firecracker/tests/integration_tests/perfor...
ERROR integration_tests/performance/test_vsock_throughput.py - FileNotFoundError: [Errno 2] No such file or directory: '/firecracker/tests/integration_tests/perfor...
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Interrupted: 6 errors during collection !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
=========================================== 6 errors in 1.59s ===========================================
[Firecracker devtool] 2023-05-24 09:18:05 UTC
[Firecracker devtool] Finished test run ...

The first one is due to test_rng.py unconditionally trying to make IMDS request to determine instance type and then checking which tests to skip. The other two are due to my ubuntu laptop having kernel 5.19 instead of 4.14 or 5.10. However, since performance tests are marked as "nonci", they wouldn't've run anyway, so this error should probably just be ignored.

To Reproduce

Run ./tools/devtool test outside of EC2, or with a kernel version thats not 4.14 or 5.10

Expected behaviour

Tests to run

Environment

  • Firecracker version: 48a3e87
  • Host and guest kernel versions: host 5.19, guest whatever the integration tests use
  • Rootfs used: whatever the integration tests use
  • Architecture: x86_64

Checks

  • Have you searched the Firecracker Issues database for similar problems?
  • Have you read the existing relevant Firecracker documentation?
  • Are you certain the bug being reported is a Firecracker issue?
@roypat roypat added the Priority: Medium Indicates than an issue or pull request should be resolved ahead of issues or pull requests labelled label May 30, 2023
@roypat roypat self-assigned this May 30, 2023
@pb8o
Copy link
Contributor

pb8o commented Jun 1, 2023

The test_rng should be fixed soon with some changes I have. The other ones related to performance baselines could be fixed by moving the baseline loading file into the BaselineProvider constructor.

@roypat roypat linked a pull request Jun 1, 2023 that will close this issue
fix: make test_rng.py work outside of EC2 #3771
Merged
9 tasks
@roypat
Copy link
Contributor Author

roypat commented Jun 1, 2023

The test_rng should be fixed soon with some changes I have. The other ones related to performance baselines could be fixed by moving the baseline loading file into the BaselineProvider constructor.

Yeah, for the baseline things, I plan to add that change to #3736 :)

@roypat
Copy link
Contributor Author

roypat commented Jul 3, 2023

For the block test, the baselines thing hasnt been fixed yet

@roypat roypat reopened this Jul 3, 2023
@roypat roypat mentioned this issue Jul 3, 2023
Closed
3 tasks
roypat added a commit to roypat/firecracker that referenced this issue Jul 6, 2023
@roypat
test(block-perf): Do not load baselines into global constant
cf8f02b 
this caused pytest to crash during test collection if running on an
unsupported kernel version.

Fixes firecracker-microvm#3726

Signed-off-by: Patrick Roy <roypat@amazon.co.uk>
roypat added a commit to roypat/firecracker that referenced this issue Jul 6, 2023
@roypat
test(block-perf): Do not load baselines into global constant
e374bcb 
this caused pytest to crash during test collection if running on an
unsupported kernel version.

Fixes firecracker-microvm#3726

Signed-off-by: Patrick Roy <roypat@amazon.co.uk>
@roypat roypat closed this as completed in 7d091c1 Jul 7, 2023
ShadowCurse pushed a commit to ShadowCurse/firecracker that referenced this issue Jul 26, 2023
@roypat @ShadowCurse
test(block-perf): Do not load baselines into global constant
dc6869f 
this caused pytest to crash during test collection if running on an
unsupported kernel version.

Fixes firecracker-microvm#3726

Signed-off-by: Patrick Roy <roypat@amazon.co.uk>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@roypat roypat

Labels
Priority: Medium Indicates than an issue or pull request should be resolved ahead of issues or pull requests labelled
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

fix: make test_rng.py work outside of EC2 roypat/firecracker
2 participants
@pb8o @roypat