Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Smoke Tests Fail due to tolerances on my machine #217

Open
scarere opened this issue Aug 8, 2024 · 9 comments · May be fixed by #314
Open

Smoke Tests Fail due to tolerances on my machine #217

scarere opened this issue Aug 8, 2024 · 9 comments · May be fixed by #314

Comments

@scarere
Copy link
Collaborator

scarere commented Aug 8, 2024

For some reason smoke tests fail locally due to results being outside tolerances. Tried starting from a fresh environment and the issue persists. Not urgent but worth looking into eventually. Creating an issue so that I don't forget about it.

System Info

13th Gen Intel® Core™ i9-13900K × 32
Ubuntu 22.04.4 LTS
NVIDIA GeForce RTX 4090
Driver Version: 555.42.06
Cuda Version: 12.1
Python Version: 3.9.19

Steps to recreate

python3.9 -m venv test_env
source test_env/bin/activate
cd ../../FL4Health
git checkout main
git pull
pip install --upgrade pip poetry
poetry install --with "codestyle, dev, dev-local, test, picai"
python -m tests.smoke_tests.run_smoke_test

Output

$ python -m tests.smoke_tests.run_smoke_test
2024-08-08 15:59:27 INFO     Running smoke tests with parameters:
2024-08-08 15:59:27 INFO     	Server : examples.fedprox_example.server
2024-08-08 15:59:27 INFO     	Client : examples.fedprox_example.client
2024-08-08 15:59:27 INFO     	Config : tests/smoke_tests/fedprox_config.yaml
2024-08-08 15:59:27 INFO     	Dataset: examples/datasets/mnist_data/
2024-08-08 15:59:27 INFO     Preloading MNIST dataset...
2024-08-08 15:59:27 INFO     Preloading MNIST dataset...
INFO :      Data directory: examples/datasets/mnist_data
2024-08-08 15:59:27 INFO     Data directory: examples/datasets/mnist_data
Downloading http://yann.lecun.com/exdb/mnist/train-images-idx3-ubyte.gz
Failed to download (trying next):
HTTP Error 403: Forbidden

Downloading https://ossci-datasets.s3.amazonaws.com/mnist/train-images-idx3-ubyte.gz
Downloading https://ossci-datasets.s3.amazonaws.com/mnist/train-images-idx3-ubyte.gz to examples/datasets/mnist_data/MNIST/raw/train-images-idx3-ubyte.gz
100%|████████████████████████████████████████████████████| 9912422/9912422 [00:00<00:00, 22336309.08it/s]
Extracting examples/datasets/mnist_data/MNIST/raw/train-images-idx3-ubyte.gz to examples/datasets/mnist_data/MNIST/raw

Downloading http://yann.lecun.com/exdb/mnist/train-labels-idx1-ubyte.gz
Failed to download (trying next):
HTTP Error 403: Forbidden

Downloading https://ossci-datasets.s3.amazonaws.com/mnist/train-labels-idx1-ubyte.gz
Downloading https://ossci-datasets.s3.amazonaws.com/mnist/train-labels-idx1-ubyte.gz to examples/datasets/mnist_data/MNIST/raw/train-labels-idx1-ubyte.gz
100%|█████████████████████████████████████████████████████████| 28881/28881 [00:00<00:00, 1481673.44it/s]
Extracting examples/datasets/mnist_data/MNIST/raw/train-labels-idx1-ubyte.gz to examples/datasets/mnist_data/MNIST/raw

Downloading http://yann.lecun.com/exdb/mnist/t10k-images-idx3-ubyte.gz
Failed to download (trying next):
HTTP Error 403: Forbidden

Downloading https://ossci-datasets.s3.amazonaws.com/mnist/t10k-images-idx3-ubyte.gz
Downloading https://ossci-datasets.s3.amazonaws.com/mnist/t10k-images-idx3-ubyte.gz to examples/datasets/mnist_data/MNIST/raw/t10k-images-idx3-ubyte.gz
100%|████████████████████████████████████████████████████| 1648877/1648877 [00:00<00:00, 13375956.93it/s]
Extracting examples/datasets/mnist_data/MNIST/raw/t10k-images-idx3-ubyte.gz to examples/datasets/mnist_data/MNIST/raw

Downloading http://yann.lecun.com/exdb/mnist/t10k-labels-idx1-ubyte.gz
Failed to download (trying next):
HTTP Error 403: Forbidden

Downloading https://ossci-datasets.s3.amazonaws.com/mnist/t10k-labels-idx1-ubyte.gz
Downloading https://ossci-datasets.s3.amazonaws.com/mnist/t10k-labels-idx1-ubyte.gz to examples/datasets/mnist_data/MNIST/raw/t10k-labels-idx1-ubyte.gz
100%|██████████████████████████████████████████████████████████| 4542/4542 [00:00<00:00, 12683441.26it/s]
Extracting examples/datasets/mnist_data/MNIST/raw/t10k-labels-idx1-ubyte.gz to examples/datasets/mnist_data/MNIST/raw

2024-08-08 15:59:33 INFO     Finished preloading MNIST dataset
2024-08-08 15:59:33 INFO     Starting server...
2024-08-08 15:59:36 INFO     Server started
2024-08-08 15:59:36 INFO     Starting client 0
2024-08-08 15:59:36 INFO     Starting client 1
2024-08-08 15:59:36 INFO     Collecting output for Client 0...
2024-08-08 15:59:46 INFO     Output collected for Client 0
2024-08-08 15:59:46 INFO     Collecting output for Client 1...
2024-08-08 15:59:46 INFO     Output collected for Client 1
2024-08-08 15:59:46 INFO     All clients finished execution
2024-08-08 15:59:46 INFO     Collecting output for Server...
2024-08-08 15:59:49 INFO     Output collected for Server
2024-08-08 15:59:49 INFO     Server has finished execution
Traceback (most recent call last):
  File "/usr/lib/python3.9/runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/home/shawn/Code/FL4Health/tests/smoke_tests/run_smoke_test.py", line 462, in <module>
    loop.run_until_complete(
  File "/usr/lib/python3.9/asyncio/base_events.py", line 647, in run_until_complete
    return future.result()
  File "/home/shawn/Code/FL4Health/tests/smoke_tests/run_smoke_test.py", line 260, in run_smoke_test
    assert len(server_errors) == 0, f"Server metrics check failed. Errors: {server_errors}"
AssertionError: Server metrics check failed. Errors: ["Saved value for metric 'val - prediction - accuracy' (0.4656666666666667) does not match the requested value (0.4627) within requested tolerance (0.0005).", "Saved value for metric 'loss_aggregated' (2.022878646850586) does not match the requested value (2.0268) within requested tolerance (0.0005).", "Saved value for metric 'val - prediction - accuracy' (0.43377777777777776) does not match the requested value (0.4323) within requested tolerance (0.0005).", "Saved value for metric 'loss_aggregated' (1.7308796644210815) does not match the requested value (1.7356) within requested tolerance (0.0005).", "Saved value for metric 'val - prediction - accuracy' (0.5417777777777778) does not match the requested value (0.5362) within requested tolerance (0.0005).", "Saved value for metric 'loss_aggregated' (1.4460455179214478) does not match the requested value (1.4518) within requested tolerance (0.0005)."]
@nerdai
Copy link
Collaborator

nerdai commented Jan 14, 2025

@emersodb is this still an issue? Or, can we close this?

@emersodb
Copy link
Collaborator

@emersodb is this still an issue? Or, can we close this?

Yes, it's still an issue I believe. Shawn runs smokes on his machine through the GPU by default and there appears to be different numerical fluctuations on the NVIDIA hardware than on CPU. So a few of our smokes fail. It's not a dealbreaker, because our remote smokes run on CPUs. So it hasn't been prioritized. It's likely an easy fix, just hasn't been addressed.

@scarere
Copy link
Collaborator Author

scarere commented Jan 14, 2025

Can confirm this is still an issue on the current master branch.

@nerdai
Copy link
Collaborator

nerdai commented Jan 14, 2025

Oh interesting. Should we just hardcode the device to CPU for these smoke tests?

@emersodb
Copy link
Collaborator

Oh interesting. Should we just hardcode the device to CPU for these smoke tests?

We do want them to work on both I suppose? Once in a while there are gotchas that only come up when you run them on GPU. Shawn found one or two of these at one point when he was doing local debugging. Perhaps we can adjust the tolerances depending on the device type detected.

@nerdai
Copy link
Collaborator

nerdai commented Jan 14, 2025 •
edited
Loading

Makes sense. Ya, was going to say that next. Something like parametrize the tests with device, and expected tolerance. I can take a look into this today. As a means to get more ramped up on the codebase but also clear outstanding issues :)

@nerdai nerdai linked a pull request Jan 16, 2025 that will close this issue
Open
@scarere
Copy link
Collaborator Author

scarere commented Jan 17, 2025

Tested out the new smoke test pytests on my machine following PR 314 and received 4 failures all due to being outside tolerances. In the PR it says the tolerance was loosened to 0.1 but on my machine the tests are saying that the tolerance is 0.0005. Perhaps some tests were missed. Here is the summarized output

FAILED tests/smoke_tests/test_smoke_tests.py::test_basic_server_client_cifar - AssertionError: Server metrics check failed. Errors: ["Saved value for metric 'val - prediction - accuracy' (0.1046) does not match the requested value (0.1031) within requested tolerance (0.0005).", "Saved value for metric 'test - prediction - accuracy' (0.1018) does not match the requested value (0.1039) within requested tolerance (0...
FAILED tests/smoke_tests/test_smoke_tests.py::test_scaffold - AssertionError: Server metrics check failed. Errors: ["Saved value for metric 'val - prediction - accuracy' (0.40755555555555556) does not match the requested value (0.4062) within requested tolerance (0.0005)."]
FAILED tests/smoke_tests/test_smoke_tests.py::test_apfl - AssertionError: Server metrics check failed. Errors: ["Saved value for metric 'val - personal - accuracy' (0.5255555555555556) does not match the requested value (0.5269) within requested tolerance (0.0005).", "Saved value for metric 'val - global - accuracy' (0.5414444444444444) does not match the requested value (0.5331) within reque...
FAILED tests/smoke_tests/test_smoke_tests.py::test_feddg_ga - AssertionError: Server metrics check failed. Errors: ["Saved value for metric 'val - personal - accuracy' (0.5255555555555556) does not match the requested value (0.5269) within requested tolerance (0.0005).", "Saved value for metric 'val - global - accuracy' (0.5414444444444444) does not match the requested value (0.5331) within reque...

@emersodb
Copy link
Collaborator

@scarere: We ended up reverting the default tolerance bump back to 0.0005. That would be a pretty high tolerance that would sort of kill the tests utility a bit.

We haven't done it yet, but we're hoping to have some device and perhaps os specific targets for these metrics. For example, even when running strictly on CPUs, the APFL and FedDG-GA smokes fail locally on my machine but are okay on the github Ubuntu machines.

It looks like using a GPU adds two other smokes that need tweaking.

@scarere
Copy link
Collaborator Author

scarere commented Jan 17, 2025 •
edited
Loading

@emersodb Yeah I agree that 0.1 is too high of a tolerance. I was talking to @nerdai about this in the slack and generating os/device specific targets might turn into an endless game of running the tests on more and more device configurations to generate more and more targets for each test. I think as a temporary fix Andrei suggested having custom tolerances based on the device. Based on the test results I got I think we could get away with a tolerance of 0.005 o cuda devices, one order of magnitude larger. Although if you're getting failures on cpu from machine to machine then maybe loosening default tolerance to 0.005 for all machines is a better solution as long as that isn't too high for your liking

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

[CU-868byqhk5] Fix smoke tests for GPU VectorInstitute/FL4Health
3 participants
@scarere @emersodb @nerdai