Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: Move target tensor to model output device in check_module_parameters_updated #3567

Merged
merged 3 commits into from
Sep 11, 2023

Conversation

jeffkinnison
Copy link
Contributor

Running any test that calls tests.integration_tests.parameter_update_utils.check_module_parameters_updated in an environment with a GPU leads to a device mismatch error, with the model on GPU and the target tensor on CPU. This fixes the device mismatch by putting the target tensor on the model device in this function.

@github-actions
Copy link

github-actions bot commented Aug 31, 2023

Unit Test Results

  6 files  ±       0    6 suites  ±0   1h 11m 47s ⏱️ - 17m 37s
34 tests  - 2 772  29 ✔️  - 2 764    5 💤  - 7  0  - 1 
88 runs   - 2 761  72 ✔️  - 2 755  16 💤  - 5  0  - 1 

Results for commit 91b90d5. ± Comparison against base commit 4a72e67.

♻️ This comment has been updated with latest results.

@jeffkinnison jeffkinnison changed the title fix: Move target tensor to model device in check_module_parameters_updated fix: Move target tensor to model output device in check_module_parameters_updated Aug 31, 2023
@jeffkinnison
Copy link
Contributor Author

Updated because some inputs have a device attribute and some don't. Instead, we set the device to match the model output passed into the loss function.

@jeffkinnison
Copy link
Contributor Author

Just so it's noted, we still get 48 tabtransformer failures with this fix, though it seems to be because tabtransformer parameters aren't updating.

FAILED tests/ludwig/combiners/test_combiners.py::test_tabtransformer_combiner_number_or_binary_without_category[None-None-concat-1-feature_list0] - AssertionError: Failed to update parameters. Parameters not updated: ['laye...
FAILED tests/ludwig/combiners/test_combiners.py::test_tabtransformer_combiner_number_or_binary_without_category[None-None-concat-1-feature_list2] - AssertionError: Failed to update parameters. Parameters not updated: ['laye...
FAILED tests/ludwig/combiners/test_combiners.py::test_tabtransformer_combiner_number_or_binary_without_category[None-None-concat-2-feature_list0] - AssertionError: Failed to update parameters. Parameters not updated: ['laye...
FAILED tests/ludwig/combiners/test_combiners.py::test_tabtransformer_combiner_number_or_binary_without_category[None-None-concat-2-feature_list2] - AssertionError: Failed to update parameters. Parameters not updated: ['laye...
FAILED tests/ludwig/combiners/test_combiners.py::test_tabtransformer_combiner_number_or_binary_without_category[None-None-sum-1-feature_list0] - AssertionError: Failed to update parameters. Parameters not updated: ['laye...
FAILED tests/ludwig/combiners/test_combiners.py::test_tabtransformer_combiner_number_or_binary_without_category[None-None-sum-1-feature_list2] - AssertionError: Failed to update parameters. Parameters not updated: ['laye...
FAILED tests/ludwig/combiners/test_combiners.py::test_tabtransformer_combiner_number_or_binary_without_category[None-None-sum-2-feature_list0] - AssertionError: Failed to update parameters. Parameters not updated: ['laye...
FAILED tests/ludwig/combiners/test_combiners.py::test_tabtransformer_combiner_number_or_binary_without_category[None-None-sum-2-feature_list2] - AssertionError: Failed to update parameters. Parameters not updated: ['laye...
FAILED tests/ludwig/combiners/test_combiners.py::test_tabtransformer_combiner_number_or_binary_without_category[None-fc_layers1-concat-1-feature_list0] - AssertionError: Failed to update parameters. Parameters not updated: ['laye...
FAILED tests/ludwig/combiners/test_combiners.py::test_tabtransformer_combiner_number_or_binary_without_category[None-fc_layers1-concat-1-feature_list2] - AssertionError: Failed to update parameters. Parameters not updated: ['laye...
FAILED tests/ludwig/combiners/test_combiners.py::test_tabtransformer_combiner_number_or_binary_without_category[None-fc_layers1-concat-2-feature_list0] - AssertionError: Failed to update parameters. Parameters not updated: ['laye...
FAILED tests/ludwig/combiners/test_combiners.py::test_tabtransformer_combiner_number_or_binary_without_category[None-fc_layers1-concat-2-feature_list2] - AssertionError: Failed to update parameters. Parameters not updated: ['laye...
FAILED tests/ludwig/combiners/test_combiners.py::test_tabtransformer_combiner_number_or_binary_without_category[None-fc_layers1-sum-1-feature_list0] - AssertionError: Failed to update parameters. Parameters not updated: ['laye...
FAILED tests/ludwig/combiners/test_combiners.py::test_tabtransformer_combiner_number_or_binary_without_category[None-fc_layers1-sum-1-feature_list2] - AssertionError: Failed to update parameters. Parameters not updated: ['laye...
FAILED tests/ludwig/combiners/test_combiners.py::test_tabtransformer_combiner_number_or_binary_without_category[None-fc_layers1-sum-2-feature_list0] - AssertionError: Failed to update parameters. Parameters not updated: ['laye...
FAILED tests/ludwig/combiners/test_combiners.py::test_tabtransformer_combiner_number_or_binary_without_category[None-fc_layers1-sum-2-feature_list2] - AssertionError: Failed to update parameters. Parameters not updated: ['laye...
FAILED tests/ludwig/combiners/test_combiners.py::test_tabtransformer_combiner_number_or_binary_without_category[64-None-concat-1-feature_list0] - AssertionError: Failed to update parameters. Parameters not updated: ['embe...
FAILED tests/ludwig/combiners/test_combiners.py::test_tabtransformer_combiner_number_or_binary_without_category[64-None-concat-1-feature_list2] - AssertionError: Failed to update parameters. Parameters not updated: ['embe...
FAILED tests/ludwig/combiners/test_combiners.py::test_tabtransformer_combiner_number_or_binary_without_category[64-None-concat-2-feature_list0] - AssertionError: Failed to update parameters. Parameters not updated: ['embe...
FAILED tests/ludwig/combiners/test_combiners.py::test_tabtransformer_combiner_number_or_binary_without_category[64-None-concat-2-feature_list2] - AssertionError: Failed to update parameters. Parameters not updated: ['embe...
FAILED tests/ludwig/combiners/test_combiners.py::test_tabtransformer_combiner_number_or_binary_without_category[64-None-sum-1-feature_list0] - AssertionError: Failed to update parameters. Parameters not updated: ['embe...
FAILED tests/ludwig/combiners/test_combiners.py::test_tabtransformer_combiner_number_or_binary_without_category[64-None-sum-1-feature_list2] - AssertionError: Failed to update parameters. Parameters not updated: ['embe...
FAILED tests/ludwig/combiners/test_combiners.py::test_tabtransformer_combiner_number_or_binary_without_category[64-None-sum-2-feature_list0] - AssertionError: Failed to update parameters. Parameters not updated: ['embe...
FAILED tests/ludwig/combiners/test_combiners.py::test_tabtransformer_combiner_number_or_binary_without_category[64-None-sum-2-feature_list2] - AssertionError: Failed to update parameters. Parameters not updated: ['embe...
FAILED tests/ludwig/combiners/test_combiners.py::test_tabtransformer_combiner_number_or_binary_without_category[64-fc_layers1-concat-1-feature_list0] - AssertionError: Failed to update parameters. Parameters not updated: ['embe...
FAILED tests/ludwig/combiners/test_combiners.py::test_tabtransformer_combiner_number_or_binary_without_category[64-fc_layers1-concat-1-feature_list2] - AssertionError: Failed to update parameters. Parameters not updated: ['embe...
FAILED tests/ludwig/combiners/test_combiners.py::test_tabtransformer_combiner_number_or_binary_without_category[64-fc_layers1-concat-2-feature_list0] - AssertionError: Failed to update parameters. Parameters not updated: ['embe...
FAILED tests/ludwig/combiners/test_combiners.py::test_tabtransformer_combiner_number_or_binary_without_category[64-fc_layers1-concat-2-feature_list2] - AssertionError: Failed to update parameters. Parameters not updated: ['embe...
FAILED tests/ludwig/combiners/test_combiners.py::test_tabtransformer_combiner_number_or_binary_without_category[64-fc_layers1-sum-1-feature_list0] - AssertionError: Failed to update parameters. Parameters not updated: ['embe...
FAILED tests/ludwig/combiners/test_combiners.py::test_tabtransformer_combiner_number_or_binary_without_category[64-fc_layers1-sum-1-feature_list2] - AssertionError: Failed to update parameters. Parameters not updated: ['embe...
FAILED tests/ludwig/combiners/test_combiners.py::test_tabtransformer_combiner_number_or_binary_without_category[64-fc_layers1-sum-2-feature_list0] - AssertionError: Failed to update parameters. Parameters not updated: ['embe...
FAILED tests/ludwig/combiners/test_combiners.py::test_tabtransformer_combiner_number_or_binary_without_category[64-fc_layers1-sum-2-feature_list2] - AssertionError: Failed to update parameters. Parameters not updated: ['embe...
FAILED tests/ludwig/combiners/test_combiners.py::test_tabtransformer_combiner_number_or_binary_without_category[add-None-concat-1-feature_list0] - AssertionError: Failed to update parameters. Parameters not updated: ['embe...
FAILED tests/ludwig/combiners/test_combiners.py::test_tabtransformer_combiner_number_or_binary_without_category[add-None-concat-1-feature_list2] - AssertionError: Failed to update parameters. Parameters not updated: ['embe...
FAILED tests/ludwig/combiners/test_combiners.py::test_tabtransformer_combiner_number_or_binary_without_category[add-None-concat-2-feature_list0] - AssertionError: Failed to update parameters. Parameters not updated: ['embe...
FAILED tests/ludwig/combiners/test_combiners.py::test_tabtransformer_combiner_number_or_binary_without_category[add-None-concat-2-feature_list2] - AssertionError: Failed to update parameters. Parameters not updated: ['embe...
FAILED tests/ludwig/combiners/test_combiners.py::test_tabtransformer_combiner_number_or_binary_without_category[add-None-sum-1-feature_list0] - AssertionError: Failed to update parameters. Parameters not updated: ['embe...
FAILED tests/ludwig/combiners/test_combiners.py::test_tabtransformer_combiner_number_or_binary_without_category[add-None-sum-1-feature_list2] - AssertionError: Failed to update parameters. Parameters not updated: ['embe...
FAILED tests/ludwig/combiners/test_combiners.py::test_tabtransformer_combiner_number_or_binary_without_category[add-None-sum-2-feature_list0] - AssertionError: Failed to update parameters. Parameters not updated: ['embe...
FAILED tests/ludwig/combiners/test_combiners.py::test_tabtransformer_combiner_number_or_binary_without_category[add-None-sum-2-feature_list2] - AssertionError: Failed to update parameters. Parameters not updated: ['embe...
FAILED tests/ludwig/combiners/test_combiners.py::test_tabtransformer_combiner_number_or_binary_without_category[add-fc_layers1-concat-1-feature_list0] - AssertionError: Failed to update parameters. Parameters not updated: ['embe...
FAILED tests/ludwig/combiners/test_combiners.py::test_tabtransformer_combiner_number_or_binary_without_category[add-fc_layers1-concat-1-feature_list2] - AssertionError: Failed to update parameters. Parameters not updated: ['embe...
FAILED tests/ludwig/combiners/test_combiners.py::test_tabtransformer_combiner_number_or_binary_without_category[add-fc_layers1-concat-2-feature_list0] - AssertionError: Failed to update parameters. Parameters not updated: ['embe...
FAILED tests/ludwig/combiners/test_combiners.py::test_tabtransformer_combiner_number_or_binary_without_category[add-fc_layers1-concat-2-feature_list2] - AssertionError: Failed to update parameters. Parameters not updated: ['embe...
FAILED tests/ludwig/combiners/test_combiners.py::test_tabtransformer_combiner_number_or_binary_without_category[add-fc_layers1-sum-1-feature_list0] - AssertionError: Failed to update parameters. Parameters not updated: ['embe...
FAILED tests/ludwig/combiners/test_combiners.py::test_tabtransformer_combiner_number_or_binary_without_category[add-fc_layers1-sum-1-feature_list2] - AssertionError: Failed to update parameters. Parameters not updated: ['embe...
FAILED tests/ludwig/combiners/test_combiners.py::test_tabtransformer_combiner_number_or_binary_without_category[add-fc_layers1-sum-2-feature_list0] - AssertionError: Failed to update parameters. Parameters not updated: ['embe...
FAILED tests/ludwig/combiners/test_combiners.py::test_tabtransformer_combiner_number_or_binary_without_category[add-fc_layers1-sum-2-feature_list2] - AssertionError: Failed to update parameters. Parameters not updated: ['embe...

@jeffkinnison jeffkinnison merged commit 63be683 into master Sep 11, 2023
@jeffkinnison jeffkinnison deleted the int-test-util-device-mismatch branch September 11, 2023 19:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants