Skip to content
This repository has been archived by the owner on Nov 3, 2023. It is now read-only.

Fix Wizard unit test GPU OOMs #3539

Merged
merged 4 commits into from
Mar 19, 2021
Merged

Fix Wizard unit test GPU OOMs #3539

merged 4 commits into from
Mar 19, 2021

Conversation

EricMichaelSmith
Copy link
Contributor

@EricMichaelSmith EricMichaelSmith commented Mar 19, 2021

Reduce the batch size of two Wizard tasks to prevent them from sometimes OOMing on the long_gpu_tests CI check

@EricMichaelSmith
Copy link
Contributor Author

@stephenroller I've been able to reproduce a GPU OOM when doing nothing but adding 2 new no-op unit tests. @klshuster is this similar to your current issue?

@klshuster
Copy link
Contributor

yes!!!!

the exact same test is OOMing for me. I'm investigating right now

@klshuster
Copy link
Contributor

that wizard test evaluates over the entire WoW dataset which is certainly not necessary

@EricMichaelSmith EricMichaelSmith changed the title [WIP] Add 2 new no-op tests Fix Wizard unit test GPU OOMs Mar 19, 2021
@EricMichaelSmith EricMichaelSmith marked this pull request as ready for review March 19, 2021 16:00
@klshuster klshuster merged commit 1a3ffe0 into master Mar 19, 2021
@klshuster klshuster deleted the new-gpu-test-debugging branch March 19, 2021 16:53
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants