Skip to content
This repository has been archived by the owner on Nov 3, 2023. It is now read-only.

[teachers] Speed up conversations teacher #4404

Merged
merged 6 commits into from
Mar 20, 2022
Merged

[teachers] Speed up conversations teacher #4404

merged 6 commits into from
Mar 20, 2022

Conversation

stephenroller
Copy link
Contributor

Patch description
Conversations teacher unnecessarily loads things into memory twice. This causes a higher peak memory usage, which causes the LCCC test to break on CI (OOMs). It also seems to have things be slower than necessary (this patch is about a 2x speedup).

Testing steps

  • pytest tests/test_conversations.py parlai/tasks/lccc/test.py
  • CI

Copy link
Contributor

@dexterju27 dexterju27 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. I don't really get why the old way would cause the 2x memory usage? was that due to the delay of json file parsing?

@stephenroller stephenroller merged commit 74e12d1 into main Mar 20, 2022
@stephenroller stephenroller deleted the roller/lccc branch March 20, 2022 18:28
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants