Skip to content
This repository has been archived by the owner on Nov 17, 2023. It is now read-only.

[FEATURE] MXIndexedRecordIO: avoid re-build index #20549

Merged
merged 1 commit into from
Sep 16, 2021

Conversation

huww98
Copy link
Contributor

@huww98 huww98 commented Aug 23, 2021

Description

When resetting, avoid re-build index for read-only instances to save time and memory (in case of forked, memory pages can be shared)

Checklist

Essentials

  • PR's title starts with a category (e.g. [BUGFIX], [MODEL], [TUTORIAL], [FEATURE], [DOC], etc)
  • Changes are complete (i.e. I finished coding on this PR)
  • All changes have test coverage
  • Code is well-documented

Changes

  • avoid reset index

Comments

This can save ~30G memory for us when running with 16 forked workers.

When resetting, avoid re-build index for read-only instances to save time and
memory (in case of forked, memory pages can be shared)
@huww98 huww98 requested a review from szha as a code owner August 23, 2021 10:23
@mxnet-bot
Copy link

Hey @huww98 , Thanks for submitting the PR
All tests are already queued to run once. If tests fail, you can trigger one or more tests again with the following commands:

  • To trigger all jobs: @mxnet-bot run ci [all]
  • To trigger specific jobs: @mxnet-bot run ci [job1, job2]

CI supported jobs: [windows-cpu, sanity, website, edge, centos-cpu, windows-gpu, clang, centos-gpu, unix-cpu, unix-gpu, miscellaneous]


Note:
Only following 3 categories can trigger CI :PR Author, MXNet Committer, Jenkins Admin.
All CI tests must pass before the PR can be merged.

@mseth10 mseth10 added pr-awaiting-testing PR is reviewed and waiting CI build and test pr-awaiting-review PR is waiting for code review and removed pr-awaiting-testing PR is reviewed and waiting CI build and test labels Aug 23, 2021
Copy link
Member

@ptrendx ptrendx left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@ptrendx ptrendx merged commit 72e6b16 into apache:master Sep 16, 2021
@huww98 huww98 deleted the patch-1 branch September 17, 2021 05:54
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
pr-awaiting-review PR is waiting for code review
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants