Integrating PlanLoader within OSS Planner #3376
Open
+631
−80
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
meta-cla
bot
added
the
CLA Signed
aliafzal
added a commit
to aliafzal/torchrec
that referenced
this pull request
Sep 23, 2025
Summary: Integrate PlanLoader functionality within the EmbeddingShardingPlanner to enable loading and reusing pre-computed sharding plans. This integration extends the OSS planner with plan loading capabilities. This diff includes: * PlanLoader Integration in EmbeddingShardingPlanner: - Added optional `plan_loader` parameter to EmbeddingShardingPlanner constructor - Integrated plan validation using context hash comparison to ensure loaded plans are compatible with current planner configuration - Fallback to normal planning when plan loader returns null * Plan Loading Workflow:Check if loaded plan context hash matches current planner context * If mismatch detected → raise PlannerError * If validation passes → load sharding options from storage * Map loaded sharding options to current search space using storage_hash * Skip planning phase and use pre-computed plan if available * Search Space Reconstruction: * Mapping of loaded sharding options to enumerated search space * Preserving all original ShardingOption metadata while replacing shard assignments Differential Revision: D81279558
aliafzal
added a commit
to aliafzal/torchrec
that referenced
this pull request
Sep 23, 2025
Summary: Integrate PlanLoader functionality within the EmbeddingShardingPlanner to enable loading and reusing pre-computed sharding plans. This integration extends the OSS planner with plan loading capabilities. This diff includes: * PlanLoader Integration in EmbeddingShardingPlanner: - Added optional `plan_loader` parameter to EmbeddingShardingPlanner constructor - Integrated plan validation using context hash comparison to ensure loaded plans are compatible with current planner configuration - Fallback to normal planning when plan loader returns null * Plan Loading Workflow:Check if loaded plan context hash matches current planner context * If mismatch detected → raise PlannerError * If validation passes → load sharding options from storage * Map loaded sharding options to current search space using storage_hash * Skip planning phase and use pre-computed plan if available * Search Space Reconstruction: * Mapping of loaded sharding options to enumerated search space * Preserving all original ShardingOption metadata while replacing shard assignments Differential Revision: D81279558
aliafzal
force-pushed
the
export-D81279558
branch
from
482ba49 to
b77ca4e
Compare
aliafzal
added a commit
to aliafzal/torchrec
that referenced
this pull request
Sep 23, 2025
Summary: Integrate PlanLoader functionality within the EmbeddingShardingPlanner to enable loading and reusing pre-computed sharding plans. This integration extends the OSS planner with plan loading capabilities. This diff includes: * PlanLoader Integration in EmbeddingShardingPlanner: - Added optional `plan_loader` parameter to EmbeddingShardingPlanner constructor - Integrated plan validation using context hash comparison to ensure loaded plans are compatible with current planner configuration - Fallback to normal planning when plan loader returns null * Plan Loading Workflow:Check if loaded plan context hash matches current planner context * If mismatch detected → raise PlannerError * If validation passes → load sharding options from storage * Map loaded sharding options to current search space using storage_hash * Skip planning phase and use pre-computed plan if available * Search Space Reconstruction: * Mapping of loaded sharding options to enumerated search space * Preserving all original ShardingOption metadata while replacing shard assignments Differential Revision: D81279558
aliafzal
added a commit
to aliafzal/torchrec
that referenced
this pull request
Sep 29, 2025
Summary: Integrate PlanLoader functionality within the EmbeddingShardingPlanner to enable loading and reusing pre-computed sharding plans. This integration extends the OSS planner with plan loading capabilities. This diff includes: * PlanLoader Integration in EmbeddingShardingPlanner: - Added optional `plan_loader` parameter to EmbeddingShardingPlanner constructor - Integrated plan validation using context hash comparison to ensure loaded plans are compatible with current planner configuration - Fallback to normal planning when plan loader returns null * Plan Loading Workflow:Check if loaded plan context hash matches current planner context * If mismatch detected → raise PlannerError * If validation passes → load sharding options from storage * Map loaded sharding options to current search space using storage_hash * Skip planning phase and use pre-computed plan if available * Search Space Reconstruction: * Mapping of loaded sharding options to enumerated search space * Preserving all original ShardingOption metadata while replacing shard assignments Differential Revision: D81279558
aliafzal
force-pushed
the
export-D81279558
branch
from
b77ca4e to
ff9a03d
Compare
aliafzal
force-pushed
the
export-D81279558
branch
from
ff9a03d to
26f40ce
Compare
aliafzal
added a commit
to aliafzal/torchrec
that referenced
this pull request
Sep 30, 2025
Summary: X-link: meta-pytorch#3376 Integrate PlanLoader functionality within the EmbeddingShardingPlanner to enable loading and reusing pre-computed sharding plans. This integration extends the OSS planner with plan loading capabilities. This diff includes: * PlanLoader Integration in EmbeddingShardingPlanner: - Added optional `plan_loader` parameter to EmbeddingShardingPlanner constructor - Integrated plan validation using context hash comparison to ensure loaded plans are compatible with current planner configuration - Fallback to normal planning when plan loader returns null * Plan Loading Workflow:Check if loaded plan context hash matches current planner context * If mismatch detected → raise PlannerError * If validation passes → load sharding options from storage * Map loaded sharding options to current search space using storage_hash * Skip planning phase and use pre-computed plan if available * Search Space Reconstruction: * Mapping of loaded sharding options to enumerated search space * Preserving all original ShardingOption metadata while replacing shard assignments Differential Revision: D81279558
aliafzal
added a commit
to aliafzal/torchrec
that referenced
this pull request
Sep 30, 2025
Summary: X-link: meta-pytorch#3376 Integrate PlanLoader functionality within the EmbeddingShardingPlanner to enable loading and reusing pre-computed sharding plans. This integration extends the OSS planner with plan loading capabilities. This diff includes: * PlanLoader Integration in EmbeddingShardingPlanner: - Added optional `plan_loader` parameter to EmbeddingShardingPlanner constructor - Integrated plan validation using context hash comparison to ensure loaded plans are compatible with current planner configuration - Fallback to normal planning when plan loader returns null * Plan Loading Workflow:Check if loaded plan context hash matches current planner context * If mismatch detected → raise PlannerError * If validation passes → load sharding options from storage * Map loaded sharding options to current search space using storage_hash * Skip planning phase and use pre-computed plan if available * Search Space Reconstruction: * Mapping of loaded sharding options to enumerated search space * Preserving all original ShardingOption metadata while replacing shard assignments Differential Revision: D81279558
aliafzal
force-pushed
the
export-D81279558
branch
from
26f40ce to
fe96d89
Compare
aliafzal
added a commit
to aliafzal/torchrec
that referenced
this pull request
Sep 30, 2025
Summary: X-link: meta-pytorch#3376 Integrate PlanLoader functionality within the EmbeddingShardingPlanner to enable loading and reusing pre-computed sharding plans. This integration extends the OSS planner with plan loading capabilities. This diff includes: * PlanLoader Integration in EmbeddingShardingPlanner: - Added optional `plan_loader` parameter to EmbeddingShardingPlanner constructor - Integrated plan validation using context hash comparison to ensure loaded plans are compatible with current planner configuration - Fallback to normal planning when plan loader returns null * Plan Loading Workflow:Check if loaded plan context hash matches current planner context * If mismatch detected → raise PlannerError * If validation passes → load sharding options from storage * Map loaded sharding options to current search space using storage_hash * Skip planning phase and use pre-computed plan if available * Search Space Reconstruction: * Mapping of loaded sharding options to enumerated search space * Preserving all original ShardingOption metadata while replacing shard assignments Differential Revision: D81279558
aliafzal
force-pushed
the
export-D81279558
branch
from
fe96d89 to
a19ff06
Compare
aliafzal
added a commit
to aliafzal/torchrec
that referenced
this pull request
Sep 30, 2025
Summary: X-link: meta-pytorch#3376 Integrate PlanLoader functionality within the EmbeddingShardingPlanner to enable loading and reusing pre-computed sharding plans. This integration extends the OSS planner with plan loading capabilities. This diff includes: * PlanLoader Integration in EmbeddingShardingPlanner: - Added optional `plan_loader` parameter to EmbeddingShardingPlanner constructor - Integrated plan validation using context hash comparison to ensure loaded plans are compatible with current planner configuration - Fallback to normal planning when plan loader returns null * Plan Loading Workflow:Check if loaded plan context hash matches current planner context * If mismatch detected → raise PlannerError * If validation passes → load sharding options from storage * Map loaded sharding options to current search space using storage_hash * Skip planning phase and use pre-computed plan if available * Search Space Reconstruction: * Mapping of loaded sharding options to enumerated search space * Preserving all original ShardingOption metadata while replacing shard assignments Differential Revision: D81279558
aliafzal
force-pushed
the
export-D81279558
branch
from
a19ff06 to
d1a45a0
Compare
aliafzal
force-pushed
the
export-D81279558
branch
from
d1a45a0 to
45b10c9
Compare
aliafzal
added a commit
to aliafzal/torchrec
that referenced
this pull request
Oct 2, 2025
Summary: X-link: meta-pytorch#3376 Integrate PlanLoader functionality within the EmbeddingShardingPlanner to enable loading and reusing pre-computed sharding plans. This integration extends the OSS planner with plan loading capabilities. This diff includes: * PlanLoader Integration in EmbeddingShardingPlanner: - Added optional `plan_loader` parameter to EmbeddingShardingPlanner constructor - Integrated plan validation using context hash comparison to ensure loaded plans are compatible with current planner configuration - Fallback to normal planning when plan loader returns null * Plan Loading Workflow:Check if loaded plan context hash matches current planner context * If mismatch detected → raise PlannerError * If validation passes → load sharding options from storage * Map loaded sharding options to current search space using storage_hash * Skip planning phase and use pre-computed plan if available * Search Space Reconstruction: * Mapping of loaded sharding options to enumerated search space * Preserving all original ShardingOption metadata while replacing shard assignments Differential Revision: D81279558
aliafzal
force-pushed
the
export-D81279558
branch
from
45b10c9 to
2514038
Compare
aliafzal
added a commit
to aliafzal/torchrec
that referenced
this pull request
Oct 2, 2025
Summary: X-link: meta-pytorch#3376 Integrate PlanLoader functionality within the EmbeddingShardingPlanner to enable loading and reusing pre-computed sharding plans. This integration extends the OSS planner with plan loading capabilities. This diff includes: * PlanLoader Integration in EmbeddingShardingPlanner: - Added optional `plan_loader` parameter to EmbeddingShardingPlanner constructor - Integrated plan validation using context hash comparison to ensure loaded plans are compatible with current planner configuration - Fallback to normal planning when plan loader returns null * Plan Loading Workflow:Check if loaded plan context hash matches current planner context * If mismatch detected → raise PlannerError * If validation passes → load sharding options from storage * Map loaded sharding options to current search space using storage_hash * Skip planning phase and use pre-computed plan if available * Search Space Reconstruction: * Mapping of loaded sharding options to enumerated search space * Preserving all original ShardingOption metadata while replacing shard assignments Differential Revision: D81279558
aliafzal
added a commit
to aliafzal/torchrec
that referenced
this pull request
Oct 2, 2025
Summary: X-link: meta-pytorch#3376 Integrate PlanLoader functionality within the EmbeddingShardingPlanner to enable loading and reusing pre-computed sharding plans. This integration extends the OSS planner with plan loading capabilities. This diff includes: * PlanLoader Integration in EmbeddingShardingPlanner: - Added optional `plan_loader` parameter to EmbeddingShardingPlanner constructor - Integrated plan validation using context hash comparison to ensure loaded plans are compatible with current planner configuration - Fallback to normal planning when plan loader returns null * Plan Loading Workflow:Check if loaded plan context hash matches current planner context * If mismatch detected → raise PlannerError * If validation passes → load sharding options from storage * Map loaded sharding options to current search space using storage_hash * Skip planning phase and use pre-computed plan if available * Search Space Reconstruction: * Mapping of loaded sharding options to enumerated search space * Preserving all original ShardingOption metadata while replacing shard assignments Differential Revision: D81279558
aliafzal
force-pushed
the
export-D81279558
branch
from
2514038 to
d292108
Compare
aliafzal
added a commit
to aliafzal/torchrec
that referenced
this pull request
Oct 2, 2025
Summary: X-link: meta-pytorch#3376 Integrate PlanLoader functionality within the EmbeddingShardingPlanner to enable loading and reusing pre-computed sharding plans. This integration extends the OSS planner with plan loading capabilities. This diff includes: * PlanLoader Integration in EmbeddingShardingPlanner: - Added optional `plan_loader` parameter to EmbeddingShardingPlanner constructor - Integrated plan validation using context hash comparison to ensure loaded plans are compatible with current planner configuration - Fallback to normal planning when plan loader returns null * Plan Loading Workflow:Check if loaded plan context hash matches current planner context * If mismatch detected → raise PlannerError * If validation passes → load sharding options from storage * Map loaded sharding options to current search space using storage_hash * Skip planning phase and use pre-computed plan if available * Search Space Reconstruction: * Mapping of loaded sharding options to enumerated search space * Preserving all original ShardingOption metadata while replacing shard assignments Differential Revision: D81279558
aliafzal
force-pushed
the
export-D81279558
branch
from
d292108 to
26c57a1
Compare
aliafzal
added a commit
to aliafzal/torchrec
that referenced
this pull request
Oct 2, 2025
Summary: X-link: meta-pytorch#3376 Integrate PlanLoader functionality within the EmbeddingShardingPlanner to enable loading and reusing pre-computed sharding plans. This integration extends the OSS planner with plan loading capabilities. This diff includes: * PlanLoader Integration in EmbeddingShardingPlanner: - Added optional `plan_loader` parameter to EmbeddingShardingPlanner constructor - Integrated plan validation using context hash comparison to ensure loaded plans are compatible with current planner configuration - Fallback to normal planning when plan loader returns null * Plan Loading Workflow:Check if loaded plan context hash matches current planner context * If mismatch detected → raise PlannerError * If validation passes → load sharding options from storage * Map loaded sharding options to current search space using storage_hash * Skip planning phase and use pre-computed plan if available * Search Space Reconstruction: * Mapping of loaded sharding options to enumerated search space * Preserving all original ShardingOption metadata while replacing shard assignments Differential Revision: D81279558
aliafzal
force-pushed
the
export-D81279558
branch
from
26c57a1 to
9043301
Compare
Summary: Fixing pre-commit issue caused due to this change D79659949 https://github.com/meta-pytorch/torchrec/actions/runs/18180801024/job/51756057934?fbclid=IwY2xjawNLCBFleHRuA2FlbQIxMQBicmlkETFOR0JqU1h4cVVkOVVpNVlKAR6QCA25THzR5ZdSRoBWwt54TGJvzh9yOoAsOaU1p8ncbWyNnAhrJjfYsQd0_g_aem_d-Vg6liqJiJg8kRY5A8esA Differential Revision: D83727377
Summary: X-link: meta-pytorch#3376 Integrate PlanLoader functionality within the EmbeddingShardingPlanner to enable loading and reusing pre-computed sharding plans. This integration extends the OSS planner with plan loading capabilities. This diff includes: * PlanLoader Integration in EmbeddingShardingPlanner: - Added optional `plan_loader` parameter to EmbeddingShardingPlanner constructor - Integrated plan validation using context hash comparison to ensure loaded plans are compatible with current planner configuration - Fallback to normal planning when plan loader returns null * Plan Loading Workflow:Check if loaded plan context hash matches current planner context * If mismatch detected → raise PlannerError * If validation passes → load sharding options from storage * Map loaded sharding options to current search space using storage_hash * Skip planning phase and use pre-computed plan if available * Search Space Reconstruction: * Mapping of loaded sharding options to enumerated search space * Preserving all original ShardingOption metadata while replacing shard assignments Differential Revision: D81279558
aliafzal
force-pushed
the
export-D81279558
branch
from
9043301 to
60c60a0
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary:
Integrate PlanLoader functionality within the EmbeddingShardingPlanner to enable loading and reusing pre-computed sharding plans. This integration extends the OSS planner with plan loading capabilities.
This diff includes:
PlanLoader Integration in EmbeddingShardingPlanner:
plan_loaderparameter to EmbeddingShardingPlanner constructorPlan Loading Workflow:Check if loaded plan context hash matches current planner context
Search Space Reconstruction:
Differential Revision: D81279558