Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Extendability refactors #1290

Merged
merged 19 commits into from
Jun 20, 2024
Merged

Extendability refactors #1290

merged 19 commits into from
Jun 20, 2024

Conversation

dakinggg
Copy link
Collaborator

@dakinggg dakinggg commented Jun 19, 2024

This PR includes a few changes for increased extendability of the code:

  • Fixes some typing
  • Adds slice_attention_mask to MPTBlock
  • Removes the need for extra imports in configuration_mpt.py just for HF checkpointing
  • Factors out the attention config validation
  • Makes the block class overridable on MPTModel
  • Adds a registry for transforms that can be applied to the TrainConfig

Loss before and after:
Screenshot 2024-06-19 at 10 54 00 PM

@dakinggg dakinggg marked this pull request as ready for review June 20, 2024 05:59
@dakinggg dakinggg requested a review from a team as a code owner June 20, 2024 05:59
Copy link
Contributor

@snarayan21 snarayan21 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

few comments

llmfoundry/models/layers/blocks.py Show resolved Hide resolved
@dakinggg dakinggg changed the title [WIP] Extendability refactors Extendability refactors Jun 20, 2024
@mosaicml mosaicml deleted a comment from snarayan21 Jun 20, 2024
llmfoundry/models/mpt/configuration_mpt.py Outdated Show resolved Hide resolved
llmfoundry/models/mpt/modeling_mpt.py Show resolved Hide resolved
llmfoundry/registry.py Show resolved Hide resolved
llmfoundry/registry.py Show resolved Hide resolved
llmfoundry/utils/builders.py Show resolved Hide resolved
llmfoundry/utils/config_utils.py Show resolved Hide resolved
llmfoundry/utils/config_utils.py Show resolved Hide resolved
llmfoundry/utils/config_utils.py Outdated Show resolved Hide resolved
llmfoundry/utils/huggingface_hub_utils.py Show resolved Hide resolved
llmfoundry/utils/huggingface_hub_utils.py Show resolved Hide resolved
@dakinggg dakinggg requested a review from milocress June 20, 2024 18:23
Copy link
Contributor

@milocress milocress left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good pending GPU test fix

tests/models/layers/test_dmoe.py:135: in test_dmoe
    expert_parallel_group = device_mesh['expert_parallel'].get_group(0)
/usr/lib/python3/dist-packages/composer/trainer/_patch_pytorch.py:1041: in device_mesh__getitem__
    submesh = _mesh_resources.create_child_mesh(self, mesh_dim_names)
E   NameError: name '_mesh_resources' is not defined

@dakinggg
Copy link
Collaborator Author

@milocress the GPU test is unrelated. It will be fixed by the next composer release (which is why that test isn't marked as required yet)

@dakinggg dakinggg merged commit 8241f9c into mosaicml:main Jun 20, 2024
10 of 11 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants