Add FusedLinerCrossEntropy support for Phi3

[tyler-romero]

Summary

Add FusedLinearCrossEntropy support for Phi3. #98

Testing Done

• Hardware Type: 4090
• run make test to ensure correctness
• run make checkstyle to ensure code style
• run make test-convergence to ensure convergence

[tyler-romero]

Need to rebase after merging #76

[ByronHsu]

@tyler-romero are you on discord? please say hi! https://discord.gg/nSeNms8u

[tyler-romero]

> make all
python -m pytest --disable-warnings test/ --ignore=test/convergence
=========================================================================================== test session starts ===========================================================================================
platform linux -- Python 3.10.13, pytest-8.3.2, pluggy-1.5.0
rootdir: /home/tromero/workspace/Liger-Kernel
plugins: devtools-0.12.2
collected 141 items                                                                                                                                                                                       

test/transformers/test_auto_model.py .                                                                                                                                                              [  0%]
test/transformers/test_cross_entropy.py ........................................................ss                                                                                                  [ 41%]
test/transformers/test_fused_linear_cross_entropy.py ......                                                                                                                                         [ 46%]
test/transformers/test_geglu.py ........                                                                                                                                                            [ 51%]
test/transformers/test_monkey_patch.py .....                                                                                                                                                        [ 55%]
test/transformers/test_rms_norm.py ................................                                                                                                                                 [ 78%]
test/transformers/test_rope.py ............                                                                                                                                                         [ 86%]
test/transformers/test_swiglu.py ................                                                                                                                                                   [ 97%]
test/transformers/test_trainer_integration.py .                                                                                                                                                     [ 98%]
test/triton/test_triton_monkey_patch.py ..                                                                                                                                                          [100%]

================================================================================ 139 passed, 2 skipped in 73.13s (0:01:13) ================================================================================
HF_DATASETS_OFFLINE=1 python -m pytest --disable-warnings test/convergence
=========================================================================================== test session starts ===========================================================================================
platform linux -- Python 3.10.13, pytest-8.3.2, pluggy-1.5.0
rootdir: /home/tromero/workspace/Liger-Kernel
plugins: devtools-0.12.2
collected 28 items                                                                                                                                                                                        

test/convergence/test_mini_models.py ..............                                                                                                                                                 [ 50%]
test/convergence/test_mini_models_no_logits.py ..............                                                                                                                                       [100%]

===================================================================================== 28 passed in 151.42s (0:02:31) ======================================================================================
flake8 .; flake8_status=$?; \
isort .; isort_status=$?; \
black .; black_status=$?; \
if [ $flake8_status -ne 0 ] || [ $isort_status -ne 0 ] || [ $black_status -ne 0 ]; then \
        exit 1; \
fi
Skipped 2 files
All done! ✨ 🍰 ✨
58 files left unchanged.

[tyler-romero]

This was breaking one of the monkeypatch tests on main

[lancerts]

@tyler-romero what is the root cause? Is it still breaking on the current main?

[tyler-romero]

Yes as of now its still broken on main:

> make test
python -m pytest --disable-warnings test/ --ignore=test/convergence
=========================================================================================== test session starts ===========================================================================================
platform linux -- Python 3.10.13, pytest-8.3.2, pluggy-1.5.0
rootdir: /home/tromero/workspace/Liger-Kernel
plugins: devtools-0.12.2
collected 141 items                                                                                                                                                                                       

test/transformers/test_auto_model.py .                                                                                                                                                              [  0%]
test/transformers/test_cross_entropy.py ........................................................ss                                                                                                  [ 41%]
test/transformers/test_fused_linear_cross_entropy.py ......                                                                                                                                         [ 46%]
test/transformers/test_geglu.py ........                                                                                                                                                            [ 51%]
test/transformers/test_monkey_patch.py ....F                                                                                                                                                        [ 55%]
test/transformers/test_rms_norm.py ................................                                                                                                                                 [ 78%]
test/transformers/test_rope.py ............                                                                                                                                                         [ 86%]
test/transformers/test_swiglu.py ................                                                                                                                                                   [ 97%]
test/transformers/test_trainer_integration.py .                                                                                                                                                     [ 98%]
test/triton/test_triton_monkey_patch.py ..                                                                                                                                                          [100%]

================================================================================================ FAILURES =================================================================================================
__________________________________________________________________________________ test_patching_apis_match_auto_mapping __________________________________________________________________________________

    def test_patching_apis_match_auto_mapping():
        # Test that all of the patching APIs present also have a corresponding entry in the auto mapping
        patching_functions = [
            func
            for name, func in inspect.getmembers(monkey_patch, inspect.isfunction)
            if name.startswith("apply_liger_kernel_to_")
        ]
    
>       assert set(patching_functions) == set(MODEL_TYPE_TO_APPLY_LIGER_FN.values())
E       assert {<function ap...86dbbe0>, ...} == {<function ap...70e7586db7f0>}
E         
E         Extra items in the left set:
E         <function apply_liger_kernel_to_gemma2 at 0x70e7586dba30>
E         Use -v to get more diff

test/transformers/test_monkey_patch.py:95: AssertionError
========================================================================================= short test summary info =========================================================================================
FAILED test/transformers/test_monkey_patch.py::test_patching_apis_match_auto_mapping - assert {<function ap...86dbbe0>, ...} == {<function ap...70e7586db7f0>}

[tyler-romero]

The test is just checking for the presence of this function in the mapping

[shimizust]

I think there were simultaneous commits merged with this test added and a new model type added. Are you able to fix the test? It's just making sure all the patching APIs are accounted for in the mapping (used with AutoModel class)

[tyler-romero]

Yes the test is also fixed by this PR!

[shimizust]

lgtm, thanks for the contribution!