[Bug] Converting Mistral-7B-Instruct-V0.2

[scarlettekk]

🐛 Bug

Converting Mistral-7B-Instruct-v0.2 for Android results in an error.

• Platform (e.g. WebGPU/Vulkan/IOS/Android/CUDA): Android
• Operating system (e.g. Ubuntu/Windows/MacOS/...): Arch Linux
• Device (e.g. iPhone 12 Pro, PC+RTX 3090, ...): PC for compilation
• How you installed MLC-LLM (conda, source): pip
• How you installed TVM-Unity (pip, source): pip
• Python version (e.g. 3.10): 3.11.6
• TVM Unity Hash Tag: 457f5bc4c94604bbb275465cb64f951f2ecdb3f4

(mlc-llm) [scarlett@zen mlc-llm]$ python3 -m mlc_llm.build --target android --max-seq-len 768 --model ./dist/models/Mistral-7B-Instruct-v0.2 --quantization q4f16_2
Using path "dist/models/Mistral-7B-Instruct-v0.2" for model "Mistral-7B-Instruct-v0.2"
Target configured: opencl -keys=opencl,gpu -max_function_args=128 -max_num_threads=256 -max_shared_memory_per_block=16384 -max_threads_per_block=256 -texture_spatial_limit=16384 -thread_warp_size=1
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/home/scarlett/git/mlc-llm/mlc_llm/build.py", line 47, in <module>
    main()
  File "/home/scarlett/git/mlc-llm/mlc_llm/build.py", line 43, in main
    core.build_model_from_args(parsed_args)
  File "/home/scarlett/git/mlc-llm/mlc_llm/core.py", line 834, in build_model_from_args
    mod, param_manager, params, model_config = model_generators[args.model_category].get_model(
                                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/scarlett/git/mlc-llm/mlc_llm/relax_model/mistral.py", line 992, in get_model
    config = MistralConfig(
             ^^^^^^^^^^^^^^
  File "/home/scarlett/git/mlc-llm/mlc_llm/relax_model/mistral.py", line 69, in __init__
    self.max_sequence_length = sliding_window * 4
                               ~~~~~~~~~~~~~~~^~~
TypeError: unsupported operand type(s) for *: 'NoneType' and 'int'

[CharlieFRuan]

Hi @scarlettekk could you try compiling using the new workflow? https://llm.mlc.ai/docs/compilation/compile_models.html Let us know if you run into issues

[scarlettekk]

Hi,
Using the new workflow allows the model to be built successfully. I would recommend changing the page on building for Android to reflect this.

However, I now encounter issues in Android Studio building the Android demo application with the new model:

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':app:mergeExtDexDebug'.
> A failure occurred while executing com.android.build.gradle.internal.tasks.DexMergingTaskDelegate
  > There was a failure while executing work items
     > A failure occurred while executing com.android.build.gradle.internal.tasks.DexMergingWorkAction
        > com.android.builder.dexing.DexArchiveMergerException: Error while merging dex archives:
          Type org.apache.tvm.API$1 is defined multiple times: /home/scarlett/git/mlc-llm/android/app/build/intermediates/external_file_lib_dex_archives/debug/0_tvm4j_core.jar:classes.dex, /home/scarlett/git/mlc-llm/android/app/build/intermediates/external_file_lib_dex_archives/debug/1_tvm4j_core.jar:classes.dex
          Learn how to resolve the issue at https://developer.android.com/studio/build/dependencies#duplicate_classes

[scarlettekk]

I think this may have something to do with how the folder structure of the android app is different than the documentation, with the MLCChat folder missing, its contents moved to the ./android/ folder, and the prepare_libs.sh script being moved to ./android/library, meaning there is a tvm4j_core.jar under both the library/build/output directory and the app/src/main/libs directory, both of which are in scope of the android project

Edit: I can build successfully by not copying tvm4j_core.jar to the app/src/main/libs directory. Whether this prevents the app from functioning remains to be seen.

[scarlettekk]

Running the app on start shows an error: "Add model failed: Failed requirement." Trying to manually add the model url errors with "Model lib null is not supported." Is my huggingface repo configured wrong? It's linked here.

[CharlieFRuan]

Hi, thanks for the updates! These should probably be solved by this PR #1494

We are currently under a revamp of model compilation workflow, hence the mismatch of compilation/runtime in docs. This is tracked here #1420. Apologies for the inconvenience.

[CharlieFRuan]

The Android PR just got merged in #1494, with an updated documentation page as well: https://llm.mlc.ai/docs/deploy/android.html

Let us know if issues persist!

[scarlettekk]

Huh, well.... Guess I'll try building an app using the pre built model tomorrow and see if that's any better?

[CharlieFRuan]

Hmm I think I saw this issue posted by someone else as well.. I guess it could be due to various reasons; yea perhaps try the prebuilt ones first, if those do not fail it is probably mistral-specific. @Kartik14 might be good to add Mistral as prebuilt as well if we have the bandwidth.

[Kartik14]

Hmm I think I saw this issue posted by someone else as well.. I guess it could be due to various reasons; yea perhaps try the prebuilt ones first, if those do not fail it is probably mistral-specific. @Kartik14 might be good to add Mistral as prebuilt as well if we have the bandwidth.

I have just added the Mistral prebuilt lib url #1514. @scarlettekk Can you try this lib when building apk?

[scarlettekk]

Now we get this when preparing libs. I may have set up the prebuilt incorrectly? I downloaded the Mistral-android.tar to dist/libs and edited app-config.json to match the tarfile and the new URL.

[100%] Linking CXX shared library libtvm4j_runtime_packed.so
ld.lld: error: model_lib/libmodel_android.a(._mistral_q4f16_1_devc.o): not an ELF file
clang++: error: linker command failed with exit code 1 (use -v to see invocation)
make[3]: *** [CMakeFiles/tvm4j_runtime_packed.dir/build.make:104: libtvm4j_runtime_packed.so] Error 1
make[2]: *** [CMakeFiles/Makefile2:268: CMakeFiles/tvm4j_runtime_packed.dir/all] Error 2
make[1]: *** [CMakeFiles/Makefile2:275: CMakeFiles/tvm4j_runtime_packed.dir/rule] Error 2
make: *** [Makefile:203: tvm4j_runtime_packed] Error 2

[Kartik14]

Someone else was getting the same error. I think this might be an issue with the tvm. Can you try fetching the latest version of 3rdparty/tvm?

[scarlettekk]

No dice, 3rdparty/tvm checks out at 72a7644159cc415788f4d819c7e8196e0eef751d and still produces the same error in prepare_libs.sh

[hferoze]

Seeing the same issue. Using the latest repo (tvm)

Llama-2-7b-chat-hf-q4f32_1-android.tar

[100%] Built target mlc_llm_static
[100%] Building CXX object CMakeFiles/tvm4j_runtime_packed.dir/mnt/c/AndroidProjects/MLC_AI/mlc-llm/3rdparty/tvm/jvm/native/src/main/native/org_apache_tvm_native_c_api.cc.o
[100%] Linking CXX shared library libtvm4j_runtime_packed.so
ld.lld: error: model_lib/libmodel_android.a(._llama_q4f32_1_devc.o): not an ELF file
clang++: error: linker command failed with exit code 1 (use -v to see invocation)
make[3]: *** [CMakeFiles/tvm4j_runtime_packed.dir/build.make:104: libtvm4j_runtime_packed.so] Error 1
make[2]: *** [CMakeFiles/Makefile2:268: CMakeFiles/tvm4j_runtime_packed.dir/all] Error 2
make[1]: *** [CMakeFiles/Makefile2:275: CMakeFiles/tvm4j_runtime_packed.dir/rule] Error 2
make: *** [Makefile:203: tvm4j_runtime_packed] Error 2

Llama-2-7b-chat-hf-q4f16_1-android.tar

[100%] Building CXX object CMakeFiles/tvm4j_runtime_packed.dir/mnt/c/AndroidProjects/MLC_AI/mlc-llm/3rdparty/tvm/jvm/native/src/main/native/org_apache_tvm_native_c_api.cc.o
[100%] Linking CXX shared library libtvm4j_runtime_packed.so
ld.lld: error: model_lib/libmodel_android.a(._llama_q4f16_1_devc.o): not an ELF file
clang++: error: linker command failed with exit code 1 (use -v to see invocation)
make[3]: *** [CMakeFiles/tvm4j_runtime_packed.dir/build.make:104: libtvm4j_runtime_packed.so] Error 1
make[2]: *** [CMakeFiles/Makefile2:268: CMakeFiles/tvm4j_runtime_packed.dir/all] Error 2
make[1]: *** [CMakeFiles/Makefile2:275: CMakeFiles/tvm4j_runtime_packed.dir/rule] Error 2
make: *** [Makefile:203: tvm4j_runtime_packed] Error 2

RedPajama-INCITE-Chat-3B-v1-q4f16_1-android.tar

[100%] Built target mlc_llm_static
[100%] Building CXX object CMakeFiles/tvm4j_runtime_packed.dir/mnt/c/AndroidProjects/MLC_AI/mlc-llm/3rdparty/tvm/jvm/native/src/main/native/org_apache_tvm_native_c_api.cc.o
[100%] Linking CXX shared library libtvm4j_runtime_packed.so
ld.lld: error: model_lib/libmodel_android.a(._lib0.o): not an ELF file
clang++: error: linker command failed with exit code 1 (use -v to see invocation)
make[3]: *** [CMakeFiles/tvm4j_runtime_packed.dir/build.make:104: libtvm4j_runtime_packed.so] Error 1
make[2]: *** [CMakeFiles/Makefile2:268: CMakeFiles/tvm4j_runtime_packed.dir/all] Error 2
make[1]: *** [CMakeFiles/Makefile2:275: CMakeFiles/tvm4j_runtime_packed.dir/rule] Error 2
make: *** [Makefile:203: tvm4j_runtime_packed] Error 2