Features/timestamp-embedding-parser

[RSMNYS]

Timestamp-embedding-parser

[github-actions]

MLCommons CLA bot All contributors have signed the MLCommons CLA ✍️ ✅

[sonarqubecloud]

Quality Gate passed

Issues
5 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarCloud

[freedomtan]

@mohitmundhragithub to share the code used to generate this embedding.

[mohitmundhragithub]

@RSMNYS, The function where the time step embedding is generated is shared here: https://github.com/mlcommons/submissions_mobile_v4.1/issues/6#issuecomment-2399205941

From QPM Tutorial package, if we go to this file: <qpm_package>\StableDiffusion1.5\model\example1\redefined_modules\diffusers\models\embeddings.py
get_timestep_embedding( ) function generates the time embeddings.

<qpm_package>\StableDiffusion1.5\model\example3\host_linux_target_android_without_native\qnn_model_execution.ipynb
Above function is called from the top level notebook: get_time_embedding( )

[RSMNYS]

@RSMNYS, The function where the time step embedding is generated is shared here: https://github.com/mlcommons/submissions_mobile_v4.1/issues/6#issuecomment-2399205941

From QPM Tutorial package, if we go to this file: <qpm_package>\StableDiffusion1.5\model\example1\redefined_modules\diffusers\models\embeddings.py
get_timestep_embedding( ) function generates the time embeddings.

<qpm_package>\StableDiffusion1.5\model\example3\host_linux_target_android_without_native\qnn_model_execution.ipynb
Above function is called from the top level notebook: get_time_embedding( )

@mohitmundhragithub Shown code is only for the embedding generation. And no operations from the cut part are shown. (fully connected layer, Logistic/Sigmoid activation, and mul operation.

[freedomtan]

@anhappdev to share the QPM Stable Diffusion Jupyter notebook with @RSMNYS

[freedomtan]

@AhmedTElthakeb to provide a small tflite with only 3 operations.

@mohitmundhragithub will try to compare tflite and onnx files.

[AhmedTElthakeb]

@AhmedTElthakeb to provide a small tflite with only 3 operations.
sdv15_time_emb_head_int8.zip

[RSMNYS]

Hi guys! So I've used the model prepared by Ahmed to generate the embedding file. Now all works good. Here the link to the colab: https://colab.research.google.com/drive/1LG_rC5dlx2CbW2ZF4EamOnCdwtwLQESv?usp=sharing to prepare the embedding file, and the link to the models and embedding file: https://drive.google.com/drive/folders/1CT6VUWwGaTw34Za6dTJE7ptms20NUKH3?usp=sharing

[freedomtan]

please use pickle for saving the embedding.

@mohitmundhragithub please help figure out the difference between the one generated by @RSMNYS and Q's.

[freedomtan]

once the pickle is done, please make the embedding part of the files to be downloaded for Stable Diffusion for the TFLite backend.

[RSMNYS]

please use .pkl for the save the embedding.

@mohitmundhragithub please help figure out the difference between the one generated by @RSMNYS and Q's.

here you can find 2 files in json format to be able to compare numbers at least: https://drive.google.com/drive/folders/1SO1akyvWd2uYz9Xf_u5OGBJLetuW6x5A?usp=sharing

[RSMNYS]

guys, I've updated the colab to generate the pkl file. Also adjusted embedding_utils code to parse this. Here you can find the pkl file: https://drive.google.com/file/d/1pDd5wZje1KbIS4JcWzhDx00aN8GmTBpc/view?usp=share_link

[RSMNYS]

once the pickle is done, please make the embedding part of the files to be downloaded for Stable Diffusion for the TFLite backend.

@anhappdev I think for this we only need to upload the new UNET model (without embedding operations), and our embedding file. Right? If yes, can you please help upload those 2 files to the storage, from where we download models and other assets:

https://drive.google.com/file/d/1pDd5wZje1KbIS4JcWzhDx00aN8GmTBpc/view?usp=share_link
https://drive.google.com/file/d/1Sf2lcRDjSfg9jgABWWbV5EeWEmXl5CsJ/view?usp=share_link

[anhappdev]

@anhappdev I think for this we only need to upload the new UNET model (without embedding operations), and our embedding file. Right? If yes, can you please help upload those 2 files to the storage, from where we download models and other assets:

https://drive.google.com/file/d/1pDd5wZje1KbIS4JcWzhDx00aN8GmTBpc/view?usp=share_link
https://drive.google.com/file/d/1Sf2lcRDjSfg9jgABWWbV5EeWEmXl5CsJ/view?usp=share_link

@RSMNYS Here is the URL for the 2 files you shared
(Please remember to update the checksum in the backend settings)
https://mobile.mlcommons-storage.org/app-resources/models/v4_1/tflite/timestep_embeddings_data.pkl
https://mobile.mlcommons-storage.org/app-resources/models/v4_1/tflite/sd_diffusion_model_dynamic.tflite

And other model files for reference:
https://mobile.mlcommons-storage.org/app-resources/models/v4_1/tflite/sd_decoder_dynamic.tflite
https://mobile.mlcommons-storage.org/app-resources/models/v4_1/tflite/sd_text_encoder_dynamic.tflite

[freedomtan]

Current noise scheduling for TFLite was derived from my C++ code, which in turn was from Keras CV's code

1. according to the Keras CV Stable Diffusion noise scheduler code and comments, it was derived from Hugging Face DDPM implementation
2. it seems to me what @mohitmundhragithub referred to is not a scheduling method, but a ODE solver.

[mohitmundhragithub]

The ts sequence values are different. Does that come from ODE solver?

[RSMNYS]

@RSMNYS Release: https://github.com/mlcommons/mobile_model_closed/releases/tag/alpha-tflite-v0.2

1. Time embedding head: Lin -> SiLU -> Lin [float32 model]
   sdv15_time_emb_head_LinSiluLin_fp32.tflite
2. UNET new time embedding input goes to SiLU layer [int8 model]
   sd_diffusion_full_model_dynamic_int8_mlperfv41_v2.tflite

Also here: mlcommons/mobile_model_closed#12 (comment)

Hi guys! The new models works good. But we have one issue with pickle file format. And the problem is it's hard to decode and parse in c++, we need to use some special libs to parse it (and I wan't able to find such), or manual parsing which is hard and complex to parse the pickle data and protocols. So what I did I saved the tilmestep and embeddings in the same format as we originally had from Mohit. So while we can have the embedding part in pkl file format, in the c++ code I still use the old bin file format.

So let's decide tomorrow if we need to stick to pkl or we can use the existing one .bin.

The code to save embedding you can see in colab:

https://colab.research.google.com/drive/1LG_rC5dlx2CbW2ZF4EamOnCdwtwLQESv#scrollTo=B6UTUZcQuj2f

[mohitmundhragithub]

we also tried at our end.
The output generated with the timestep sequences from AIMET and the embeddings generated using the script which Sergji shared, works fine and gives good accuracy.
we still want to try generating the accruacy score with the timestep sequences from tflite.

NB: small change is needed in the script to match to the pkl format that we use.

[freedomtan]

Let's check @mohitmundhragithub can share.

@RSMNYS to check if the tflite backend can use the "original" hard-coded embedding from Q.

which is supposed to be generated from

print("Loading scheduler")
scheduler = DPMSolverMultistepScheduler(beta_start=0.00085,
                                        beta_end=0.012,
                                        beta_schedule="scaled_linear",
                                        num_train_timesteps=1000)
scheduler.set_timesteps(config.diffusion_steps)
scheduler.config.prediction_type = 'epsilon'

[mohitmundhragithub]

Hi,

This is the script that we used to generate the pickle file. It contains couple of changes to generate the .pkl file in required format.
time_emeddings.py.txt

I was slightly wrong in the meeting. The qti backend cpp code doesn't consume the pkl file directly. The pkl file is consumed by the flatten.py file shared here: https://github.com/mlcommons/mobile_app_open/blob/submission-v4.1/mobile_back_qti/DLC/util/StableDiffusion/flatten.py#L147

This generates a bin file, which gets packed along with other model files, and is consumed by the qti backend along with the models.
@RSMNYS was also suggesting something similar in the meeting.

[freedomtan]

@anhappdev please help put the embedding file to the right place (Cloudflare?)
@mohitmundhragithub please share the "final" embedding binary file (so that we can double-check it)

@anhappdev please create 5.0 submission branch after the checking is done. And then we can fixed the file location, and then free the submission branch.

[anhappdev]

@RSMNYS Please change the base branch of this PR from master to submission-v4.1.

[anhappdev]

@RSMNYS Can you please create a public GitHub repo and upload all the relevant files there like I did in this one and use those GitHub links for the backend settings.

After we have a running code with final models / files, I will replace the GitHub links with the CloudFare one. This way I don't miss or upload wrong files.

[mohitmundhragithub]

@mohitmundhragithub please share the "final" embedding binary file (so that we can double-check it)

unet_time_step_embeddings_20.pkl.txt
timestep_steps_20_int32_embedding_1x1280_float32.bin.ts.txt

Sharing the .pkl files and the .bin files generated as described in the steps here:
#928 (comment)

Please note that i was unable to upload .pkl and .bin.ts files (github's restriction), so had to append .txt in the filenames. Please remove the .txt extension to use those.

[RSMNYS]

@mohitmundhragithub please share the "final" embedding binary file (so that we can double-check it)

unet_time_step_embeddings_20.pkl.txt timestep_steps_20_int32_embedding_1x1280_float32.bin.ts.txt

Sharing the .pkl files and the .bin files generated as described in the steps here: #928 (comment)

Please note that i was unable to upload .pkl and .bin.ts files (github's restriction), so had to append .txt in the filenames. Please remove the .txt extension to use those.

that works good!

[RSMNYS]

Hi guys! I can run stable diffusion on my android device (Samsung Galaxy S22), but at the end I have 0 as a result, however I see all the steps in logs. Checking why. Also, during the stable diffusion process we are not updating the progress, so it's hard to understand what is going on.

[mohitmundhragithub]

Hi guys! I can run stable diffusion on my android device (Samsung Galaxy S22), but at the end I have 0 as a result, however I see all the steps in logs. Checking why. Also, during the stable diffusion process we are not updating the progress, so it's hard to understand what is going on.

can you share the loadgen and logcat logs?

[freedomtan]

Let's

1. make sure this app works well (check the app log directory, @RSMNYS )
2. verify tflite files are the same with what @AhmedTElthakeb provided (https://github.com/mlcommons/mobile_model_closed/releases/tag/alpha-tflite-v0.3)
3. after we merge this, add tflite files, .ts, files to both the mobile_models file and the CloudFlare bucket (@anhappdev)
4. add .pickle and tflite files to the mobile_open repo (https://github.com/mlcommons/mobile_open/releases). @anhappdev
5. 5.0 submission branch

[RSMNYS]

Benchmark result.json
here is my benchmark result file. But this for performance only. I think we need the accuracy as well.

[sonarqubecloud]

Quality Gate passed

Issues
6 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud