[Feature Request] Support for bundled LoRA/embedding files

[efhosci]

Onetrainer recently added an option to simultaneously train embeddings and LoRAs (I think this is called pivotal tuning?) and bundle them into a single safetensor file. I believe the format for this was actually created a while ago (see the first link). I experimented a bit with training using this method and got good results, and it seems like it can be an effective method for multi-concept LoRAs.

In A1111/web UI these work as expected, just include "lora:loraname:1 embeddingname" in the prompt. However I can't find a way to activate the embedding in a bundled file in Stable Swarm or the ComfyUI backend, only the LoRA. Separate embedding files work when loaded with embed:embeddingname but bundled files are a lot more convenient for large multi-concept LoRAs. Would support for this be something that can be implemented by ComfyUI?

Ideally, bundled embeddings should only be "accessible" when the associated LoRA is active, which is how web UI appears to work from my testing. Or there may be a better way to avoid conflicts between identically named embeddings in multiple bundle files.

Some links for reference:
Original web UI pull request for the bundle format
OneTrainer pull request
Relevant section of the OneTrainer code

[mx]

As requested by mcmonkey, here's a couple example bundles. "bundle_example.safetensors" is a SD 1.5 lora, "bundle_example_sdxl.safetensors" is an SDXL lora. You can see the format, and how it corresponds to the existing 1.5/XL/etc. standards, just by enumerating the keys in the file.

It would be good to get this supported; it's available in both A1111 and SD.Next, and people are out there creating loras with bundled embeddings and they simply do not work in ComfyUI.
models.zip

[mcmonkey4eva]

Relevant commits from that example file and discord comments after
1c08bf3
e1c5281