Does OmniGen support changing the sdxl model?

[XueniLuo]

Hii, Thanks for this stunning work!
If I understood the paper in a proper way...
Since the SDXL and VAE part is fixed, only the transformer is trained. Is it possible to have a solely transformer conditioning Comfyui node, thus support common checkpoint loader and KSampler nodes in OmniGen workflow. In this way, we are allowed to change different SDXL checkpoints, even support flux checkpoint?

[set-soft]

OmniGen is derived from Phi3, which is an LLM model (technically an SLM).
Derives from the one that has vision included https://huggingface.co/microsoft/Phi-3-vision-128k-instruct
The fact that uses the SDXL VAE doesn't make it even similar to SDXL.
I created some nodes that separate the functionality: https://github.com/set-soft/ComfyUI_OmniGen_Nodes/
You can use the VAE loader and decoder from core nodes, and internally I use the VAE class from ComfyUI to encode the input images.