Basic initial Comfy integration gives far better/accurate results than default github loadout for Gradio App

[AdamMinder]

Beyond some weirdness with downloading the models in a temp folder that i had to move manually, the github install went fine, no errors.

I initially just downloaded the simple All-in-one safetensors file and the basic workflow shared by comfy to use in their nightly ui update, and ran some tests, was pretty blown away by how well it worked.

Then i saw all the additional options like retake/edit etc in the gradio and installed the github. I get completely different results, mostly ones that seem to accept the lyrics written but ignores the tags and music styles.

So, question is, what are the right settings here? Comfy uses simple ksampler with default res_multistep + 4.0 cfg and a sd3 flow shifter in the workflow. Whatever settings under the hood there, how do they translate to this app?

I haven't tried every combo yet (ive tried changing to cfg/star/messed with guidance scales) but i can clearly say the default settings arent working the same. Either it's bugged or can someone enlighten me to the right settings to use?

Edit: just to clarify, i get music outputs (so no functional issues on my windows 12.6 cuda setup), it's just prompt adherence seems to barely work if it all when it comes to music type, whereas the comfy setup, as limited as it is, follows my tags much more closely to the point where i wasn't really questioning if it was working. If i tried Gradio as my first experience with this i'd have probably been unaware and discarded it.

[ChuxiJ]

Hi, Adam

Thank you for bringing this to our attention! We’ve primarily been testing and debugging through Gradio, so we weren’t aware of the performance gap compared to ComfyUI. If ComfyUI delivers significantly better results (especially in prompt adherence for music styles/tags), this is absolutely worth investigating—we really appreciate you catching this!

Next Steps for Investigation:

1. Parameter Alignment – We’ll compare the default settings between the two interfaces (e.g., sampler type, CFG scale, res_multistep, and SD3 flow shifter usage) to identify discrepancies.
   Prompt Processing – Verify if lyrics/tags are being parsed or weighted differently in Gradio (e.g., tag stripping or unintended prioritization of lyrics).

2. Workflow Translation – Attempt to replicate ComfyUI’s "All-in-one safetensors + basic workflow" setup within Gradio to isolate the cause.

3. Backend Differences – Check for version mismatches in dependencies (e.g., diffusers, transformers) or model-loading logic.
   We’ll prioritize this and keep you updated—any fixes or optimizations will be pushed immediately. If you have additional details (e.g., specific lyrics/tags examples where Gradio failed), those would be incredibly helpful!

Thanks again for your sharp eye and constructive feedback. We’ll follow up soon.

[AdamMinder]

Well the thing that initially clued me in to there being an issue was i had a hard time getting a female voice when using the female voice tag. Had no issues in the comfy tests and then i loaded it up in gradio and i was getting either male voices or distorted female voices. Also the sound of the music also felt distinctly off, like the flow and beats weren't quite as natural as what comfy was outputting. Harder to describe otherwise because there is just a fundamental difference in the settings used between the two, so first issue is the results will always be different and then next issue is basically seeing how statistically off it is compared to the other.

I think as a start it's definitely worth investigating to see what's in the workflow and how that differs from what you chose for the default loadout.

[ChuxiJ]

Thanks again for your detailed feedback—it’s been incredibly helpful for our investigation. Here’s what we’ve found so far regarding the differences between the Gradio and ComfyUI versions:

Key Discovered Differences

1. SD3 Flow Shifter Strength

   • Training Default: 3.0
   • ComfyUI Node Setting: 4.0 (This could explain the more natural flow/beats you observed in ComfyUI.)

2. Gradio’s APG Inference & Text/Lyric Weights

   • Default weights: text_cfg=15, lyric_cfg=15 (This might over-prioritize lyrics/tags, leading to less musical output.)
   • Suggested Adjustment: Try lowering these for finer control:
     • text_cfg: 5–7 (for better tag adherence)
     • lyric_cfg: 1.5–2 (to reduce lyrical dominance)

3. Random Seed & Duration Sensitivity

   • The model is highly sensitive to random_seed and audio_duration. Even slight differences here can drastically alter outputs. (Did you use identical seeds/duration between Gradio and ComfyUI?)

4. KSampler Performance

   • ComfyUI’s KSampler might indeed be more optimized for inference, but we haven’t fully validated this yet. (Your observation about its superior results is a strong signal!)

Request for Debugging Help

To replicate and fix these issues, it’d be immensely helpful if you could share the exact parameters from your ComfyUI tests, such as:

• audio_duration
• random_seed
• tags & lyrics (the specific prompts where Gradio failed)
• cfg_guidance (or other non-default settings you tweaked)

This will let us A/B test rigorously and align Gradio’s behavior with ComfyUI’s better performance.

Let us know if you’re able to share these details—we’re committed to resolving this and will keep you updated on progress!

[AdamMinder]

Yeah, comparisons are hard to test because the settings are very different at a base level between the two interfaces. So one of the initial goals of posting the issue was to get an understanding of what are 1 to 1 equivalent settings would be between the two. Your above post helps a bit with this, so I'll try shifting some cfg around to see if i can get comparable results, because obviously seed/steps/duration/prompt isnt enough to create matching results as you have separate cfg controls and additional modifier checkboxes and different samplers.

Can you say what the equivalent to the cfg 4 setting in comfy is? Is that changing just the global guidance from 15 to 4?

[ChuxiJ]

Here are my test results showing comparable quality between both versions:

Test Parameters

Audio Duration: 202.2 seconds
Tags: alternative rock, pop, rock
Seed: 296772417648513
Inference Steps: 50

Lyrics Input:

[verse]
Bright lights flashing in the city sky
Running fast and we don't know why
Electric nights got our hearts on fire
Chasing dreams we'll never tire

[verse]
Grit in our eyes wind in our hair
Breaking rules we don't even care
Shouting loud above the crowd
Living life like we're unbowed

[chorus]
Running wild in the night so free
Feel the beat pumping endlessly
Hearts collide in the midnight air
We belong we don't have a care

[verse]
Piercing through like a lightning strike
Every moment feels like a hike
Daring bold never backing down
Kings and queens without a crown

[chorus]
Running wild in the night so free
Feel the beat pumping endlessly
Hearts collide in the midnight air
We belong we don't have a care

[bridge]
Close your eyes let your spirit soar
We are the ones who wanted more
Breaking chains of the mundane
In this world we'll make our claim

Configuration Differences:

Parameter	Gradio Version	ComfyUI Version
Sampler	APG	CFG + KSampler
Guidance Scale	15	4.0
Flow Shift	3.0	4.0

Observations:

1. Vocal Quality:
   ComfyUI shows slightly worse vocal results, likely due to:

   • Higher effective guidance scale (CFG=4.0 vs APG's adaptive scaling)
   • Lack of guidance interval optimization

2. APG Advantages:
   As shown in the APG paper, our implementation:

   • Better handles high guidance scales
   • Uses guidance intervals to reduce artifacts
   • Maintains stability at scale=15 where traditional CFG might degrade

Audio Samples:

Attached comparison: gradio_vs_comfyui.zip

Would appreciate your thoughts on these findings - particularly whether the vocal quality difference aligns with your expectations given the parameter differences.

[ChuxiJ]

Finally, I found the key difference is that the ComfyUI use the res_multistep as default sample scheduler

https://github.com/comfyanonymous/ComfyUI/blob/c7c025b8d16f7f34b01409ead4dba4476cc64dae/comfy/k_diffusion/sampling.py#L1270

[kunibald413]

@ChuxiJ nice find! If it's "better" or not is then up for evaluation? and would it be easy to support it in the ace repo?

Edit: nvm found it #63

[sdbds]

comfyui clearly uses CPU random seed, so the result is likely to be different

[AdamMinder]

I've done further testing today and I think i've found where the difference is. Still really hard to explain the difference beyond my perception being in comfy the vocals sound naturally sharper, less post-processed, and more distinct with more ups and downs in the music flow, with 'maybe' better adherence to music styles in the tags. (again, given its impossible to create matching settings here and we're dealing with music listening, something a bit subjective, the issue is compounded by the randomness)

I think it comes down to the settings in the advanced tab. I've had the most luck getting close to comfy sound by adjusting the 'guidance interval'. Processing time is increased when changing the interval from .5 to 1, and the inference step processing time more closely matches that of comfy (around 20 seconds for me). With a 0.5, processing is about 6 seconds quicker because it only seems to be applying guidance only some of the time, as the description says. Voices sound more muffled and post-processed at 0.5, whereas with a max 1, everything is sharper and the song flow seems to have better progression.

I don't think this is a sampler or scheduler issue, although given i have tried many different ones in comfy, they can alter or improve a run on a seed in many ways, so having those options in gradio would be good. Obviously some scamplers produce noise, but others work well, although like i mentioned in my original post, res_multi is being used in comfy, and eular/huen is only available in the gradio, along with comfy using the 'simple' scheduler whereas i cant see any option for that choice in gradio. Ive tried matching the settings in comfy as close as i can to the gradio and Im still hearing a difference. I dont know what scheduler gradio is using.

I'd also like to know what comfy is using for these settings, guidance interval, granularity scale, min guidance scale, and the 3 checks for ERG for tag/lyric/diffusion. Overnight comfy added a separate weight for the lyrics so im looking at that to improve some word skipping. Im assuming comfy is at 1 on the interval, but the other settings i have no idea. Is the min guidance the flow shift from comfy?

Hopefully this helps, again there is placebo happening with the randomness on some of these observations, but im fairly confident the interval setting is the primary key difference here.

And as for testing settings, ive been using the stupid fennic girl song with the default tags from the basic comfy workflow that was provided yesterday. I started at seed 100, cfg 4, changed sampler to huen/eular in comfy, 4 shift (tried 3 shift too, there is some slight difference, but not a big one happening with shift), and used 120 duration. In gradio i initially set cfg with 4 guidance scale and 1 lyric weight, but then changed this to 15 guidance with apg (which combined with 1 guidance interval made a decent improvement)

Edit: i am still having an issue with male voices appearing in gradio when i prompt female however. I have yet to encounter this problem in Comfyui.

For example using the tags 'female, pure, pop, inspirational' almost always produces male voices in gradio using APG 15 guidance, Heun, 1 lyric scale, 3 ERG checked, 0.8-1 interval, 20 granular scale, 0 decay, 3 min guidance scale, 70 steps. Same lyrics as comfy, same seeds (although that doesnt matter). Comfy is 4 cfg, 4 shift, 70 steps, huen/simple, 1 lyric weight.

I almost always get a male voice in gradio, and a female voice in comfy

[rkfg]

While the accuracy might be better in CUI, I really don't like how the singer voice sounds. It's too loud, the music is either missing during the vocal part or it's so quiet it's impossible to hear. And the vocals sound like 32kbps MP3, it's that kind of artifact which is hard to explain but it's pretty consistent. It's like a flanger wah-wah effect, it might be subtle but it's present in every generation I tried. Doesn't happen in the gradio app. I'll try playing with the parameters but the defaults are like that.

[AdamMinder]

While the accuracy might be better in CUI, I really don't like how the singer voice sounds. It's too loud, the music is either missing during the vocal part or it's so quiet it's impossible to hear. And the vocals sound like 32kbps MP3, it's that kind of artifact which is hard to explain but it's pretty consistent. It's like a flanger wah-wah effect, it might be subtle but it's present in every generation I tried. Doesn't happen in the gradio app. I'll try playing with the parameters but the defaults are like that.

Yeah as i said above i think the 'guidance interval' acts as a sort of sharpener for the voices/music flow. The granularity scale also has an impact that seems less obvious that im looking at now. Still not sure what impact these have on prompt adherence though.

Also increasing step count can help with lower quality sounding voices. Ive had instances where vocals that sound a bit rough/loud get smoothed out by running at 80-100 steps

[rkfg]

Yes, it's a good catch, however even at 1 guidance interval the vocals are still clear in the gradio app, I couldn't reproduce the same low-bitrate vocal quality that I get in CUI. The sound isn't muffled but it's like it's missing some important frequencies.

I tried 100 steps in CUI and the result was still mostly the same as with 50 steps. Granularity helps with missing lines/words, setting it to 20-30 helps to capture all the lyrics without rerolling the whole song. But it might sound less interesting too, structure-wise.

[AdamMinder]

Yes, it's a good catch, however even at 1 guidance interval the vocals are still clear in the gradio app, I couldn't reproduce the same low-bitrate vocal quality that I get in CUI. The sound isn't muffled but it's like it's missing some important frequencies.

I tried 100 steps in CUI and the result was still mostly the same as with 50 steps. Granularity helps with missing lines/words, setting it to 20-30 helps to capture all the lyrics without rerolling the whole song. But it might sound less interesting too, structure-wise.

Might be the APG as the dev said being better at higher guidance with less distortion. Im not detecting as huge a difference in bitrate as you are though. I have good headphones, noise canceling sony wh xm model. I can say comfy results do seem to produce 'louder' voices which can then showcase bad compression depending on the seed, but then again im having the issue of more muffled results with gradio which can also sometimes sound like low bitrate stuff.

At the end of the day this is clearly just a settings issue which is what i was after, now we just need to expose more settings in both apps to get parity.