Since the field of AI image generation is developing extremely quickly, IntraPaint was designed so that its image generation capabilities could easily be updated as new tools become available. Instead of directly handling image generation itself, IntraPaint includes several image generator modules that let it offload image generation tasks to another program running in the background. You will need to install one of those programs yourself before you can use IntraPaint's AI features. Fortunately, this is a fairly simple process, and you have several convenient options available.
If you just want to get it running as quickly as possible, skip to the Stability Matrix setup section and pick "Stable Diffusion WebUI reForge" when asked to choose between Stable Diffusion packages.
- Stable Diffusion generator options compared
- Stable Diffusion model files
- Installation and setup
- Extra feature setup
There are two popular programs available for running Stable Diffusion that IntraPaint can use for image generation: the Stable Diffusion WebUI, and ComfyUI. Both options are nearly equivalent when used with IntraPaint, but each provides a few minor exclusive features.
Each of these programs also comes with its own interface, which can be accessed through a web browser. These interfaces are less suited for a digital art workflow than IntraPaint, but are occasionally worth using for tasks like bulk image generation or loading extensions.
The Stable Diffusion WebUI is one of the first interfaces created for using Stable Diffusion to see widespread use. Several independent versions of the WebUI exist, with varying goals. IntraPaint supports the original Stable Diffusion WebUI (A1111), WebUI Forge, and WebUI reForge. Currently, WebUI reForge is the recommended version for use with IntraPaint.
 |
|---|
| The original Stable Diffusion WebUI interface. |
A brief comparison of the three WebUI options:
- The original (A1111) Stable Diffusion WebUI is much less efficient than the alternatives, and performs poorly on older graphics cards. It has the most powerful ControlNet capabilities, but it's also the only one where you need to manually install ControlNet as an extension.
- Stable Diffusion WebUI Forge comes with vastly improved performance compared to the original WebUI, but it also comes with a lot of experimental features that break extension support. IntraPaint cannot use ControlNet through WebUI Forge unless you downgrade it to an older version.
- Stable Diffusion WebUI reForge includes all the performance enhancements from WebUI Forge, without any of the experimental features that break compatibility.
- The "Interrogate" button on the Image generation panel that automatically generates a prompt by using AI image analysis is exclusive to the WebUI.
- The Saved Prompt Style window can only be accessed when using the WebUI generator.
- The LoRA option window is only capable of loading image previews when used with the WebUI generator.
- The ControlNet Panel provides extra controls for image resizing and VRAM management when used with the WebUI generator.
- The Stable Diffusion settings tab provides some WebUI-exclusive options that let you directly control how the generator caches and loads AI models.
- When using the WebUI generator with a dedicated inpainting model, that model will automatically be configured for improved inpainting. When using the ComfyUI generator, you need to check an option in the ComfyUI extras tab to properly load inpainting models.
- Options from the WebUI extras tab are exclusive to the WebUI image generator:
- Batch variation controls, which lets you set how similar different images within the same batch will be.
- Seed resize controls, which help you regenerate similar images at different sizes.
- The "Tiling" checkbox, which lets you generate images that can be seamlessly tiled.
- The "Restore faces" checkbox, which applies a secondary facial restoration AI to correct issues with face generation.
ComfyUI is a popular Stable Diffusion interface with complex and powerful node-based controls.
 |
|---|
| The ComfyUI interface. |
Its speed and efficiency are at least as good as WebUI Forge and reForge, if not slightly better. It is much more flexible than any of the WebUI options, but the learning curve is also much steeper.
- Only the ComfyUI generator supports queued image generation. If you attempt to generate images with the ComfyUI generator active and some other program is using ComfyUI for image generation, you will be placed in a queue, and the loading screen will tell you your position in that queue. When using the WebUI generator, image generation will immedately abort if you attempt it under the same circumstances.
- Options from the ComfyUI extras tab are exclusive to the ComfyUI image generator:
- The option to apply a specific config file when loading models, instead of selecting one automatically.
- Tiled VAE encoding/decoding controls, which can decrease the amount of memory needed for generating very large images.
- A "Clear Memory" button, that makes the ComfyUI generator unload cached AI models from memory and otherwise free up memory resources.
To use Stable Diffusion, you'll need at least one Stable Diffusion model file. These files (also called checkpoints), contain the data structures that Stable Diffusion uses to create images. A huge variety of fine-tuned models are available on sites like CivitAI and Hugging Face, each with its own strengths and weaknesses. See AI model selection for more information about model types and selecting image generation models.
If you want to use IntraPaint's ControlNet Panel, you will also need to download ControlNet model files. Each ControlNet model provides a new way to control image generation. The official ControlNet models for use with Stable Diffusion 1.5 model variants can be downloaded from this page on Hugging Face. More options, including ones with support for Stable Diffusion XL, are linked on the Stable Diffusion WebUI ControlNet extension GitHub wiki.
To install model files, it's usually enough to just copy them to the right folder. The exact folder you need varies depending on the generator you use and the way you install it. Exact locations for each option can be found below in the general installation instructions.
There are a few options available for setting up Stable Diffusion beyond just selecting WebUI or ComfyUI. In most cases you will probably want to use Stability Matrix.
Stability Matrix is an extremely useful tool that automates the Stable Diffusion installation process. Both ComfyUI and all three supported WebUI versions can be easily installed through Stability Matrix.
 |
|---|
| The main Stability Matrix interface. |
- First, download the appropriate version of Stability Matrix for your system:
- Extract and launch the downloaded file. On the first launch it will ask you to select a data directory, and ask you to select a package to install. If you aren't sure which to pick, Stable Diffusion WebUI reForge is the recommended option, but Stable Diffusion WebUI, Stable Diffusion WebUI Forge, and ComfyUI will all also work with IntraPaint.
- If you've already downloaded Stable Diffusion or ControlNet models, go to the Checkpoint Manager tab on the left. Click the ⋯ menu button on the upper right, and choose "Models Folder", and it will open the folder you need in the system file manager. Copy Stable Diffusion models into the "StableDiffusion" subfolder.
- Next add any ControlNet models you've downloaded. Although the folder from the last step does contain a ControlNet section, it doesn't seem to properly share ControlNet models from that folder with Stable Diffusion. The location where you should actually put ControlNet models within the Stability Matrix data folder depends on which package you chose:
- reForge:
Packages/Stable Diffusion WebUI reForge/models/ControlNet - Forge:
Packages/Stable Diffusion WebUI Forge/models/ControlNet - Original WebUI:
Packages/Stable Diffusion WebUI/models/ControlNet - ComfyUI:
Packages/ComfyUI/models/controlnet
- reForge:
- For WebUI packages only, you'll need to enable API access. Skip this step if you installed ComfyUI.
Click the settings button next to the WebUI package:
Scroll to the bottom of the list, add--apito the Extra Launch Arguments section, and click save.
- Click the Launch button under the Stable Diffusion package you installed. A log screen will open up, showing the status of the package as it starts up. This might take a couple of minutes the first time it runs, as it finishes the last few parts of the setup process.
- Once the "Open Web UI" button appears at the top of the window, Stable Diffusion is ready to use. If you launch IntraPaint on the same computer, it will automatically connect to Stable Diffusion. If IntraPaint is already running, you can also use the image generator selection window to activate the appropriate Stable Diffusion image generator.
Stable Diffusion WebUI Forge provides a one-click Windows installation package you can use to easily install the WebUI. If you use one of the older versions, this option supports all WebUI features, including ControlNet. This method won't work outside of Windows and requires a bit more file management, but it's still a good option if you don't need the extra features in Stability Matrix.
- Download the last stable package version here. There are several other options linked on the WebUI Forge README, but newer versions may cause compatibility issues.
- Extract the downloaded package into a new folder. If you're not using Windows 11, you may need to install 7-Zip to extract the compressed files.
- Within the new WebUI-Forge folder, open the
webui\webui-user.batfile in a text editor. Find theset COMMANDLINE_ARGS=line near the top of the file, and change it toset COMMANDLINE_ARGS=--api. Save and close the file. This step is needed to make the WebUI acceptimage generation requests from IntraPaint. - If you haven't already found at least one Stable Diffusion model file, do that now. Copy it into the WebUI-Forge folder under
webui\models\Stable-diffusion. - If you want to use ControlNet, put downloaded ControlNet models in the
webui\models\ControlNetfolder now. - Launch the
run.batfile in the WebUI-Forge folder to start Stable Diffusion. A terminal window will open and print startup information as the WebUI initializes. - Once you see
Running on local URL: http://0.0.0.0:7860in the terminal window, Stable Diffusion is ready to use. Restart IntraPaint, or activate the "Stable Diffusion WebUI API" generator in the image generator selection window to start using IntraPaint's AI image generation features.
ComfyUI also provides a pre-packaged Windows version you can download and run, if you'd rather not use Stability Matrix or the WebUI.
- Download ComfyUI_windows_portable_nvidia.7z from the most recent version listed on the ComfyUI GitHub release page.
- Extract the downloaded package into a new folder. If you're not using Windows 11, you may need to install 7-Zip to extract the compressed files.
- If you haven't already found at least one Stable Diffusion model file, do that now. Copy it into the ComfyUI folder under
ComfyUI\models\checkpoints. - If you want to use ControlNet, put downloaded ControlNet models in the
ComfyUI\models\controlnetfolder now. - Launch the run_nvidia_gpu.bat file in the ComfyUI folder to start Stable Diffusion. A terminal window will open and print startup information as Comfy initializes.
- Once you see
To see the GUI go to: http://127.0.0.1:8188in the terminal window, Stable Diffusion is ready to use. Restart IntraPaint, or activate the "Stable Diffusion ComfyUI API" generator in the image generator selection window to start using IntraPaint's AI image generation features.
This method requires a lot more technical knowledge than the previous ones, but it also provides the most flexibility and requires the least amount of storage space. This is only recommended if you've set up a Python virtual environment and debugged Python dependency issues before.
Because this option isn't recommended, IntraPaint won't provide a full guide, but here's some resources that should help:
- AUTOMATIC1111/stable-diffusion-webui: install on Windows using Anaconda
- Using Git to switch between WebUI versions
- ComfyUI Manual installation
A few features supported by IntraPaint require a bit more setup, depending on which approach you used to install Stable Diffusion.
ControlNet features in IntraPaint will only activate if IntraPaint detects installed ControlNet models. If you followed the instructions above, that should already be done. If you're using the original Stable Diffusion WebUI, you'll also need to follow a few extra steps to install the ControlNet extension.
- Start the Stable Diffusion WebUI. On the same computer running the WebUI, open a web browser and navigate to (http://127.0.0.1:7860/) to access the web interface.
- Click the "Extensions" tab near the top of the window. Within the Extensions tab, click the "Available" tab on the next row down, and click the "Load from" button. A long list of extensions should load.
- In the search box just above the extension list, enter "sd-webui-controlnet". An extension with that name should be listed, linking to (https://github.com/Mikubill/sd-webui-controlnet). Click the "Install" button next to that option, and wait for it to finish processing. This may take a few minutes.
- Once it finishes, restart the Stable Diffusion WebUI. When it starts up again, the ControlNet extension should be available.
All Stable Diffusion image generators support AI upscaling through basic upscaling models. All Stable Diffusion WebUI versions provide several preset options, which will automatically be downloaded the first time they're used. ComfyUI does not do this, so you'll need to download models yourself and place them in the ComfyUI/models/upscale_models directory. This Hugging Face Page links several of the more popular options. When available, IntraPaint will include these options in the scåle image window.
IntraPaint also supports the Ultimate SD Upscale script through both the WebUI and ComfyUI generators. This option uses Stable-Diffusion itself, optionally combined with a secondary upscaler and the ControlNet tile model, for extremely high quality AI upscaling. To use this option, you'll need to install the upscale script as an extension in the WebUI or ComfyUI.
The setup process is identical for all versions of the WebUI:
- Start the Stable Diffusion WebUI. On the same computer running the WebUI, open a web browser and navigate to (http://127.0.0.1:7860/) to access the web interface.
- Click the "Extensions" tab near the top of the window. Within the Extensions tab, click the "Available" tab on the next row down, and click the "Load from" button. A long list of extensions should load.
- In the search box just above the extension list, enter "Ultimate SD Upscale". An extension with that name should be listed, linking to (https://github.com/Coyote-A/ultimate-upscale-for-automatic1111.git). Click the "Install" button next to that option, and wait for it to finish processing.
- Once it finishes, restart the Stable Diffusion WebUI. When you open the image scaling window and select "Advanced AI upscaling", you should now see a checkbox that you can use to enable the Ultimate SD Upscale script.
- To install ComfyUI extensions, you'll first want to install the ComfyUI Manager.
- If you installed ComfyUI using Stability Matrix, click the "Extension" button (with the puzzle piece icon) within ComfyUI's entry on the packages tab. Search for "ComfyUI-Manager", select it from the list, and click the "Install" button at the bottom.
- If you didn't use Stability Matrix, follow the installation instructions on the ComfyUI-Manager GitHub page.
- Once you restart ComfyUI, you should now see a "Manager" button near the bottom right. Click it, and the ComfyUI Manager Menu will open.
- In the manager menu, click "Custom Nodes Manager", and a list of extensions should open.
- In the search box at the top, enter "UltimateSDUpscale", and the list should filter down to only one option. Click the "Install" button next to it and wait.
- Once the button text changes to "Restart Required", the installation is finished. Restart ComfyUI. When you open the image scaling window and select "Advanced AI upscaling", you should now see a checkbox that you can use to enable the Ultimate SD Upscale script.