Start Here

What Is a VAE in Stable Diffusion?

PhoneDiffusion
What Is a VAE in Stable Diffusion?

Basics

A VAE (variational autoencoder) in Stable Diffusion is the part that converts between pixels and the small latent image the model works on. Its decoder turns the finished latent into the picture you see, and its encoder turns your photo into a latent for img2img and inpainting. A mismatched or missing VAE gives dull colors and soft details, so use the one the checkpoint recommends.

What a VAE does in Stable Diffusion

Stable Diffusion does not draw pixels directly. It denoises a compressed latent image, and the VAE translates between that latent and a normal image. The Stable Diffusion v1.5 model card describes the compression: the autoencoder shrinks each side by a factor of 8 and stores 4 channels, so a 512 × 512 image becomes a 64 × 64 × 4 latent.

That latent holds about 48 times fewer values than the 512 × 512 color image, so each denoising step has far less to compute. The cost is that the final look depends on how well the decoder rebuilds fine detail from it: textures, eyes, small text and color all pass through the VAE at the end.

Encoder and decoder: when each one runs

In text-to-image, only the decoder runs: the sampler finishes the latent and the VAE decodes it into the image. AUTOMATIC1111's settings describe the same split: for txt2img the VAE creates the image after sampling, and for img2img it also encodes your input image before sampling.

That is why img2img and inpainting can shift colors slightly: your image is encoded, changed and decoded again. Each extra pass repeats that round trip, so keep the same VAE across passes. The txt2img guide shows where VAE Decode sits in a full workflow.

Baked-in VAE vs a separate VAE file

A checkpoint file usually carries a VAE inside it (AUTOMATIC1111's ‘None’ option uses that one), but some creators recommend a separate VAE file. For Stable Diffusion 1.x, Stability AI's sd-vae-ft-mse is a fine-tuned decoder meant as a drop-in replacement for the original autoencoder. Its card says only the decoder was fine-tuned, on a mix of aesthetic and human images to improve faces, and that the MSE version gives somewhat smoother outputs.

SDXL ships its own VAE. The SDXL paper says it uses the same autoencoder architecture as the original Stable Diffusion but trained with a much larger batch size (256 instead of 9) and averaged weights, to improve local, high-frequency details. Use a VAE made for the same model family as the checkpoint: an SD 1.x VAE for SD 1.5 models and an SDXL VAE for SDXL models.

PhoneDiffusion generates on-device on iPhone or iPad, with optional cloud models. Download on the App Store →

How to choose and load a VAE

Start with the checkpoint's download page: if it names a VAE, use that one. In AUTOMATIC1111, put the file in models/VAE and pick it under Settings, SD VAE. ‘Automatic’ uses a VAE with the same file name as the checkpoint, and ‘None’ uses the VAE inside the checkpoint. In ComfyUI, add a Load VAE node and connect it to VAE Decode instead of the VAE output of Load Checkpoint. In Diffusers, load it with AutoencoderKL and pass it to the pipeline as vae.

To check the effect, generate the same prompt and seed with and without the VAE and compare colors, faces and small details at full size.

Signs of a VAE problem

Flat, grayish colors are the classic sign of a missing or mismatched VAE; why Stable Diffusion colors look washed out walks through that fix and the other causes. Mushy faces or smeared fine texture in an otherwise good image can also point to the decoder.

With SDXL on half precision (fp16), the original VAE can produce NaN values because its internal activations get too large. The SDXL-VAE-FP16-Fix card describes a fine-tuned version that keeps outputs close while avoiding this, and says AUTOMATIC1111's --no-half-vae workaround is then no longer needed. Large images can also run out of memory at the decode step: Diffusers' VAE tiling decodes in overlapping tiles, with possible slight tone differences between tiles but no obvious seams.

PhoneDiffusion availability and article scope

PhoneDiffusion runs curated Core ML model packs (SD 1.5, SD 2.1, and SDXL) on supported iPhone and iPad devices with iOS 17 or later; local generation is private and works offline once a pack is downloaded. Each local pack ships with its own VAE, so there is no VAE to choose or swap in the app, and no general production importer for arbitrary checkpoints or VAE files.

It also offers optional cloud models, such as FLUX and Seedream, on paid monthly or yearly plans; those are text-to-image only, and the text prompt is sent online to generate the image. The AUTOMATIC1111, ComfyUI and Diffusers steps above describe desktop and server tools.

Field notes

  • The VAE converts between pixels and the 8× smaller latent image Stable Diffusion works on.
  • txt2img only decodes; img2img and inpainting also encode your image first.
  • Use the VAE the checkpoint recommends, made for the same model family.
  • Dull, gray colors usually mean a missing or mismatched VAE.
  • SDXL in fp16 may need the FP16-Fix VAE; tiling helps large decodes fit in memory.

Try it on your phone

  1. Check the checkpoint's page for a recommended VAE.
  2. Load it: SD VAE in AUTOMATIC1111, Load VAE in ComfyUI, or vae= in Diffusers.
  3. Generate the same prompt and seed with and without it.
  4. Compare colors, faces and fine detail at full size.
  5. Keep the same VAE for every img2img or inpainting pass.

Sources and verification

Primary and maintainer documentation checked for this guide. Product-specific details were reviewed against the current PhoneDiffusion scope.