Prompting

How to Prompt Emotion and Facial Expression in Stable Diffusion

PhoneDiffusion
How to Prompt Emotion and Facial Expression in Stable Diffusion

Prompts

To prompt emotion and facial expression in Stable Diffusion, name the emotion and describe what the face does (eyes, brows, mouth), put it near the start of the prompt, and frame the face close enough to show it. Keep the seed fixed while you test wording, and use inpainting on the face when you only want to change the expression.

How to prompt emotion: the short answer

A single mood word such as ‘sad’ is vague: the model can apply it to the lighting, colors or setting instead of the face. Name the emotion and then describe the face that shows it: ‘a woman laughing, eyes squeezed half shut, wide open-mouthed smile’ gives the model three visible details instead of one abstract word.

Then make the face big enough to carry that detail, put the expression early in the prompt, and change one thing at a time with the same seed. When the rest of the image is right and only the expression is wrong, repaint just the face instead of generating again.

Describe the face, not just the feeling

A useful starting vocabulary is the set of seven expressions that Paul Ekman's research describes as universal: anger, contempt, disgust, enjoyment, fear, sadness and surprise. For each one, add the parts of the face you would see. The Ekman Group's page for each emotion describes them: eyebrows down and together with glaring eyes for anger, a one-sided expression for contempt, a wrinkled nose for disgust, a smile with crow's feet at the eyes for real enjoyment, raised but straight eyebrows, upper eyelids lifted to show the whites of the eyes, and tensed, stretched lips for fear, inner eyebrow corners angled up for sadness, and raised curved eyebrows, wide eyes and a dropped jaw for surprise.

Add intensity words when you need them (a slight smile, a tearful face, screaming in anger) and context that supports the feeling, such as a pose, a gesture or a situation. The Diffusers prompting guide recommends a similar structure: start with the subject, then add style and context such as action, setting and mood, written as a description rather than a keyword list.

Avoid contradictions you did not mean. ‘Smiling, crying’ can work for tears of joy, but ‘angry, happy, calm’ asks for three faces at once, and the model may blend them or favor one.

Where to put the expression and how to strengthen it

Put the expression right after the subject. This is a practical habit: the AUTOMATIC1111 wiki does not say word order changes a word's weight. It does say Stable Diffusion usually accepts 75 tokens and that longer prompts are split into 75-token chunks, each processed separately, so an expression buried after a long list of style words can land in a different chunk from the subject it describes.

In AUTOMATIC1111 and with the Diffusers helper libraries you can also weight it. The AUTOMATIC1111 wiki documents (word) as 1.1 times the attention, ((word)) as 1.21, [word] as weaker, and (word:1.5) as an explicit weight. Diffusers does not read weights in the plain prompt string; it takes embeddings built by a helper library. sd_embed reads the (word) and (word:1.5) form, while Compel has its own syntax: word+ is 1.1, word- is 0.9, and a number goes after the closing parenthesis, as in (word)1.5. Start small, for example (crying:1.2), or (crying)1.2 in Compel, and raise it only if the expression still does not show.

Use the negative prompt for the expression you keep getting by accident. The wiki explains that the negative prompt replaces the empty prompt the sampler normally steers away from, so ‘smile’ in the negative prompt helps when every portrait comes out grinning but you wanted a serious face.

PhoneDiffusion generates on-device on iPhone or iPad, with optional cloud models. Download on the App Store →

Frame the face big enough

An expression lives in a few small features. In a full-body or wide shot the face may cover only a few dozen pixels, which leaves little room for a convincing frown. Ask for a close-up, a portrait or a head-and-shoulders shot when the emotion matters; if the face still comes out mangled, see how to fix distorted faces in Stable Diffusion.

Anime models: use expression tags

Anime checkpoints are usually trained on tag-style captions rather than sentences. The Animagine XL 4.0 model card recommends the order subject count (1girl, 1boy or 1other), character, series, rating, then everything else in any order, ending with quality tags, at CFG 4 to 7 and 25 to 28 steps. Expression words go in the ‘everything else’ part as short tags. The card does not list them; common examples are smile, open mouth, closed eyes, tears and blush.

Change only the expression with inpainting

When you like everything except the face, inpaint it. In inpainting the white part of the mask is repainted from your prompt and the black part is kept, so mask the face (or just the mouth and eyes), describe the new expression, and keep the rest of the prompt the same. The Diffusers inpainting guide explains that strength sets how much noise is added: lower values stay close to the original face, higher values allow a bigger change.

Generate a few variations, keep the best, and save its seed and settings so you can repeat the result; Stable Diffusion seeds explained covers how.

PhoneDiffusion availability and article scope

In PhoneDiffusion, write the expression in plain words near the start of the prompt. The app does not parse weight syntax such as (word:1.2) or word+, so parentheses, numbers and plus signs are not a way to strengthen a word there. Local generation has an optional negative prompt, and local inpainting lets you mask the face and redraw it with a Replacement strength setting. The curated catalog includes an Animagine XL 4 pack (on the most capable devices) for tag-style anime prompts. It is a faster 8-step build at 768 × 768, so the card's step and size advice does not apply there.

Local packs run on supported iPhone and iPad devices with iOS 17 or later, privately and offline once downloaded. Optional cloud models on paid monthly or yearly plans are text-to-image only and send the text prompt online, without a negative prompt.

Field notes

  • Name the emotion, then describe the eyes, brows and mouth that show it.
  • Put the expression right after the subject, not at the end of a long prompt.
  • Frame the face close enough for the expression to be visible.
  • Weights such as (crying:1.2) work in AUTOMATIC1111 and sd_embed; Compel writes (crying)1.2. Test small changes first.
  • Inpaint the face to change only the expression and keep the rest of the image.

Try it on your phone

  1. Write subject, emotion and two or three facial details, then style and context.
  2. Choose a close-up or portrait framing.
  3. Fix the seed and generate a first image.
  4. Change one word at a time, or add the unwanted expression to the negative prompt.
  5. If only the face is wrong, mask it and inpaint with the new expression.

Sources and verification

Primary and maintainer documentation checked for this guide. Product-specific details were reviewed against the current PhoneDiffusion scope.

  • Paul Ekman Group: Universal EmotionsResearch summary naming seven universal facial expressions of emotion: anger, contempt, disgust, enjoyment, fear, sadness and surprise. Its linked page for each emotion describes the face, such as eyebrows down and together for anger, a wrinkled nose for disgust and inner eyebrow corners angled up for sadness.
  • Hugging Face Diffusers: Prompting (guidance, negatives, prompt weighting)Maintainer guide: build prompts from subject, style and context; negative_prompt and guidance_scale on Stable Diffusion pipelines; prompt weighting through prompt_embeds, with sd_embed reading (cat), ((cat)) and (cat:1.5), or with Compel, which the guide describes as a simpler syntax.
  • AUTOMATIC1111 web UI Features wiki: attention/emphasis, prompt length, negative promptProject wiki: (word) raises attention by 1.1, ((word)) by 1.21, [word] lowers it by 1.1, (word:1.5) sets a weight; prompts are split into 75-token chunks; the negative prompt replaces the empty unconditional prompt during sampling.
  • Compel: prompt weighting syntax (doc/syntax.md)Maintainer doc: append + or - or a weight between 0 and 2; + is 1.1, ++ is 1.1², - is 0.9, -- is 0.9²; (picking apricots)1.3 sets a number for a group; nested modifiers stack; downweighting is a weighted blend with the prompt without the term; .blend() and .and() take quoted prompt parts.
  • Cagliostro Research Lab: Animagine XL 4.0 model card (prompt structure)Model card: tag-based prompts in the order 1girl/1boy/1other, character, series, rating, everything else, then quality tags; CFG 4 to 7 (5 recommended), 25 to 28 steps.
  • Hugging Face Diffusers: InpaintingMaintainer guide: the mask's white pixels are repainted from the prompt and black pixels are kept; strength sets how far the masked area moves from the original.