---
name: image-model-evaluation
description: Run fair, evidence-backed comparisons of image generation and editing models; verify actual provider/model routing, preserve reference identity, evaluate outputs, and produce validated transparent cutouts when needed.
version: 0.1.0
author: Hermes Agent
license: MIT
metadata:
  hermes:
    tags: [image-generation, image-editing, model-comparison, creative-qa, background-removal, transparency]
---

# Image Model Evaluation

Use this skill when comparing image models, translating a reference into a new visual treatment, testing image-edit fidelity, or isolating a generated subject for downstream design.

## Operating principles

1. **Prove which model ran.** Record the provider and model returned by each generation call. Never infer the backend from intent, configuration, or a registry recommendation.
2. **Hold the creative brief constant.** Use the same prompt, source image, composition request, exclusions, and output intent across models.
3. **Separate generation quality from editing fidelity.** A model that invents the strongest image may be a poor preservation-focused editor.
4. **Inspect actual files.** Verify pixel dimensions, mode, and alpha channel; response metadata can describe the request rather than the returned pixels.
5. **Do not hide post-processing.** Label deterministic matting, cropping, or compositing separately from model-native output.

## Fair generation bakeoff

1. Analyze the source or desired style and list its invariants: focal subject, pose, topology, line systems, background geometry, palette, texture, and cropping.
2. Select current models from live provider schemas. Recency is a soft signal; capability fit and real execution matter more.
3. Write one shared prompt with three layers:
   - **Preserve:** subject identity, pose, symbolic objects, composition.
   - **Transform:** medium, palette, line quality, material, period, contrast.
   - **Prevent:** unwanted text, extra limbs, malformed hands, forbidden colors, modern gloss.
4. Use comparable quality tiers where possible, but report tier differences honestly.
5. Save outputs locally with provider/model identifiers in filenames.
6. Re-open every output and inspect anatomy, attribute accuracy, prompt adherence, stray typography/pseudoglyphs, palette, and artifacts.
7. Rank by use case rather than declaring a universal winner.

When useful, include both reference-guided edits and a text-only reinterpretation. This reveals the difference between source fidelity and independent invention.

## Editing fidelity benchmark

Use the exact same source and extraction/edit brief for each model. Explicitly define:

- **Keep:** face, expression, pose, exact limb count and arrangement, clothing, held objects, proportions, linework.
- **Remove:** scenery, rays, portals, roots, clouds, floor, globe, shadows, text, and all environmental layers.
- **Do not redesign:** no added, removed, or rearranged limbs/objects; reconstruct only tiny occluded edges required by removal.
- **Output field:** uniform pure white if native alpha is not supported.

Judge preservation before beauty:

- identity and face drift
- limb/gesture drift
- moved or redesigned objects
- silhouette and pose changes
- altered line density or rendering style
- residual background
- damaged edges or halos

A beautiful redraw can be a worse edit than a less dramatic but faithful extraction.

## Transparent isolation workflow

Do not claim transparency unless the file actually contains alpha.

1. Check file mode and alpha extrema.
2. If the edit endpoint only returns opaque RGB, request a uniform pure-white field.
3. Compare native model edits before post-processing.
4. Convert the selected white field to RGBA deterministically.
5. Render a contrasting checkerboard preview.
6. Inspect fine edges, enclosed holes, halos, missing pale details, and residual texture.
7. Confirm alpha includes `0` and `255`, then record the subject bounding box.

Use `scripts/white_to_alpha.py` for a reproducible white-to-alpha conversion and checkerboard QA preview. Tune thresholds for background texture while protecting pale subject highlights.

## Verification checklist

- Returned provider/model captured for every run
- Same prompt/source used across compared models
- Actual dimensions inspected from image bytes
- File mode and alpha inspected
- Output visually re-opened after generation
- Identity, anatomy, held objects, text artifacts, and edges checked
- Native model output kept distinct from deterministic post-processing
- Fresh capability/pricing claims tied to live provider evidence

## Reporting format

For each model, report:

- provider and exact deployment/tier
- source modality and requested output shape
- actual dimensions and mode
- strongest qualities
- preservation or prompt-adherence defects
- best-fit use case

Conclude with a workflow recommendation such as “Model A for original generation, Model B for preservation-focused editing, deterministic alpha removal for delivery.”

## References

- `references/provider-observations-2026-07.md` — session-backed GPT Image 2, Seedream 5 Pro Edit, and Grok comparison notes; re-verify before reuse.
- `scripts/white_to_alpha.py` — deterministic near-white background matting with JSON metrics and optional checkerboard preview.

## Pitfalls

- Do not silently substitute a different model when the user names one.
- Do not trust a result `size` field as proof of actual dimensions.
- Do not treat an opaque white background as transparency.
- Do not compare models with different prompts and call it a model benchmark.
- Do not score isolation solely on attractiveness; inspect source preservation.
- Do not conclude an endpoint is absent solely because a shared registry search returned zero; verify the provider's live docs/OpenAPI.