image-model-evaluation

Image Model Evaluation

Use this skill when comparing image models, translating a reference into a new visual treatment, testing image-edit fidelity, or isolating a generated subject for downstream design.

Operating principles

1. Prove which model ran. Record the provider and model returned by each generation call. Never infer the backend from intent, configuration, or a registry recommendation.
2. Hold the creative brief constant. Use the same prompt, source image, composition request, exclusions, and output intent across models.
3. Separate generation quality from editing fidelity. A model that invents the strongest image may be a poor preservation-focused editor.
4. Inspect actual files. Verify pixel dimensions, mode, and alpha channel; response metadata can describe the request rather than the returned pixels.
5. Do not hide post-processing. Label deterministic matting, cropping, or compositing separately from model-native output.

Fair generation bakeoff

1. Analyze the source or desired style and list its invariants: focal subject, pose, topology, line systems, background geometry, palette, texture, and cropping.
2. Select current models from live provider schemas. Recency is a soft signal; capability fit and real execution matter more.
3. Write one shared prompt with three layers: - Preserve: subject identity, pose, symbolic objects, composition. - Transform: medium, palette, line quality, material, period, contrast. - Prevent: unwanted text, extra limbs, malformed hands, forbidden colors, modern gloss.
4. Use comparable quality tiers where possible, but report tier differences honestly.
5. Save outputs locally with provider/model identifiers in filenames.
6. Re-open every output and inspect anatomy, attribute accuracy, prompt adherence, stray typography/pseudoglyphs, palette, and artifacts.
7. Rank by use case rather than declaring a universal winner.

When useful, include both reference-guided edits and a text-only reinterpretation. This reveals the difference between source fidelity and independent invention.

Editing fidelity benchmark

Use the exact same source and extraction/edit brief for each model. Explicitly define:

• Keep: face, expression, pose, exact limb count and arrangement, clothing, held objects, proportions, linework.
• Remove: scenery, rays, portals, roots, clouds, floor, globe, shadows, text, and all environmental layers.
• Do not redesign: no added, removed, or rearranged limbs/objects; reconstruct only tiny occluded edges required by removal.
• Output field: uniform pure white if native alpha is not supported.

Judge preservation before beauty:

• identity and face drift
• limb/gesture drift
• moved or redesigned objects
• silhouette and pose changes
• altered line density or rendering style
• residual background
• damaged edges or halos

A beautiful redraw can be a worse edit than a less dramatic but faithful extraction.

Transparent isolation workflow

Do not claim transparency unless the file actually contains alpha.

1. Check file mode and alpha extrema.
2. If the edit endpoint only returns opaque RGB, request a uniform pure-white field.
3. Compare native model edits before post-processing.
4. Convert the selected white field to RGBA deterministically.
5. Render a contrasting checkerboard preview.
6. Inspect fine edges, enclosed holes, halos, missing pale details, and residual texture.
7. Confirm alpha includes 0 and 255, then record the subject bounding box.

Use scripts/white_to_alpha.py for a reproducible white-to-alpha conversion and checkerboard QA preview. Tune thresholds for background texture while protecting pale subject highlights.

Verification checklist

• Returned provider/model captured for every run
• Same prompt/source used across compared models
• Actual dimensions inspected from image bytes
• File mode and alpha inspected
• Output visually re-opened after generation
• Identity, anatomy, held objects, text artifacts, and edges checked
• Native model output kept distinct from deterministic post-processing
• Fresh capability/pricing claims tied to live provider evidence

Reporting format

For each model, report:

• provider and exact deployment/tier
• source modality and requested output shape
• actual dimensions and mode
• strongest qualities
• preservation or prompt-adherence defects
• best-fit use case

Conclude with a workflow recommendation such as “Model A for original generation, Model B for preservation-focused editing, deterministic alpha removal for delivery.”

References

• references/provider-observations-2026-07.md — session-backed GPT Image 2, Seedream 5 Pro Edit, and Grok comparison notes; re-verify before reuse.
• scripts/white_to_alpha.py — deterministic near-white background matting with JSON metrics and optional checkerboard preview.

Pitfalls

• Do not silently substitute a different model when the user names one.
• Do not trust a result size field as proof of actual dimensions.
• Do not treat an opaque white background as transparency.
• Do not compare models with different prompts and call it a model benchmark.
• Do not score isolation solely on attractiveness; inspect source preservation.
• Do not conclude an endpoint is absent solely because a shared registry search returned zero; verify the provider's live docs/OpenAPI.