---
name: controllable-image-model-integration
description: Integrate advanced controllable image models whose point, box, sketch, reference, and semantic-layer capabilities may be transported through prompts and images rather than dedicated API fields.
version: 1.0.0
author: Hermes Agent
license: MIT
metadata:
  hermes:
    tags: [image-editing, grounding, coordinates, layers, multi-reference, api-integration, canvas]
    related_skills: [fal-studio-app-patterns, multi-provider-api-resilience, image-model-provider-evaluation]
---

# Controllable Image Model Integration

## Overview

Use this skill when building or auditing an interface for an image model that supports spatial grounding, marked-image editing, multi-reference composition, semantic layer separation, or iterative design edits.

The central rule is to separate three contracts:

1. **Model semantics** — what the upstream model owner documents.
2. **Provider transport** — what a wrapper endpoint accepts and returns.
3. **Runtime evidence** — what a cost-bounded canary actually produces.

A wrapper may transport advanced behavior through an ordinary `prompt` plus ordered image array. The lack of a dedicated schema field is not proof that the model capability is absent. Conversely, product marketing is not proof of a stable machine contract. Reconcile both with a canary.

## Trigger conditions

Load this skill for tasks involving any of:

- Click-to-edit, point, anchor, bounding-box, or coordinate-grounded editing
- Lasso, arrow, colored-box, doodle, or freehand-sketch guidance
- Multi-image reference fusion with role-sensitive image ordering
- Semantic multi-layer output with transparent assets
- Layer-aware browser canvases or iterative image-editing chat interfaces
- Provider wrappers whose visible OpenAPI schema appears narrower than upstream model claims

## Evidence hierarchy

Collect evidence in this order:

1. Owner-authored model documentation and examples
2. Exact provider endpoint schema and current pricing
3. Provider product page, examples, share records, and demo media
4. A minimal paid canary with an explicit maximum cost
5. Third-party guides, clearly labeled as secondary evidence

Do not collapse evidence classes. Report, for example:

- “Owner docs define this coordinate syntax.”
- “The wrapper accepts the required free-form prompt and ordered images.”
- “The wrapper does not document a typed field.”
- “The behavior has/has not been canary-proven on this deployment.”

## Spatial-control decision tree

### Typed coordinate syntax exists upstream

- Convert browser selections into the owner-documented normalized coordinate system.
- Insert the exact point/box tokens into the final model prompt.
- Keep image numbering stable and visible in the UI.
- Preserve the unmodified source image.
- Show the generated grounding tokens in a developer preview or annotation chip so the interaction is auditable.

A point or box drawn only for display, without corresponding model input, is UI theater.

### No typed lasso/polygon syntax exists

Use visual grounding:

1. Draw marks into a copy of the source image.
2. Send the marked raster as an ordered model input.
3. Describe each visible mark in natural language.
4. Tell the model to remove all annotation strokes from the result.
5. Keep the clean original available for retries and comparison.

Point/box tokens and raster annotations can be combined, but do not rasterize everything by default when a more exact owner-documented token exists.

## Multi-reference discipline

- Treat image order as part of the API contract.
- Label inputs consistently as `Image 1`, `Image 2`, or the provider’s documented equivalent.
- Give each reference an explicit role in the prompt.
- Never silently reorder uploads during compression, storage, or retry.
- Include reference-image fees in the spend cap when pricing is per output.
- Preserve source dimensions and image-to-prompt label mappings in persisted job metadata.

## Semantic layer experiments

Semantic decomposition is not ordinary background removal.

A good layer prompt specifies:

- The exact useful semantic pieces
- Full-canvas registration and identical dimensions
- Back-to-front order
- Genuine transparent alpha outside isolated elements
- A clean inpainted background where foreground content was removed
- Preservation of shadows, reflections, soft edges, and occlusion boundaries
- No contact sheet, crops, labels, masks, or flattened-only output

Do not assume a provider’s “number of images” parameter is the layer count. It may mean independent rerolls. Verify the endpoint definition and keep variant count separate from semantic layer count.

## Cost-bounded canary procedure

1. Re-check current provider pricing and input-reference surcharges.
2. Use the lowest adequate resolution.
3. Select a simple source with visually separable elements.
4. Request three or four named layers before trying ten or more.
5. Use one provider generation unless the wrapper explicitly documents a layer-count field.
6. Require explicit approval against a conservative upper bound.
7. Save a sanitized response-shape fixture without keys or signed sensitive data.
8. Download and inspect every returned asset before coding a parser around assumptions.

For per-output pricing with additional-input fees, a conservative formula is:

```text
(base_output_price + max(0, input_count - 1) * additional_input_price) * possible_output_count
```

## Output verification

For every provider output, in original order:

1. Download immediately to durable authenticated storage.
2. Preserve original bytes separately from user transforms.
3. Decode the real file; do not trust extensions or response labels.
4. Record format, dimensions, and alpha-channel presence.
5. Scan alpha values; an alpha channel alone does not prove transparency.
6. Mark `alpha_min < 255` as evidence that at least one pixel is not fully opaque.
7. Verify dimensions and registration match before stacking.
8. Label opaque results as candidates rather than validated transparent layers.

Recommended metadata:

```text
order, filename, width, height, format, has_alpha,
alpha_min, alpha_max, alpha_mean, transparent,
provider_request_id_present, response_order_preserved
```

## Lightweight interface pattern

Use four coupled surfaces:

- **Canvas:** composition plus select, point, box, lasso, and sketch tools
- **Composer:** concise chat-like instruction history and prompt shortcuts
- **Reference strip:** ordered source and donor images
- **Layer stack:** visibility, reorder, drag, scale, opacity, original download, composite export

Useful safeguards and affordances:

- Offline synthetic transparent-layer demo for zero-cost UI testing
- Explicit cost-cap checkbox immediately before submission
- Developer preview of normalized point/box tokens and sanitized response metadata
- “Use current composite as next source” for iterative edits
- Same-origin authenticated asset serving to avoid tainted canvas export
- Server-side credentials only
- Job states that survive page refresh or process restart

## Verification gates

Before claiming the app is done:

- Unit tests cover coordinate normalization, prompt assembly, cost bounds, response normalization, alpha inspection, and safe asset paths.
- Production build succeeds.
- Local login, upload, offline demo, canvas transforms, and export are exercised.
- Provider readiness is distinguished from a paid generation test.
- PM2/process manager owns the expected port.
- Reverse proxy configuration validates before reload.
- Public HTTPS, app authentication, health endpoint, and mobile layout are exercised.
- A real provider canary is run only with an approved spend cap.

## Pitfalls

- **Schema-field tunnel vision:** absence of a dedicated field does not negate prompt-driven owner-documented behavior.
- **Marketing overreach:** a demo does not define request JSON, response order, billing, or metadata.
- **Variant/layer confusion:** never map requested layer count directly to a provider’s independent-generation count without evidence.
- **Fake click controls:** visible marks must become coordinate tokens or marked-image inputs.
- **Flattening the only source:** always preserve clean source and returned originals.
- **PNG assumption:** PNG can be fully opaque; inspect decoded alpha.
- **Generic segmentation substitution:** never silently replace semantic design decomposition with foreground/background removal.
- **Understated spend:** include each additional reference-image fee for every possible output.
- **Premature deployment claim:** report a live app only after the public URL and complete representative flow are verified.

## References

- `references/seedream-5-pro-fal-contract.md` — provider-specific Seedream 5 Pro coordinate, marked-raster, layer, pricing, and canary notes.