image-model-provider-evaluation

Image Model Provider Evaluation

Use this skill when the user asks where to access an image model, which provider exposes the most capabilities, whether advertised features are available by API, which provider has the best transparency/layer support, or how to activate/fund direct provider access.

Core distinction

Always separate four layers of evidence:

1. Native/model capability — what the lab demonstrates or claims the model can do.
2. Official API contract — fields and outputs documented by the model owner.
3. Provider wrapper contract — exact request fields, limits, formats, outputs, and pricing exposed by each deployment.
4. Executed canary evidence — what a live request actually returned.

A model-family name does not imply equivalent deployments. Do not turn marketing claims into API claims without a schema or canary.

Conversely, do not conclude that an advertised native feature is nonexistent merely because one wrapper omits it. When the owner advertises the capability but an aggregator schema lacks the control/output contract, trace the feature through the owner tutorial, owner API reference, model-specific capability table, pricing semantics, SDK/sample code, and executed responses. Route that operation to the provider surface that actually exposes it; do not fake the feature by stuffing prose into an incompatible wrapper field.

Source hierarchy

Prefer, in order:

1. Official model-owner API reference and tutorials.
2. Provider OpenAPI/JSON schema or exact endpoint contract.
3. Provider pricing page and live quote endpoint.
4. Provider landing/marketing page.
5. Independent tests and community reports.

Record verification time and distinguish source-backed release dates from first observation.

Mine executed public examples before paying for a canary

When a provider publishes share links or example generations, inspect the underlying public request record—not only the rendered playground. Recover the exact request fields, output JSON, endpoint alias, timing, and status. This often reveals whether a marketed interaction is implemented by typed JSON, prompt syntax, or pixels baked into an annotated image.

For fal, public example records may be available at:

GET https://rest.alpha.fal.ai/requests/share/{share_id}

Treat this as executed-example evidence, below a canary run under the user's own account but above marketing copy. Do not infer private billing or undocumented guarantees from the record.

Evaluation workflow

1. Normalize the requested job

Identify whether the user needs: - text-to-image - single-image editing - multi-reference fusion - sequential/coherent sets - exact dimensions or maximum resolution - text/layout accuracy - transparent background - editable layer decomposition - seed/reproducibility - privacy or uncensored/private execution - downstream handoff to video or design tools

Do not treat transparent background and editable layer decomposition as the same feature.

2. Build an exact provider matrix

For every candidate deployment, capture: - exact provider and endpoint/model ID - generation vs edit endpoint split - input image field and maximum references - coherent/batch output versus independent variants - output count limits - size, aspect ratio, resolution, and format controls - seed, safety, prompt optimization, and search controls - output schema, URL retention, and delivery behavior - declared transparent output and typed layer metadata - pricing unit and all surcharges - credential availability in the current environment, without exposing values

Hard-filter on required capabilities before ranking price or recency.

3. Audit advanced features

Coordinate or region editing

A provider may implement this through prompt markup (<point>, <bbox>) or marks embedded in the source image rather than dedicated JSON fields. State whether support is official, passthrough-likely, or canary-only.

Search-assisted generation

A model landing page may claim web/online search while the API lacks a controllable search field, citations, or provenance. Do not promise grounded output unless the request contract and response evidence support it.

Layer separation and transparency

A PNG option is not proof of transparency. A generic images[] response is not proof of a structured layer package. Verify: - multiple returned files - alpha channel exists - alpha contains non-opaque pixels - background and element layers are identifiable/orderable - hidden regions are inpainted where promised - billing behavior per returned layer

If a wrapper returns one opaque composite from a prompt that requested layers, inspect the complete provider response before changing parsers. If the response truly contains only one image, the parser is not the fault. Compare the wrapper request against the owner contract: wrapper-specific output-count fields may force independent variants or a single-image path and can suppress a native prompt-directed multi-file behavior. Implement a capability-specific provider adapter when necessary, then require a bounded owner-route canary before declaring the integration fixed.

If the user only needs subject cutout/background removal, prefer a dedicated background-removal model after generation and verify alpha pixels.

4. Normalize pricing

Compare the same operation, dimensions/pixel area, number of references, output count, quality tier, and format. Include: - base output charge - pixel/resolution threshold changes - extra reference-image charges - per-layer/per-output billing - provider markup - free quota versus paid/postpaid activation

Do not call a provider cheapest from a headline “from” price.

5. Verify credentials safely

• Secrets belong in the active profile .env, mode 0600, not source, config files, memory, or skill content.
• Prefer non-billing authentication probes such as model listing or account metadata.
• Never print or repeat the key.
• If a key appears in a group/public thread, recommend rotation and a private replacement.
• Do not run a paid generation solely to test authentication if a non-billing endpoint suffices.

6. Handle billing/onboarding

Before telling a user to “add credits,” determine whether the provider uses: - prepaid wallet credits - postpaid card billing - coupons/free quota - resource packs - invoiced credit limits - bank transfer

Common pitfall: direct cloud providers often require account-profile completion, a default payment method, and explicit model activation rather than prepaid top-up.

7. Canary before productization

When a provider's marketing and schema diverge, run the smallest bounded canary only after disclosing expected cost. Save: - exact request - model ID - returned metadata/files - actual dimensions/formats - alpha/layer verification - latency - billed output count/cost if available

Recommendation format

Report: 1. Best overall official implementation 2. Best provider already accessible 3. Best value 4. Best for the user's exact capability 5. Capabilities not reliably exposed 6. Required canary before production

Use compact tables, but clearly label native capability, documented API support, marketing-only support, and live canary evidence.

Pitfalls

• Provider landing pages can describe the upstream model rather than the wrapper's actual contract.
• Do not react to that mismatch by declaring the native feature unavailable. Find the owner route or another deployment with a verified contract, and route only that operation there.
• Output-count fields are provider-specific: they may mean independent rerolls, coherent sequence members, or native layer files. Never map one meaning onto another by name alone.
• Extra-reference pricing can be per request on the owner API but per output on an aggregator that runs multiple generations; model it per provider and operation.
• PNG does not mean transparent.
• Independent variants are not coherent sequential generation.
• An array output does not necessarily carry layer semantics.
• Model-native web search without a request switch or citations is not auditable grounding.
• API-key validity does not prove paid model activation or available billing.
• A missing “top up” button may be correct for a postpaid account.
• Do not infer direct provider access from a key name in an example file; verify configured credential presence and, when appropriate, authenticate with a read-only endpoint.

Provider-specific references

• See references/seedream-5-provider-transparency-byteplus.md for the July 2026 Seedream 5 Pro/Lite provider matrix, transparency/layer contract analysis, BytePlus authentication probe, and postpaid activation workflow.