ai-video-story-pipeline

AI Video Story Pipeline

App Overview

• URL: video-story.apps.poofc.com
• Path: /home/avalon/apps/video-story
• Stack: React + Vite PWA (frontend), Express + SQLite (backend)
• PM2: video-story process

LLM Configuration

• Primary: OpenRouter (routes to Anthropic, better rate limits)
• Fallback: Direct Anthropic API
• Never use Anthropic as primary — causes 429 rate limits during batch operations

YOLO Pipeline (10 steps)

1. Voices — Assign voices to characters
2. Script — Generate script from story
3. Audio — Generate audio narration
4. Scenes — Break story into scenes
5. Shots — Break each scene into shots (per-scene retry, 3 attempts each)
6. Refs — Generate reference images for characters, sets, props
7. Frames — Generate frame images for each shot (uses FLUX.2)
8. Videos — Generate video clips from frames (uses WAN 2.2 via Replicate)
9. Lip Sync — Lip sync dialogue shots (Kling via Replicate, ~$0.014/sec)
10. Export — Assemble final video (prefers lip-synced clips when available)

Critical Conventions

Story Generation

• Story prompts MUST exclude character appearance details (hair, clothing, build, age, skin)
• Appearance is handled separately in the Analysis step's appearance field
• This prevents conflicts when user edits appearance in detail panel

Reference Images

• Set references: Must exclude all people/characters — empty environment only
• Prop references: Must exclude all people/characters — object only
• Character references: Character portrait with appearance details; PuLID used when guide images exist
• Frame generation uses ALL reference types: character + set + prop images (via scene_props relationship)
• FLUX.2 supports up to 8 reference images per generation

Guide Images (multi-upload system)

• guide_images table stores multiple guide photos per entity (entity_type, entity_id, image_url, sort_order)
• Legacy guide_image_url column on characters/sets/props stays in sync (first guide image)
• API: POST/GET /:entityType/:id/guide-images, DELETE /guide-images/:guideId
• Entity list endpoints return guide_images array attached to each entity via attachGuideImages() helper
• All generate endpoints (standard, advanced, YOLO bulk) query getGuideImageUrls() from guide_images table
• Characters with guides use PuLID (identity-preserving, 1 ref); sets/props pass all guides as FLUX refs
• Model maxRefs vary by provider reality, not preference: PuLID=1, Reve/Qwen/Kontext=1, Nano Banana=4, FLUX.2 Edit=9, Nano Banana 2/Pro=14. Qwen Pro Edit currently must be capped at 3 refs because Fal returned 422 “Maximum 3 reference images allowed”; do not raise it without live docs/testing.
• Guide images beyond model capacity are visually faded and not sent to API
• CRITICAL regeneration rule: advanced regeneration must use guide images + explicit extra refs only; do NOT silently fall back to the existing AI-generated reference_image_url unless the user explicitly requests an "edit existing reference" behavior

AI Image Editing / Regeneration UX Parity

• Treat AI image generation/editing/regeneration as long-running jobs with persistent user-visible state, not as button-local spinners. If a user refreshes or reopens the PWA, active and recent jobs should rehydrate from SQLite/server status.
• Users must be able to run multiple image jobs in parallel. Do not globally disable all generate/edit buttons while one job runs; show a compact activity/log panel with all active jobs, statuses, errors, and completions.
• Every uploaded, generated, and edited image should automatically enter the image-analysis/enrichment path. If analysis cannot run, show a clear “Needs setup”/provider issue rather than silently omitting analysis or pretending a generic caption is enough.
• Editing/regenerating from an existing image must actually send the source/guide/reference image(s) to the provider. Before changing prompts, inspect the provider payload and model reference caps (image_url vs image_urls, PuLID single ref, Qwen caps, etc.).
• UI feedback should happen where the user is looking: on the placeholder/preview or active card, with logs collapsible nearby. Button placement should support the flow: prompt/model controls below the preview, not detached from the image being edited.

Analysis Phase UI (single-column cards)

• Single-column layout (not grid) — gives each entity a full-width hero image
• Each card: hero reference image, entity type pill, status badge (Reference/Guides added/No images)
• Guide image thumbnail strip with inline + button directly on cards (no drawer needed to upload)
• Footer action: "Set Up & Generate" or "Edit & Regenerate" opens detail panel
• fileInputRefs use a ref object keyed by ${type}_${id} for per-card file inputs

Detail Panel (layout order — no accordion/advanced section)

1. Guide Images (top, most prominent) — multi-upload grid with always-visible delete badges, big empty-state CTA - Shows active vs overflow guides based on selected model's maxRefs - Overflow guides: faded, greyscale, "unused" overlay - Warning message with suggestion to switch models for more refs
2. Current Reference Image — display only (if exists), with both "upload your own" and clear current reference actions - Clearing the current reference should set reference_image_url = NULL without deleting guide images - Endpoint pattern: DELETE /api/:entityType/:id/reference-image
3. Detail Fields — name, description, appearance, personality etc.
4. Model + Generate (bottom of form) — model dropdown + single generate button - Model selector label: always "Default (FLUX.2 Pro)" — never expose PuLID to user (implementation detail) - Generate button shows: model name, price, guide count - NO separate prompt textarea in the default detail flow — prompt built server-side from entity structured fields - generate-advanced endpoint builds prompt via buildCharacterPrompt/buildSetPrompt/buildPropPrompt when empty prompt sent - IMPORTANT: disable generate while entity save is in flight (saving) so users cannot save and immediately regenerate against stale DB state

UX Principle: No Duplicate Controls

• NEVER put duplicate prompts, guide images, or settings inside an "advanced" accordion
• "Advanced" should only ADD controls (e.g. model selector) on top of existing UI
• Alex explicitly rejected the pattern of an accordion that duplicated the prompt and guide images
• The model selector is just a dropdown near the generate button, not a separate section

Landing Page / Account Panel

• Home page now has a top-right ⚙️ Account button above the Video Story title
• The account view is a right-side drawer, following the same fixed inset-0 z-50 flex justify-end pattern as ActivityPanel / DetailPanel
• Account data comes from GET /api/account/overview
• server/account-status.js is the source of truth for:
• provider credit/account summaries
• grouped model inventory by platform
• This endpoint should report reality, not aspirational balances. If a provider does not expose credit balance with the current key/API, surface that clearly in note

Provider Billing Reality

• fal.ai billing balance is available via GET https://api.fal.ai/v1/account/billing?expand=credits
• BUT fal.ai only allows this for ADMIN keys. Standard generation keys return 403 with an insufficient-permissions message
• Therefore, if the app uses a non-admin FAL_KEY, the UI must show balance unavailable and explain that billing reads require an ADMIN key
• Replicate GET https://api.replicate.com/v1/account returns account identity (e.g. username) but does not expose remaining credit balance in the currently used API flow
• Therefore, the app should show Replicate account identity if available, but mark credit balance unavailable instead of faking a number
• server/account-status.js imports dotenv/config directly so standalone scripts/tests loading it still see .env

PWA Patterns

• Service Worker: skipWaiting + clientsClaim for immediate updates on deploy
• registerType: 'autoUpdate' in Vite PWA config
• iOS downloads: Use navigator.share() with blob — <a download> fails on iOS PWA
• Desktop downloads: Blob URL + programmatic click
• Server has /api/projects/:id/download-video endpoint with Content-Disposition: attachment
• Close button safe area: Use env(safe-area-inset-top) for phone status bar clearance
• Min tap targets: 44x44px for mobile

YOLO Progress & Live Updates

• Status endpoint returns yoloStep (camelCase), recent logs array
• Frontend polls every 3s during active runs
• Animated spinner + live activity feed with timestamps
• Verbose server-side logging for each sub-step (per-scene, per-frame progress)
• refreshKey pattern: ProjectPage increments refreshKey every 5s during YOLO. ALL phase components (Analysis, Timeline, Images, Video, Export) must accept refreshKey prop and include it in their useEffect fetch dependency array: useEffect(() => { fetchData() }, [project.id, refreshKey])
• Bug history: originally only AnalysisPhase had refreshKey — Timeline/Images/Video/Export were stale during YOLO until user switched tabs. Fixed by passing refreshKey to every phase.

Audio-Video Sync (FIXED)

• Audio is the master timeline — shots derive durations from script_segments, not LLM
• breakdownShots() LLM decides cinematography; code computes duration_ms from segment timestamps
• Shot durations use "full slot" timing: from segment's start_time_ms to next segment's start_time_ms (includes 150ms silence gaps)
• Durations rounded to 0.5s for Replicate (WAN 2.2 accepts floats 0.5–10)
• Export trims each clip to exact target duration with ffmpeg -t flag
• Final merge uses audio duration as target: -t ${audDuration} (not -shortest)
• actual_video_duration_ms stored after generation for drift monitoring
• Venice AI rejected as provider: only supports integer-second enums, pipeline needs fractional durations

Advanced Image Models (fal.ai)

• All models use image_urls (array) for reference images — including Qwen standard Edit
• Bug history: Qwen Edit was sending image_url (single string) which caused 422. ALL Qwen variants need array.
• Model param mapping in advanced-images.js: kontext/reve use image_url (single), everything else uses image_urls (array)
• When no prompt provided to generate-advanced, server auto-builds from entity fields using same functions as default generate
• IMPORTANT nuance: if the default detail-panel flow sends an empty prompt for advanced generation, project style is still injected indirectly because the server rebuilds via buildCharacterPrompt/buildSetPrompt/buildPropPrompt
• If you want truly manual/no-style-injection advanced prompting, the client must send an explicit full prompt instead of prompt: ''

Lip Sync (dialogue shots)

• Module: server/lipsync.js
• Model: Kling Lip Sync via Replicate (kwaivgi/kling-lip-sync) at $0.014/sec
• Fallback: Sync Lipsync 2 (sync/lipsync-2) at $0.05/sec
• How it works: Post-processing step — takes existing video clip + extracted audio segment, produces lip-synced version
• Pipeline integration: Step 9 in YOLO (between Videos and Export). Non-blocking — errors don't stop export.
• Audio extraction: ffmpeg -ss {start} -t {duration} from full narration to get per-shot dialogue audio
• DB: lipsync_url and lipsync_status columns on shots table
• Export: COALESCE(lipsync_url, video_url) — prefers lip-synced version when available
• Only dialogue shots: Identified by segment_type = 'dialogue' on the shot's linked script_segment
• API endpoints:
• GET /api/projects/:id/lipsync-status — status + cost estimate
• POST /api/projects/:id/lipsync-all — batch process (runs in background, logs progress)
• POST /api/shots/:id/lipsync — single shot (synchronous)
• Kling constraints: video 2-10 sec, 720p-1080p — matches our shot lengths perfectly
• Cost: ~$0.31 for 7 dialogue shots (22 sec total). Very cheap addition to pipeline.

Pitfalls

• Video generation queue is in-memory but shot status is SQLite-backed. After process restart, stale video_status='generating' rows can survive with no active job. getQueueStatus(projectId) should reset those rows to pending when activeJobs===0 and queue.length===0, and /video-status should expose active/waiting/halted/maxConcurrent so the UI can explain what is actually running. See references/video-queue-refresh-and-qwen-pro-ref-cap-2026-05-18.md.
• PWA cache: After deploy, users may need to close+reopen PWA for SW update
• Safari vs PWA: Separate caches — Safari browser cache ≠ PWA SW cache
• Shot breakdown: Must retry per-scene (not fail entire step on one scene failure)
• YOLO must validate ALL scenes have shots before proceeding to refs
• OpenRouter 429s: Already handled with fallback, but batch operations need throttling
• Steps nav: horizontal scrollable top bar (not bottom tab bar) — user preference
• Mobile touch: hover-only UI (opacity-0 group-hover:opacity-100) doesn't work on touch devices. Delete buttons on guide images must be always-visible (red circle badge at -top-1 -right-1), not hover overlays.
• No duplicate controls in "advanced" sections: Alex explicitly rejected accordion patterns that duplicate prompt/guide-images inside an advanced panel. Advanced = just a model selector dropdown, reusing existing data.
• Prompt staleness after entity edits: if a user edits prompt-driving fields (appearance, set description, prop description, mood, etc.), previously auto-generated reference_image_prompt values can become stale. On save, invalidate auto-generated prompts (but preserve [ADVANCED:...] and [CUSTOM_UPLOAD] prompts) so later regeneration rebuilds from current structured fields.
• Frame/reference reset UX matters: when users clear a reference image or frame, only null the generated asset URL (reference_image_url, first_frame_url, last_frame_url). Do not delete guide images or prompts. Clearing is for resetting generation state, not deleting upstream inputs.

Hermes automation API workflow

• Prefer a small authenticated local API bridge for Hermes-driven operation instead of browser-driving the app.
• Useful endpoints:
• POST /api/hermes/projects/create-and-run
• GET /api/hermes/projects/:id/status
• GET /api/hermes/projects/:id/export
• The Hermes task is not complete when the pipeline finishes server-side; it is complete only after Hermes fetches the export and delivers the final video back to the user/chat.
• Good defaults that worked here:
• aspect_ratio: 16:9 unless portrait requested
• reference_image_model: qwen

• frame_image_model: qwen

• references/video-queue-refresh-and-qwen-pro-ref-cap-2026-05-18.md — session detail for stale video generating rows after PM2 restart, maxConcurrent status reporting, and the live Fal Qwen Pro Edit 3-reference cap.

• Progress/state polling should drive ALL major tabs/components, not just analysis, or the UI goes stale during long runs.
• Save prompt fields needed for retries BEFORE generation starts; do not only persist them on success.
• Retry flow should switch providers intelligently for safety/moderation failures instead of hammering the same provider again.
• For frame retries, reuse the same ranked reference bundle as the primary path; do not degrade to weaker text-only retries if continuity refs exist.
• Lip-sync stalls may actually be post-provider finalization hangs (download/upload/final save), not the external model itself still running.
• Keep export non-blocking with COALESCE(lipsync_url, video_url) so one failed lipsync does not destroy the whole pipeline.