systematic-debugging

/home/avalon/.hermes/skills/software-development/systematic-debugging/SKILL.md · raw

Systematic Debugging

Overview

Random fixes waste time and create new bugs. Quick patches mask underlying issues.

Core principle: ALWAYS find root cause before attempting fixes. Symptom fixes are failure.

Violating the letter of this process is violating the spirit of debugging.

The Iron Law

NO FIXES WITHOUT ROOT CAUSE INVESTIGATION FIRST

If you haven't completed Phase 1, you cannot propose fixes.

When to Use

Use for ANY technical issue: - Test failures - Bugs in production - Unexpected behavior - Performance problems - Build failures - Integration issues

Use this ESPECIALLY when: - Under time pressure (emergencies make guessing tempting) - "Just one quick fix" seems obvious - You've already tried multiple fixes - Previous fix didn't work - You don't fully understand the issue

Don't skip when: - Issue seems simple (simple bugs have root causes too) - You're in a hurry (rushing guarantees rework) - Someone wants it fixed NOW (systematic is faster than thrashing)

The Four Phases

You MUST complete each phase before proceeding to the next.


Phase 1: Root Cause Investigation

BEFORE attempting ANY fix:

When the user is visibly frustrated with repeated UI/provider fixes, treat that as a signal to slow down and inspect the architecture first: map persisted state vs in-memory/UI state, provider payloads, queue status, and refresh/restart behavior before touching CSS or prompts. Alex especially expects root-cause evidence for screenshot-driven visual bugs, AI editing/reference bugs, and long-running job feedback.

For screenshot-driven web/mobile bugs, verify the exact route and component tree before declaring the fix. The same product can render different shells on /, /chat, /account, etc.; a CSS/component fix on one route may not touch the screenshot route. Reproduce the screenshot route at the same viewport, inspect visible buttons/classes/ARIA labels, then patch the owning component or add an app-level consumer override if the element comes from a shared UI kit.

1. Read Error Messages Carefully

Action: Use read_file on the relevant source files. Use search_files to find the error string in the codebase.

2. Reproduce Consistently

Action: Use the terminal tool to run the failing test or trigger the bug:

# Run specific failing test
pytest tests/test_module.py::test_name -v

# Run with verbose output
pytest tests/test_module.py -v --tb=long

3. Check Recent Changes

Action:

# Recent commits
git log --oneline -10

# Uncommitted changes
git diff

# Changes in specific file
git log -p --follow src/problematic_file.py | head -100

4. Gather Evidence in Multi-Component Systems

WHEN system has multiple components (API → service → database, CI → build → deploy):

BEFORE proposing fixes, add diagnostic instrumentation:

For EACH component boundary: - Log what data enters the component - Log what data exits the component - Verify environment/config propagation - Check state at each layer

Run once to gather evidence showing WHERE it breaks. THEN analyze evidence to identify the failing component. THEN investigate that specific component.

5. Trace Data Flow

WHEN error is deep in the call stack:

WHEN a UI panel shows stale/wrong “recommendations,” “details,” or generated markdown:

UI/canvas/scrolling pitfall: When a visualization appears to redraw, duplicate, or accumulate marks during scroll/zoom, do not first reduce rendered data, cap labels, or change visual density. Alex may be reporting data mutation, not clutter. Trace event handlers (onScroll, pinch/zoom, debounce timers), fetch triggers, cache merge/prune logic, and server sampling/ranking before changing rendering limits. If the requirement is “all waves/labels,” preserve full data and remove scroll-stop/gesture-driven append paths instead.

Image-render verification pitfall: When a screenshot/vision check says an image-heavy page is blank, do not rely on the visual observation alone. Check network/auth status for the image URLs, then inspect document.images for complete, naturalWidth, naturalHeight, and currentSrc. If the DOM says images loaded but the screenshot still looks blank, draw the image to a small canvas and sample pixels to distinguish a real render failure from capture/timing/tool artifacts. This is especially important for lazy-loaded, progressively swapped, or protected-route media pages.

Screenshot route/component mismatch pitfall: When a user says a visual bug still exists after a fix, do not assume they are viewing the same route/component you just tested. Inspect the exact URL/path visible in the screenshot, reproduce that route at the same viewport, and enumerate visible interactive elements/classes before patching again. Shared UI kits can render different toggles than app-specific routes; for example a public / landing page may use shared chat classes while /chat uses tenant-specific classes. Fix the actual component producing the screenshot artifact, then verify on that exact route.

Transparent video alpha pitfall: When a transparent WebM appears to have a background in a web viewer, first separate the video asset from the page’s review surface. Checkerboard/cream test backgrounds can be mistaken for baked-in video pixels. Verify URL delivery/auth, then serve same-origin or with CORS and draw the video to canvas; sample corner/background pixels for alpha 0. Treat VP9 WebM metadata carefully: pix_fmt=yuv420p plus ALPHA_MODE=1, or alphaextract failure, is not by itself proof that transparency is absent. See references/transparent-video-alpha-debugging.md.

Circular dial / compass pitfall: For compass, clock, orbital, heading, or azimuth UIs, always inspect wraparound at the 0°/360° boundary before blaming sensors or animation libraries. CSS transforms may interpolate the long way around when normalized angles jump from 359 → 0; use shortest-angle deltas and an unwrapped visual angle for transforms while keeping normalized angles for readouts/calculations. Also distinguish fixed screen indicators from world-space marks: if targets rotate under a fixed needle, cardinal labels/azimuth markers should usually rotate too.

For timeline/canvas duplicate glyph clusters, compare semantic event identity rather than raw IDs: scope + transitingPlanet + aspect + natalPlanet + exact-time bucket. Overlapping scroll fetches can return the same event with slightly different timestamps/IDs; coalesce by semantic tuple and keep the lower-orb/best sample. Treat rendering smoothness separately from dedupe: use curve smoothing for pointy paths, but do not hide duplicate data by lowering density. See references/canvas-scroll-duplicate-rendering.md for the session-derived checklist and fix pattern.

For timeline/canvas filled ribbon artifacts that appear only after repeated scrolling, inspect both client cache growth and path clipping. Long station/outer-planet events may need duration-aware dedupe windows; a fixed 48-72h exact-time bucket can miss resampled duplicates. Prune stale cached events after fetches, and skip drawing tiny visible curve fragments that are too short to represent a real wave. See references/canvas-scroll-ribbon-artifacts.md.

Queue-backed ingestion pitfall: For apps that split upload/extraction/embedding/LLM transformations across API + worker + database queues, verify each stage separately before declaring the app broken. Check API health, source extraction/text length, queued command status, worker logs, embedding chunk counts, and text/vector search. If commands remain new with no worker activity, restart the worker with updated env and verify it picks up existing commands. Also disable default whole-document LLM transformations during ingestion smoke tests: context-length failures in an auto-summary step can mask successful extraction and embedding. For the Open Notebook worked example, see vps-app-deployment/references/open-notebook-ingestion-ops.md.

Action: Use search_files to trace references:

# Find where the function is called
search_files("function_name(", path="src/", file_glob="*.py")

# Find where the variable is set
search_files("variable_name\\s*=", path="src/", file_glob="*.py")

Phase 1 Completion Checklist

STOP: Do not proceed to Phase 2 until you understand WHY it's happening.


Phase 2: Pattern Analysis

Find the pattern before fixing:

1. Find Working Examples

Action: Use search_files to find comparable patterns:

search_files("similar_pattern", path="src/", file_glob="*.py")

2. Compare Against References

3. Identify Differences

4. Understand Dependencies


Phase 3: Hypothesis and Testing

Scientific method:

1. Form a Single Hypothesis

2. Test Minimally

3. Verify Before Continuing

4. When You Don't Know


Phase 4: Implementation

Fix the root cause, not the symptom:

1. Create Failing Test Case

2. Implement Single Fix

3. Verify Fix

# Run the specific regression test
pytest tests/test_module.py::test_regression -v

# Run full suite — no regressions
pytest tests/ -q

4. If Fix Doesn't Work — The Rule of Three

5. If 3+ Fixes Failed: Question Architecture

Pattern indicating an architectural problem: - Each fix reveals new shared state/coupling in a different place - Fixes require "massive refactoring" to implement - Each fix creates new symptoms elsewhere

STOP and question fundamentals: - Is this pattern fundamentally sound? - Are we "sticking with it through sheer inertia"? - Should we refactor the architecture vs. continue fixing symptoms?

Discuss with the user before attempting more fixes.

This is NOT a failed hypothesis — this is a wrong architecture.


Red Flags — STOP and Follow Process

If you catch yourself thinking: - "Quick fix for now, investigate later" - "Just try changing X and see if it works" - "Add multiple changes, run tests" - "Skip the test, I'll manually verify" - "It's probably X, let me fix that" - "I don't fully understand but this might work" - "Pattern says X but I'll adapt it differently" - "Here are the main problems: [lists fixes without investigation]" - Proposing solutions before tracing data flow - "One more fix attempt" (when already tried 2+) - Each fix reveals a new problem in a different place

ALL of these mean: STOP. Return to Phase 1.

If 3+ fixes failed: Question the architecture (Phase 4 step 5).

Mobile sensor / browser permission debugging

When debugging phone-sensor features (compass, device orientation, geolocation), treat each browser permission and each event stream as a separate component boundary. Location success plus API data success does not prove compass heading is active.

Specific iOS/WebKit pitfall: DeviceOrientationEvent.requestPermission() must be called directly inside the original user gesture. Do not await geolocation or another permission dialog first; start the compass permission request immediately from the tap handler, then await location in parallel. If heading events still do not arrive, distinguish Safari/PWA behavior from in-app webviews such as Telegram.

Also avoid misleading fallback headings: render “waiting for compass” until a real heading event arrives instead of defaulting missing heading to for turn/alignment instructions.

See references/ios-device-orientation-permission-sequencing.md for the worked Astral Hermes Sky Compass pattern.

Common Rationalizations

Excuse Reality
"Issue is simple, don't need process" Simple issues have root causes too. Process is fast for simple bugs.
"Emergency, no time for process" Systematic debugging is FASTER than guess-and-check thrashing.
"Just try this first, then investigate" First fix sets the pattern. Do it right from the start.
"I'll write test after confirming fix works" Untested fixes don't stick. Test first proves it.
"Multiple fixes at once saves time" Can't isolate what worked. Causes new bugs.
"Reference too long, I'll adapt the pattern" Partial understanding guarantees bugs. Read it completely.
"I see the problem, let me fix it" Seeing symptoms ≠ understanding root cause.
"One more fix attempt" (after 2+ failures) 3+ failures = architectural problem. Question the pattern, don't fix again.

Quick Reference

Phase Key Activities Success Criteria
1. Root Cause Read errors, reproduce, check changes, gather evidence, trace data flow Understand WHAT and WHY
2. Pattern Find working examples, compare, identify differences Know what's different
3. Hypothesis Form theory, test minimally, one variable at a time Confirmed or new hypothesis
4. Implementation Create regression test, fix root cause, verify Bug resolved, all tests pass

Hermes Agent Integration

Investigation Tools

Use these Hermes tools during Phase 1:

Doc-reading method preference

When the user wants documentation checked, prefer this order: 1. web_extract for direct doc/page extraction 2. web_search to discover the right docs page 3. Browser tools only as fallback when extraction/search fails, content is JS-rendered, or interactive discovery is required

Why: users may prefer Firecrawl/web extraction over browser snapshots for straight documentation reading, and browser browsing is slower/noisier unless necessary.

API-doc debugging pattern

For third-party API integration bugs, add this Phase 1 checklist before proposing fixes:

Common lesson: docs drift faster than app registries. Before assuming a provider requires special access, verify whether the endpoint path, payload shape, or local access-check logic is stale or misleading.

SDK stream-parsing debugging pattern

When TypeError: 'NoneType' object is not iterable (or similar parse errors) originate from INSIDE the SDK (stack trace points to lib/_parsing/* or streaming/responses/*), the API is sending null where the SDK expects []. Bypass or harden the SDK path — read the raw HTTP/SSE stream with httpx or collect streamed events before final parsing so you can see what the API actually sends in the terminal response.completed event. Patch every runtime that uses the Responses stream, not only the main chat loop: auxiliary clients used for vision/compression can hit the same parse error. Catch the TypeError both during stream iteration and around stream.get_final_response(), then synthesize a minimal final response from response.output_item.done items or output_text.delta chunks when no function calls were involved.

See references/sdk-stream-parsing-debug.md for the full technique and a real worked example (ChatGPT Codex backend sending output: null).

With delegate_task

For complex multi-component debugging, dispatch investigation subagents:

delegate_task(
    goal="Investigate why [specific test/behavior] fails",
    context="""
    Follow systematic-debugging skill:
    1. Read the error message carefully
    2. Reproduce the issue
    3. Trace the data flow to find root cause
    4. Report findings — do NOT fix yet

    Error: [paste full error]
    File: [path to failing code]
    Test command: [exact command]
    """,
    toolsets=['terminal', 'file']
)

With test-driven-development

When fixing bugs: 1. Write a test that reproduces the bug (RED) 2. Debug systematically to find root cause 3. Fix the root cause (GREEN) 4. The test proves the fix and prevents regression

Real-World Impact

From debugging sessions: - Systematic approach: 15-30 minutes to fix - Random fixes approach: 2-3 hours of thrashing - First-time fix rate: 95% vs 40% - New bugs introduced: Near zero vs common

No shortcuts. No guessing. Systematic always wins.