Dataface Tasks
    Heartbeat snapshot (JSON)

    Streaming YAML generation with early query execution

    IDMCP_ANALYST_AGENT-STREAMING_YAML_GENERATION_WITH_EARLY_QUERY_EXECUTION
    Statusnot_started
    Priorityp3
    Milestonemx-far-future-ideas
    Ownerdata-ai-engineer-architect

    Problem

    When AI generates a dashboard, stream the YAML and begin executing queries as they arrive rather than waiting for the full document. Queries typically appear first in the YAML and are the slowest part of the pipeline — streaming lets them get a head start while charts and layout are still being generated. Requires partial YAML parsing, incremental compilation, and progressive rendering. Inspired by json-render SpecStream. See ai_notes/research/json-render-deep-dive.md Priority 4.

    Context

    • Research origin: ai_notes/research/json-render-deep-dive.md — Priority 4 section. Inspired by json-render's SpecStream (JSONL streaming with RFC 6902 patches).
    • Current AI integration: dataface/ai/ — MCP server, prompts, tools
    • Compilation pipeline: dataface/core/compile/ → execute/ → render/
    • Key insight: queries appear first in YAML and are the bottleneck. Early execution while charts are still generating gives a massive head start.
    • Related task: patch-based-ai-edits-for-dashboard-yaml.md — patch-based edits and streaming generation share infrastructure (incremental YAML parsing, partial compilation)

    Possible Solutions

    • A - Keep the current full-document wait model: simpler control flow, but it leaves avoidable latency on the table.
    • B - Recommended: stream YAML generation and start executing stable query blocks early: define clear execution triggers, invalidation behavior, and observability so partial work remains safe to reason about.
    • C - Execute on every token or partial line as soon as it appears: potentially fastest, but far too brittle for reliable operation.

    Plan

    1. Identify the earliest stable generation checkpoints where query blocks can be treated as executable rather than still in flux.
    2. Design the streaming and invalidation model, including how the system cancels, retries, or reuses early executions when later YAML changes.
    3. Implement the bounded early-execution path with tracing or debug output that makes the behavior inspectable.
    4. Measure latency improvement and failure modes against the current path before deciding whether to expand the feature further.

    Implementation Progress

    Review Feedback

    • [ ] Review cleared