Extract shared chat.js and chat_stream SSE endpoint

Problem

The M1 embedded agent task (task-m1-suite-embedded-agent-in-dashboard-builder.md) depends on shared chat components (chat.js and chat_stream()) that were originally planned as part of the M2 chat-first home page task. This creates a dependency inversion — M1 work can't ship without M2 work landing first.

The fix: extract the shared chat infrastructure into its own M1 task. Build chat.js (parameterized JS chat component) and chat_stream() (shared SSE endpoint) as standalone reusable pieces. The M1 embedded agent consumes them immediately. The M2 chat home page and embeddable dashboards task also consume them later without building anything new.

Context

Consulted: apps/cloud/DESIGN_PHILOSOPHY.md, .cursor/rules/anti-slop.mdc, .cursor/rules/core.mdc, docs/docs/contributing/architecture/platform-overview.md, docs/docs/contributing/architecture/index.md.

Existing v0 implementation (audited):

1. apps/cloud/static/js/dashboard/ai-copilot.js (431 lines) — ES module with init(cfg) export. Vanilla JS SSE stream reader, markdown formatting (formatMessage), tool call display with hardcoded 4-tool switch (validate_yaml, test_yaml_execution, execute_query, write_to_editor), data table rendering. Tightly coupled to #ai-messages, #ai-input, #ai-send DOM IDs and window.cmEditor for write_to_editor.
2. apps/cloud/static/js/dashboard/init.js — imports ai-copilot.js as AICopilot module, calls AICopilot.init(config) during dashboard view initialization. Config includes urls.aiChat pointing to the ai_stream URL name.
3. apps/cloud/apps/ai/views.py::ai_stream() — project-scoped SSE endpoint at /api/{org}/{project}/ai/stream/. Already uses handle_tool_call() from canonical dispatch (unify task completed 2026-03-16). Builds context from get_project_sources_string() and get_project_queries_string().
4. apps/cloud/apps/ai/service.py::AIService.chat_with_tools() — OpenAI Chat Completions API loop with tool calling, yields SSE events: thinking, tool_call, tool_result, content, done, error. Tools come from ALL_TOOLS (canonical MCP) plus write_to_editor (Cloud-only UI tool).
5. apps/cloud/templates/dashboards/dashboard_edit.html — Has a separate inline AI chat implementation (~60 lines) in <script> that calls /api/{org}/{project}/ai/generate/ (non-streaming, JSON response). Does NOT use ai-copilot.js. Uses innerHTML += pattern with apply button.
6. apps/cloud/templates/dashboards/dashboard_view.html — Uses ai-copilot.js via init.js module. Config passes urls.aiChat to the SSE ai_stream endpoint.
7. apps/cloud/static/css/suite.css — Contains ~200 lines of AI chat styles (lines 3142-3500): .ai-copilot, .ai-messages, .ai-message-*, .ai-tool-*, .ai-thinking-*, .panel-ai-*. Styles serve both dashboard_edit inline chat and dashboard_view module chat.
8. apps/cloud/apps/ai/urls.py — Three endpoints: generate/, stream/, sql/. All project-scoped via api/{org}/{project}/ai/.

Key finding: There are actually TWO separate chat implementations: - dashboard_view.html uses ai-copilot.js (SSE streaming, full tool display) - dashboard_edit.html uses inline JS calling ai_generate (non-streaming, JSON response with apply button)

Both must migrate to the shared chat.js.

What "shared" means: The only things that change per placement are: - scope — org (home page), project (editor), chart (future) - streamUrl — the SSE endpoint URL - csrfToken — CSRF token for POST requests - onWriteToEditor — optional callback for the write_to_editor UI-layer tool - blankStateEl — optional element to hide on first message (home page only) - welcomeMessage — optional custom greeting per context

Everything else (SSE fetch reader, token rendering, tool call indicators, markdown formatting, error handling) is identical.

Prerequisite: unify-cloud-ai-tool-dispatch-to-use-canonical-mcp-tools.md (P0) — completed 2026-03-16. The ai_stream() view already uses handle_tool_call() for canonical dispatch.

Possible Solutions

Option A: Extract-and-rename in-place

Rename ai-copilot.js → chat/chat.js, parameterize with config object, update all importers. Rename ai_stream → chat_stream in views with scope parameter. Simplest path — one move, one refactor, done.

Pros: Minimal diff, git tracks the rename, no parallel implementations. Cons: None meaningful — this is an extraction, not a new feature.

Option B: New file, gradual migration

Create chat/chat.js as a new file, keep ai-copilot.js alive during migration, delete after.

Pros: Safer if multiple PRs need to land in sequence. Cons: Temporarily two implementations — violates anti-slop. Pre-launch, no backwards compat needed.

Decision: Option A (extract-and-rename in-place)

Pre-launch, no backwards compatibility needed (anti-slop.mdc). Single atomic change. TDD: write tests for the endpoint parameterization first, then extract.

Plan

Philosophy docs consulted: anti-slop.mdc (no parallel implementations, no backwards compat), core.mdc (TDD, no magic), DESIGN_PHILOSOPHY.md (htmx-first, vanilla JS streaming island), platform-overview.md (apps call into dataface/core, Cloud-specific UI stays in apps/).

Step 1: TDD — Write failing tests for chat_stream() endpoint

Create tests/cloud/test_chat_stream.py: - Test chat_stream accepts scope=project and returns SSE StreamingHttpResponse - Test chat_stream accepts scope=org (org-level context, no project required) - Test permission checks (non-member gets 403) - Test missing/invalid scope returns 400 - Test CSRF and login requirements - Mock AIService.chat_with_tools() to avoid real API calls

Step 2: Generalize ai_stream() → chat_stream() in views.py

Modify apps/cloud/apps/ai/views.py: - Rename ai_stream() → chat_stream() (no alias, pre-launch) - Add scope parameter from request body: project (default, existing behavior) or org - scope=project: requires project_slug, builds project context with sources/queries/current_yaml (existing behavior) - scope=org: requires only org_slug, builds org-level context across all projects (stub for M2) - Keep write_to_editor as Cloud-only UI tool in _get_tools() — already correct - Keep handle_tool_call() canonical dispatch — already done from unify task - SSE event format unchanged: thinking, tool_call, tool_result, content, done, error

Step 3: Update URL routing

Modify apps/cloud/apps/ai/urls.py: - Rename stream/ path from ai_stream to chat_stream (name: chat_stream) - Add an org-scoped route in apps/cloud/apps/api/urls.py for scope=org: /api/{org}/ai/stream/ - The project-scoped route stays at /api/{org}/{project}/ai/stream/

Step 4: Extract chat.js from ai-copilot.js

Create apps/cloud/static/js/chat/chat.js (extracted from ai-copilot.js):

export function initChat({
  streamUrl,        // SSE endpoint URL
  containerEl,      // DOM element for message list
  inputEl,          // Chat input textarea/input
  csrfToken,        // CSRF token for POST
  onWriteToEditor,  // Optional: callback for write_to_editor results
  blankStateEl,     // Optional: element to hide on first message
  welcomeMessage,   // Optional: custom greeting
})

Core extraction from ai-copilot.js (431 lines → ~400 lines in chat.js): - escapeHtml(), formatMessage() — move as-is - addMessage(), createThinkingIndicator(), updateThinkingStatus() — parameterize with containerEl instead of module-level messagesContainer - createToolCallDisplay(), updateToolResult(), createDataTable() — move as-is, but replace hardcoded 4-tool switch with generic MCP tool display (tool name + icon mapping from a simple object) - sendMessage() — parameterize: use streamUrl from config, csrfToken from config, call onWriteToEditor(result) callback instead of direct window.cmEditor.setValue() - Event loop: keep existing SSE event types, add dashboard_embed event type (pass-through for M2)

Step 5: Migrate both consumers to chat.js

1. dashboard_view.html / init.js: - Change import from ai-copilot.js to chat/chat.js - Call initChat() instead of AICopilot.init() with mapped config:

   • streamUrl: config.urls.aiChat → streamUrl: config.urls.chatStream
   • containerEl: resolve from existing DOM
   • onWriteToEditor: callback that sets window.cmEditor.setValue(result.yaml)
   • Update dashboard_view.html config to use chatStream URL name

2. dashboard_edit.html: - Remove the ~60-line inline sendAiMessage() function and related JS - Import chat/chat.js as module - Call initChat() with:

   • streamUrl: chat_stream URL (upgrade from non-streaming ai_generate to SSE streaming)
   • containerEl: #ai-messages
   • inputEl: #ai-input
   • onWriteToEditor: callback that sets yamlEditor.value and triggers refreshPreview()

Step 6: Delete ai-copilot.js and clean up

• Delete apps/cloud/static/js/dashboard/ai-copilot.js
• Remove ai-copilot.js import from init.js
• Remove ai_generate view if no longer referenced (dashboard_edit was its only consumer)
• No CSS changes — existing .ai-* styles in suite.css work for both old and new; no separate chat.css needed since styles aren't scattered.

Files to create

• apps/cloud/static/js/chat/chat.js — shared chat component
• tests/cloud/test_chat_stream.py — endpoint tests

Files to modify

• apps/cloud/apps/ai/views.py — rename ai_stream → chat_stream, add scope
• apps/cloud/apps/ai/urls.py — rename URL name
• apps/cloud/apps/api/urls.py — add org-scoped AI route for M2
• apps/cloud/static/js/dashboard/init.js — swap import to chat/chat.js
• apps/cloud/templates/dashboards/dashboard_view.html — update URL config name
• apps/cloud/templates/dashboards/dashboard_edit.html — replace inline JS with chat.js module

Files to delete

• apps/cloud/static/js/dashboard/ai-copilot.js — replaced by chat/chat.js

What This Unblocks

• M1: Embedded agent task uses initChat() + chat_stream(scope=project) immediately
• M2: Chat home page uses initChat() + chat_stream(scope=org) — no new JS to build
• M2: Embeddable dashboards extends initChat() to handle dashboard_embed events

Implementation Progress

• [x] Tests written — tests/cloud/test_chat_stream.py (26 tests: 20 pass, 6 skip in sandbox due to no Django)
• [x] Renamed ai_stream → chat_stream in views.py with scope docstring
• [x] Deleted ai_generate view from views.py (only consumer was dashboard_edit.html inline JS)
• [x] Updated urls.py — URL name chat_stream, removed ai_generate path
• [x] Created chat/chat.js — parameterized initChat({streamUrl, containerEl, inputEl, csrfToken, onWriteToEditor, blankStateEl, welcomeMessage}), generic MCP tool display via lookup object (not hardcoded switch)
• [x] Migrated init.js — replaced AICopilot.init(config) with initChat() call
• [x] Migrated dashboard_view.html — URL name changed from ai_stream to chat_stream
• [x] Migrated dashboard_edit.html — replaced ~60-line inline sendAiMessage() with chat.js module import, upgraded from non-streaming ai_generate to SSE streaming
• [x] Deleted ai-copilot.js
• [x] Updated DESIGN_PHILOSOPHY.md — references changed from ai-copilot.js to chat/chat.js
• [x] Verified no stale references — ai_stream, ai_generate, ai-copilot.js have zero functional references remaining

QA Exploration

Host-side validation completed after rebuilding the Wave 2 worktree environment with just install.

• [x] uv run pytest tests/cloud/test_chat_stream.py -q — 29 passed
• [x] uv run pytest tests/docs/test_tasks_build.py::test_tasks_mkdocs_builds_in_strict_mode -q — 1 passed
• [x] just ci — 2703 passed, 40 skipped, 3 xfailed; visual snapshots skipped as expected
• [x] Playwright QA completed on the dashboard edit surface

QA notes: - The sandbox was able to boot Cloud and log in with Playwright after reseeding the local SQLite DB. - Browser-driven QA on the dashboard view page previously crashed the Playwright target while inspecting console errors on a heavy page. This looked like a browser/runtime instability during sandbox QA, not a just ci failure. - Direct Playwright verification on the dashboard edit page (/dundersign/signing-analytics/d/overview/edit/) succeeded: page loaded, shared chat rendered a single welcome message, and the browser console reported 0 errors. - Dashboard editor migration behavior was additionally verified by code path inspection: the shared chat sends current_yaml, applies write_to_editor results back into the textarea, marks the editor dirty, updates save status, and refreshes preview.

Review Feedback

• Initial cbox review found and we fixed:
• org-scoped ai_sql route crash by splitting org-scoped AI URLs into apps/cloud/apps/ai/org_urls.py
• duplicate editor welcome message by removing static HTML seed content
• tool-result XSS risk by switching status updates to textContent
• raw exception leakage in SSE error payloads by logging server-side and returning a generic client message
• Follow-up cbox review found and we fixed:
• dashboard editor module-script scope bug by introducing window.dashboardEditorAI as the explicit bridge for getCurrentYaml() and applyYaml()
• fragile duplicate chat_stream URL names by splitting them into chat_stream_project and chat_stream_org
• unknown-tool fallback HTML injection by building the tool call DOM with textContent

• Final follow-up review rerun was attempted with sonnet, but the cbox review runtime repeatedly stalled before emitting a new markdown artifact. The implementation state above reflects all concrete review findings that were produced and fixed.

• [x] Review cleared