dft core
Purpose
Open-source core runtime, YAML contract, compatibility, and release quality. This is the engine: the compile/execute/render pipeline that turns YAML dashboard definitions into executable queries and rendered output. Covers the CLI surface, the YAML spec and its normalizer, query execution against warehouses, and the HTML/SVG rendering layer. Owns the OSS release process, versioning contract, and migration story. Adjacent to but distinct from cloud-suite (which is the hosted UX on top of core) and graph-library (which owns chart design within core's rendering layer).
Owner
- Sr Engineer Architect
Tasks by Milestone
A runnable prototype path exists for the YAML contract, compiler/normalizer, execution adapters, and release/versioning, with concrete artifacts that prove the flow works end-to-end in the current codebase. Core assumptions are documented, known constraints are explicit, and the team can explain what is real versus mocked without ambiguity.
- Prototype gaps and follow-on capture Completed — Document top gaps and risks in versioning and migrations that must be addressed next.
- Prototype implementation path Completed — Implement a runnable end-to-end prototype path for YAML contract and normalizer.
- Prototype validation and proof Completed — Validate execution/runtime adapters with concrete proof artifacts and repeatable steps.
Internal analysts can execute at least one weekly real workflow that depends on the YAML contract, compiler/normalizer, execution adapters, and release/versioning in the 5T Analytics environment, without bespoke engineering intervention for every run. Instrumentation and feedback capture are in place so failures, friction points, and adoption gaps are visible and triaged with owners.
- Remove init_sql and migrate setup patterns to query composition Done — Deprecate and remove `init_sql`/`init_sql_file`, then migrate setup use cases to reusable query imports (for example `_…
- Add chart renderer registry and mechanical Vega renderer Completed — Introduce renderer-family dispatch, rebuild standard Vega-Lite translation around resolved chart semantics, and make la…
- Bootstrap dft init for one-command serve in dbt repos CompletedPR #782PR at 2026-03-24T20:53:55-07:00 — Problem: serving a real dbt analytics repo currently needs long commands with explicit face paths project-dir and adapt…
- Build chart intent and enrichment pipeline Completed — Introduce ChartIntent, EnrichmentPatch, ResolvedChart, RenderArtifact, and the authored/enriched/config precedence flow…
- Compile effective chart presentation defaults before render CompletedPR #826 — Move static chart presentation defaults out of renderer-time merge layering and into a compiled effective defaults cont…
- Create canonical chart profile mapping layer before Vega-Lite emission CompletedPR #848PR at 2026-03-27T09:21:05-07:00 — Establish a dedicated mapping/profile boundary between the compiled Dataface chart contract and emitted Vega-Lite specs…
- Create Dataface chart option profile registry and stance overlay — Define the authored Dataface stance layer over the full chart contract so each chart option family records keep/rename/…
- Document faces/ directory convention and dbt project setup Completed — Fix all docs to use faces/ as the canonical dashboard directory. Add Adding Dataface to an Existing dbt Project guide t…
- Ensure dbt generate_schema_name-safe BigQuery ref source resolution Completed — Audit and harden Dataface dbt SQL execution so ref and source resolution for BigQuery uses dbt relation identity semant…
- Generate chart spec docs examples and regression corpus from the typed contract Waiting on create-dataface-chart-option-profile-registry-and-stance-overlay — Generate human-facing chart spec pages, examples, and regression fixtures from the typed chart contract plus Dataface s…
- Generate full Vega-Lite Pydantic chart layer from vendored schema CompletedPR #851 — Generate the full Vega-Lite chart contract as a fully defined Pydantic compatibility layer from the vendored schema, th…
- Move chart output conversion into render converters Completed — Relocate chart export conversion out of chart semantics and route SVG, PNG, and PDF conversion through render/converter…
- Rebuild geo renderer on resolved chart pipeline Completed — Adapt geo rendering to the new intent/enrichment/resolution pipeline with shared source, join, projection, and tooltip…
- Refactor chart renderer boundaries and dispatch Completed — The chart rendering layer still mixes orchestration, Dataface JSON serialization, SVG-native export conversion, and Veg…
- refactor: Move database/source detection from playground to core Completed — Move database/source detection into dft core so playground and hosted surfaces share one canonical detection path.
- Scan chart code for hardcoded defaults config wrappers and unnecessary local aliases CompletedPR #850 — Audit chart compile/render code for hardcoded in-code values, get_some_value style config wrappers, and unnecessary loc…
- Type chart config and settings from compiled presentation contract CompletedPR #846PR at 2026-03-27T07:45:36-07:00 — Tighten Dataface-owned chart config and settings surfaces into typed models after the compiled presentation-defaults co…
- Deduplicate geo chart rendering and config ownership Completed — The geo rendering code still duplicates point-map and layered-point-map behavior across geo source resolution, projecti…
- Extract shared Vega-Lite spec builder helpers Completed — Chart-specific Vega-Lite generators still repeat the same spec assembly patterns for title handling, dimensions, toolti…
- Tighten chart enrichment and type inference boundaries Completed — Chart enrichment and type inference still cross module boundaries in ways that blur responsibilities between render-tim…
- Wire chart aspect ratio from YAML config (global + per-type) and document layout sizing Completed — Implement aspect-ratio-driven sizing using default_config.yml as the single source of truth; add per-chart-type overrid…
- Wire markdown-svg to Dataface font config and vendored Inter CompletedPR #771PR at 2026-03-24T20:20:23-07:00 — Face markdown uses mdsvg without passing font_path; Style from _get_compact_style omits config font_family. Align mdsvg…
the YAML contract, compiler/normalizer, execution adapters, and release/versioning is hardened enough for regular use by multiple internal teams and initial design partners, with a predictable response loop for issues and requests. Quality expectations are documented, and prioritized improvements from real usage are actively incorporated into delivery.
- Implement YAML versioning and migrations — Add built-in schema version handling and migration workflow for dashboard YAML changes.
- Adoption hardening for internal teams — Harden YAML contract and normalizer for repeated use across multiple internal teams and first design partners.
- Design-partner feedback loop operations — Operationalize rapid feedback-to-fix loop for execution/runtime adapters with explicit decision logs.
- feat: chart decisions Phase 5 — YAML annotation overrides — Support YAML annotation overrides for chart decisions so analysts can steer defaults without patching generated specs.
- Highlight charts using branch dev schema versus prod fallback CompletedPR #746PR at 2026-03-23T21:46:57-07:00 — Add chart-level change highlighting in dashboard render paths by detecting whether each query resolved to branch dev sc…
- Quality standards and guardrails — Define and enforce quality standards for versioning and migrations to keep output consistent as contributors expand.
- refactor: Consolidate config system - single source of truth in YAML — Consolidate config loading into one YAML-driven source of truth to eliminate conflicting runtime settings paths.
- Return normalized Dataface format for JSON export (not Vega-Lite specs) — Make JSON export return normalized Dataface spec objects instead of raw Vega-Lite so tooling can round-trip safely.
- Add settings: attribute pattern for all chart types — Standardize a `settings` attribute pattern across chart types so YAML is consistent and easier to generate.
- On-demand result-set profiling in chart decisions pipeline — Expand the ColumnProfile in decisions.py _profile_columns to compute richer statistical characteristics on-demand from…
- Support bar-line combo charts (mixed marks per Y series) — Finance and ops dashboards often need one categorical or temporal x with multiple quantitative columns where some serie…
Launch scope for the YAML contract, compiler/normalizer, execution adapters, and release/versioning is complete, externally explainable, and supportable: user-facing behavior is stable, documentation is publishable, and operational ownership is explicit. Remaining gaps are non-blocking, risk-assessed, and tracked as post-launch follow-up rather than unresolved launch debt.
- Launch docs and external readiness — Publish external-facing documentation and examples for execution/runtime adapters that are executable by new users.
- Launch operations and reliability readiness — Finalize operational readiness for versioning and migrations: telemetry, alerting, support ownership, and incident play…
- Public launch scope completion — Complete launch-critical scope for YAML contract and normalizer with production-safe behavior and rollback clarity.
- Publish dbt-core release — Release dbt-core package with install docs, compatibility matrix, and changelog notes.
- Decide final dft product and CLI naming for public launch — Choose the final external name for dft/dataface before M3 launch, reconciling product positioning, CLI/docs impact, and…
- Decide markdown report schema naming for boards versus faces — Settle the frontmatter and embed naming for markdown report objects before M3 ships, so report syntax, reusable grouped…
- Declarative schema definition outside Python code — Define the Dataface YAML spec as a declarative schema document outside of Python Pydantic models — a YAML or JSON schem…
- Scaffold faces index md and hello yaml in dft init Waiting on decide-markdown-report-schema-naming-for-boards-versus-faces, support-markdown-report-pages-in-faces-with-frontmatter-and-chart-board-embeds — Extend dft init to create a helpful /faces landing page plus hello.yml together, so a fresh repo serves a self-guided s…
- Support markdown report pages in faces with frontmatter and chart board embeds Waiting on decide-markdown-report-schema-naming-for-boards-versus-faces — Add .md Dataface report pages under /faces with YAML frontmatter plus explicit chart and board embeds, compiling them i…
- [P3] Support raw Vega-Lite specs in Dataface YAML — Allow raw Vega-Lite blocks in Dataface YAML for advanced use cases while preserving safety and compatibility checks.
- Discuss authored chart width, height, and aspect ratio controls — Evaluate whether Dataface should expose chart-level width, height, and possibly aspect ratio in authored YAML for Vega-…
- Extensible schema with custom elements and chart types — Enable users and plugin authors to extend the Dataface schema with custom element types — custom chart types, form inpu…
- Wire up batch query prefetch before render — Add a prefetch step at the top of render that collects all visible chart queries, uses the existing batch.py infrastruc…
Post-launch stabilization is complete for the YAML contract, compiler/normalizer, execution adapters, and release/versioning: recurring incidents are reduced, support burden is lower, and quality gates are enforced consistently before release. The team has a repeatable operating model for maintenance, regression prevention, and measured reliability improvements.
- Regression prevention and quality gates — Add or enforce regression gates around execution/runtime adapters so release quality is sustained automatically.
- Sustainable operating model — Document and adopt sustainable operating model for versioning and migrations across support, triage, and release cadenc…
- v1.0 stability and defect burn-down — Run stability program for YAML contract and normalizer with recurring defect burn-down and reliability trend tracking.
- Async prefetch with future-based cache for progressive rendering — Extend the batch prefetch system to support non-blocking async execution. Replace the simple dict cache in Executor wit…
v1.2 delivers meaningful depth improvements in the YAML contract, compiler/normalizer, execution adapters, and release/versioning based on observed usage and retention signals, not just roadmap intent. Enhancements improve real customer outcomes, and release readiness is demonstrated through metrics, regression coverage, and clear migration guidance where relevant.
- Quality and performance improvements — Ship measurable quality/performance improvements in execution/runtime adapters tied to user-facing outcomes.
- v1.2 depth expansion — Deliver depth expansion in YAML contract and normalizer prioritized by observed usage and retention outcomes.
- v1.2 release and migration readiness — Prepare v1.2 release/migration readiness for versioning and migrations, including communication and upgrade guidance.
- Add optional markdown-svg raw HTML foreignObject passthrough CompletedPR #756PR at 2026-03-24T00:57:44-07:00 — Add an opt-in markdown-svg configuration flag, off by default, that allows raw HTML embedded in markdown to be rendered…
- Build and release Dataface markdown integrations for MkDocs and popular doc tools Waiting on decide-markdown-report-schema-naming-for-boards-versus-faces, decide-final-dft-product-and-cli-naming-for-public-launch, support-markdown-report-pages-in-faces-with-frontmatter-and-chart-board-embeds — After the markdown report contract is stable, ship supported integrations for MkDocs and other common markdown/doc ecos…
Long-horizon opportunities for the YAML contract, compiler/normalizer, execution adapters, and release/versioning are captured as concrete hypotheses with user impact, prerequisites, and evaluation criteria. Ideas are ranked by strategic value and feasibility so future investment decisions can be made quickly with less rediscovery.
- Experiment design for future bets — Design validation experiments for versioning and migrations so future bets can be tested before major investment.
- Future opportunity research — Capture long-horizon opportunities for YAML contract and normalizer with user impact and strategic fit.
- Prerequisite and dependency mapping — Map enabling prerequisites and dependencies for execution/runtime adapters to reduce future startup cost.