Bootstrap dft init for one-command serve in dbt repos

Problem

Serving a real dbt analytics repo currently requires long commands with explicit face paths, project-dir, and adapter environment setup. After initialization, users should be able to run dft serve from repo root with sensible defaults.

This task should define: 1. What dft init should scaffold vs what should be autodetected. 2. How to safely detect dbt project and faces locations and persist defaults. 3. How to ensure BigQuery/dbt behavior respects dbt target + generate_schema_name. 4. Migration guidance for existing repos like Fivetran/analytics.

Required guardrails for this task: 1. Keep faces/ as the default face root for now (no required custom config for basic usage). 2. Make dft serve follow dbt conventions by default for profile + target resolution, rather than introducing a parallel warehouse config path. 3. Treat dbt as the source of truth for adapter/auth/project context; dft-specific config should focus on serving/runtime behavior only.

Context

What already exists

• dft init (dataface/cli/commands/init.py, dataface/cli/main.py) — Creates faces/, faces/partials/, starter faces/hello.yml. Idempotent; no dataface.yml yet. Related completed planning: tasks/workstreams/dashboard-factory/tasks/add-dft-init-for-dbt-native-repo-bootstrap.md.
• dft serve (dataface/cli/commands/serve.py, dataface/core/serve/server.py) — create_server(..., project_dir=None) already defaults project_dir to Path.cwd(), so from repo root, URLs like /faces/hello/ resolve to ./faces/hello.yml if the user runs dft serve from the dbt project root.
• Render context discovery (dataface/core/project_roots.py) — Walks up from the face file to find dataface.yml / _sources.yaml / dbt_project.yml for compile/execute context.
• Execution — When dbt_project.yml is found, DbtAdapter is registered (dataface/core/serve/server.py). It resolves profile from dbt_project.yml and uses target_name default dev unless constructor args override (dataface/core/execute/adapters/dbt_adapter.py). SqlAdapter is still registered with CLI dialect default duckdb for non-dbt query paths.

Gaps this task closes

1. No persisted “repo anchor” — Users still rely on remembering cwd; monorepos or running from subdirs break without --project-dir. A minimal dataface.yml (or agreed marker file) can anchor discovery and optional serve defaults without duplicating warehouse secrets.
2. dialect=duckdb default — Harmless for pure DbtAdapter paths but wrong for anything that falls through to SqlAdapter with profile_type=dialect in a BigQuery/Snowflake repo unless the user passes flags every time.
3. dbt target / schema behavior — Analysts expect DBT_TARGET, profiles.yml, and macros like generate_schema_name to match dbt run / dbt compile. Today serve does not document or consistently thread target override into DbtAdapter (CLI has no --target).
4. dft init output — Next steps say dft serve but do not create config that encodes “this repo is dbt-native; prefer dbt execution” or optional non-secret defaults.

Files likely touched (implementation phase)

Constraints (from problem statement)

• Default face root remains faces/ for basic usage.
• No parallel warehouse config — credentials and database identity stay in dbt profiles.yml / env; dataface.yml only holds Dataface/runtime/serving preferences.
• dbt is source of truth for adapter/auth/project context; Dataface config complements, not replaces.

Possible Solutions

Option A — Docs only

• Document: always cd to dbt root, run dft serve, pass --dialect bigquery (and --connection when needed).
• Pros: No code. Cons: Does not meet “one command”; repeats friction; easy to get wrong.

Option B — dft init writes dataface.yml; dft serve reads only that file

• Pros: Explicit anchor. Cons: Temptation to put warehouse settings in YAML; duplicates dbt; conflicts with guardrails unless schema is tightly limited.

Option C — Hybrid: minimal dataface.yml + dbt-first serve defaults (Recommended)

• dft init writes a small dataface.yml when missing (never overwrite without --force), e.g. commented template with:
• serve or top-level keys allowed only for: default face root (faces — default), optional dbt target hint (non-secret string), optional flags that map to existing dbt env conventions (DBT_TARGET, DBT_PROFILES_DIR) in docs rather than duplicating secrets.
• dft serve resolution order (design to implement): 1. Explicit --project-dir wins. 2. Else walk up from cwd for dataface.yml / dbt_project.yml to set repo root (same spirit as discover_render_context). 3. Dialect for SqlAdapter: if a dbt project is found, infer adapter type from resolved dbt profile target (read-only introspection) or from an allowed key in dataface.yml only if it cannot be inferred — avoid requiring users to pass --dialect for standard dbt repos. 4. DbtAdapter target: honor DBT_TARGET / dbt_project’s target when consistent with dbt CLI; optional --target on dft serve as escape hatch.
• Pros: Matches guardrails; one-command serve from root; monorepo-friendly; escape hatches stay explicit. Cons: Requires careful config schema + tests so dataface.yml does not become a second profiles file.

Plan

Phase 0 — Design spec (before coding)

1. Write a short RFC or section in this task listing allowed dataface.yml keys for M1 (serve + paths only). Explicitly forbidden in that list: connection strings, passwords, service account JSON paths (prefer GOOGLE_APPLICATION_CREDENTIALS / dbt’s own mechanisms).
2. Define resolution algorithm for “repo root” when both dbt_project.yml and dataface.yml exist in different directories (edge case: faces/ only under package subfolder). Default rule: prefer directory containing dbt_project.yml as serve root; dataface.yml beside it is canonical.
3. Define behavior when cwd is not repo root — error message vs walk-up vs require --project-dir.

Phase 1 — dft init alignment

1. Extend init_command to create dataface.yml when absent (idempotent; skip if exists unless --force with clear policy on merge vs overwrite).
2. Template content: version comment, faces root default, pointer to docs for DBT_TARGET / profiles, optional commented serve: block if needed.
3. Update dft init tests: assert file created; re-run no-op; --force behavior.

Phase 2 — dft serve defaults

1. Implement project root discovery from cwd (reuse or extend project_roots helpers to avoid drift with compile paths).
2. Wire discovered root into create_server when --project-dir omitted.
3. Infer SqlAdapter dialect from dbt profile type when dbt project present; keep --dialect as override.
4. Pass dbt target into DbtAdapter from DBT_TARGET env and optional --target; document interaction with dbt_project.yml default target.

Phase 3 — BigQuery / generate_schema_name

1. Verify (add integration or smoke test if feasible) that queries executed via DbtAdapter use the same adapter instance / schema logic as dbt for ref() / source() (macro resolution path). If a gap exists, file a sub-issue or fix in the same PR with a regression test.
2. Document that raw SQL in faces must respect compiled schema names (same as dbt); link to dbt docs on custom schemas.

Phase 4 — Migration (e.g. Fivetran/analytics)

1. Runbook: dft init in repo root → review generated dataface.yml → set DBT_TARGET as needed → dft serve from root.
2. Note any analytics-repo-specific paths (if dashboards live outside default faces/) and whether --project-dir or a config key is required — prefer one blessed layout for M1.

Phase 5 — Docs and CLI help

1. Update installation / CLI reference: “after init, from repo root: dft serve”.
2. Document env vars: DBT_TARGET, DBT_PROFILES_DIR, and when --connection is still needed (non-dbt SQL paths, demos).

Acceptance criteria

• [ ] From a minimal dbt repo fixture (with dbt_project.yml + faces/hello.yml), dft serve with no flags (run from root) serves /faces/hello/ without requiring --project-dir or --dialect for typical dbt-backed queries.
• [ ] dft init creates dataface.yml once; safe re-run.
• [ ] DBT_TARGET (or --target) changes which warehouse target DbtAdapter uses, consistent with dbt expectations.
• [ ] Docs include migration steps for an existing repo; no recommendation to duplicate warehouse config in dataface.yml.

Out of scope (unless explicitly pulled in)

• Replacing dbt profiles with Dataface-native connection config.
• Changing default face root away from faces/ for M1.
• Cloud/playground product surfaces (CLI + local serve only unless shared code is trivial).

Implementation Progress

Phase 1 — dft init creates dataface.yml

• Added DATAFACE_YML template to dataface/cli/commands/init.py — comments-only YAML that documents serve: keys (dialect, target) and anchors the project root.
• Added "dataface.yml" as the first scaffold entry so it's created before faces/. Idempotent: skips if exists; --force overwrites.

Phase 2 — dft serve defaults (project root discovery + dialect inference + target)

Project root discovery (dataface/core/project_roots.py): - discover_project_root(start_dir) walks up from start_dir (default: cwd) looking for dataface.yml, dataface.yaml, or dbt_project.yml. Returns the directory containing the first match, or start_dir if none found. - infer_dialect_from_dbt(project_dir, target_name) reads dbt_project.yml → profile name → profiles.yml target → type field. Falls back to ~/.dbt/profiles.yml. Returns None when unresolvable.

Serve CLI (dataface/cli/commands/serve.py): - dialect default changed from "duckdb" to None. When None, auto-detects from dbt profile; falls back to "duckdb". - Added --target parameter threaded through to DbtAdapter. - serve_command now calls discover_project_root() when --project-dir omitted.

CLI main (dataface/cli/main.py): - --dialect default None (was "duckdb"), --target option added.

Server (dataface/core/serve/server.py): - create_server accepts optional target param, stored in app.state.target. - _render_face_file and _handle_inspect_route accept and thread target to DbtAdapter(target_name=target).

DbtAdapter (dataface/core/execute/adapters/dbt_adapter.py): - target_name resolution: explicit arg → DBT_TARGET env → "dev".

Tests (23 new, 143 total related tests passing)

tests/core/test_bootstrap_serve.py: - TestDiscoverProjectRoot — 6 tests (dbt root, subdir walk-up, dataface.yml, both markers, nothing found, cwd default) - TestDbtAdapterTarget — 4 tests (default dev, explicit, env var, explicit overrides env) - TestInitDatafaceYml — 5 tests (create, skip existing, force, valid YAML, non-dbt repo) - TestInferDialectFromDbt — 6 tests (bigquery, duckdb, specific target, no dbt project, no profiles, home profiles fallback) - TestServeProjectRootDiscovery — 2 tests (server uses discovered root, threads target)

Acceptance criteria status

• [x] dft serve with no flags from repo root serves faces without --project-dir or --dialect
• [x] dft init creates dataface.yml once; safe re-run
• [x] DBT_TARGET (or --target) changes which target DbtAdapter uses
• [ ] Docs include migration steps (Phase 4–5, deferred to follow-up)

QA Exploration

N/A — CLI-only task, no UI/browser surfaces affected.

• [x] QA exploration completed (or N/A for non-UI tasks)

Review Feedback

• [ ] Review cleared