Dataface Tasks
    Heartbeat snapshot (JSON)

    evaluate snowflake semantic views and lineage as context sources

    IDCONTEXT_CATALOG_NIMBLE-EVALUATE_SNOWFLAKE_SEMANTIC_VIEWS_AND_LINEAGE_AS_CONTEXT_SOURCES
    Statusnot_started
    Priorityp3
    Milestonem5-v1-2-launch
    Ownerdata-ai-engineer-architect

    Problem

    Investigate whether Snowflake semantic views, Cortex Analyst instructions, and GET_LINEAGE metadata should be ingested as provenance-aware context inputs in Dataface alongside inspect profiles, dbt metadata, and database comments. Define what to ingest, how to map it into the context stack without collapsing it into plain comments, and what an M5 implementation path would look like.

    Context

    • Dataface already combines inspect profiles, dbt metadata, and comments, but it does not yet distinguish provenance-rich warehouse-native semantic assets from plain descriptive text.
    • Snowflake semantic views, Cortex Analyst instructions, and GET_LINEAGE could add entity definitions, business semantics, and upstream/downstream lineage that current inspect-only context misses.
    • This should stay evaluation-first for M5: define ingest value, mapping rules, and trust/provenance handling before adding a broad new ingestion surface.

    Possible Solutions

    • A - Treat all Snowflake metadata as plain comments in the existing context schema: fast, but it loses provenance and source-specific semantics.
    • B - Recommended: define a provenance-aware source model for semantic views, analyst instructions, and lineage edges, then map only the highest-value fields into the context stack with explicit attribution.
    • C - Defer all Snowflake-native ingestion until later and rely only on dbt plus inspect: narrower scope, but it may miss useful warehouse-native semantics customers already maintain.

    Plan

    1. Inventory the specific Snowflake metadata surfaces and list which fields are worth ingesting versus intentionally ignoring.
    2. Define how each source would map into the context stack, including provenance, freshness, and consumer-facing semantics.
    3. Compare that model against existing inspect/dbt/comment inputs and call out overlap, conflict, and trust-order rules.
    4. Write a concrete implementation recommendation for the first supported scope, code touchpoints, and rollout risks.

    Implementation Progress

    QA Exploration

    • [ ] QA exploration completed (or N/A for non-UI tasks)

    Review Feedback

    • [ ] Review cleared