Dataface Extensions & Plugins System

Version: 2.0 (Python) Status: Design Phase

---

Overview

Dataface is designed to be extensible at multiple points, allowing users to:

1. Add custom chart types (visualizations)
2. Add custom variable controls (inputs/filters)
3. Add custom data sources (adapters)
4. Add custom actions (on-click, on-submit behaviors)
5. Add custom transforms (data processing)

This enables Dataface to grow beyond dashboards into a full data applications platform.

---

Extension Points

1. Chart Types (Visualizations)

Purpose: Add custom visualizations beyond built-in types (line, bar, area, etc.)

Use cases: - Domain-specific visualizations (gene sequencing, network graphs, maps) - Custom branding (company-specific chart styles) - Specialized analytics (cohort analysis, funnels, etc.)

Example:

# In dashboard YAML
charts:
  funnel_chart:
    type: custom:conversion_funnel  # Extension prefix "custom:"
    query: q_conversions
    steps: [signup, activation, purchase]
    plugin: dataface-plugin-funnel

Plugin structure:

# dataface_plugin_funnel/__init__.py
from dataface.extensions import ChartExtension
import altair as alt

class ConversionFunnelChart(ChartExtension):
    """Custom funnel chart extension"""

    type_name = "conversion_funnel"

    def generate_vega_spec(self, chart_config, data):
        """Generate Vega-Lite spec for funnel chart"""
        return {
            "$schema": "https://vega.github.io/schema/vega-lite/v5.json",
            "data": {"values": data},
            "mark": {"type": "bar", "cornerRadiusEnd": 4},
            "encoding": {
                "x": {"field": "step", "type": "ordinal"},
                "y": {"field": "count", "type": "quantitative"},
                "color": {"field": "step", "scale": {"scheme": "blues"}}
            }
        }

    def validate_config(self, chart_config):
        """Validate chart configuration"""
        if "steps" not in chart_config:
            raise ValueError("Funnel chart requires 'steps' field")

# Register extension
def register_extension():
    return ConversionFunnelChart()

Installation:

pip install dataface-plugin-funnel

Registration (automatic via entry points):

# In plugin's setup.py / pyproject.toml
[project.entry-points."dataface.charts"]
conversion_funnel = "dataface_plugin_funnel:register_extension"

---

2. Variable Controls (Inputs)

Purpose: Add custom input types beyond built-in controls (select, daterange, etc.)

Use cases: - Star ratings, emoji pickers - Color pickers, icon selectors - Advanced date pickers (fiscal calendar) - Multi-step wizards - Custom validation with JavaScript

Key Design Decision: No Python Required

After analysis, we determined that all custom control needs can be met with HTML/CSS/JS:

Need	Solution
Custom data fetching	Use options.query with existing adapters
Validation	Client-side JavaScript
External APIs	Client-side JS (Maps API, date libraries, etc.)
Complex transformations	Jinja2 filters in templates

This keeps the barrier to entry very low - anyone who knows HTML/CSS/JS can create controls.

Example - Add a star rating control:

my_project/
└── controls/
    └── rating/
        ├── rating.html
        ├── rating.css    # Optional
        └── rating.js     # Optional

rating.html (Jinja2 template):

<div class="dft-variable-control dft-variable dft-variable-rating"
     data-variable-id="{{ name }}"
     data-value="{{ value or default or 0 }}">

  <label class="dft-variable-label">{{ label }}:</label>

  <div class="dft-rating-stars">
    {% for i in range(1, 6) %}
    <button type="button"
            class="dft-rating-star{% if i <= (value or 0) %} active{% endif %}"
            onclick="updateVariable('{{ name | js_escape }}', {{ i }})">
      ★
    </button>
    {% endfor %}
  </div>
</div>

rating.css (uses CSS custom properties from theme):

.dft-variable-rating .dft-rating-stars { display: flex; gap: 2px; }
.dft-variable-rating .dft-rating-star {
  background: none;
  border: none;
  font-size: 20px;
  cursor: pointer;
  color: var(--dft-muted-color, #d0d0d0);
  transition: color 0.15s, transform 0.1s;
}
.dft-variable-rating .dft-rating-star:hover { transform: scale(1.15); }
.dft-variable-rating .dft-rating-star.active { color: var(--dft-accent-color, #f5a623); }

Usage in dashboard:

variables:
  satisfaction:
    label: Satisfaction
    input: rating        # Matches controls/rating/rating.html
    default: 3

Flexible Directory Organization:

controls/
├── rating/              # Self-contained directory (recommended)
│   ├── rating.html
│   ├── rating.css
│   └── rating.js
├── toggle.html          # Flat single file
├── toggle.css
├── pickers/             # Categorized with namespace
│   ├── color/
│   │   └── color.html
│   └── icon/
│       └── icon.html
└── _shared.css          # Underscore = ignored (for shared imports)

Resolution order: 1. Project controls ({project}/controls/) 2. Built-in controls (dataface/render/templates/controls/) 3. Fallback to readonly display

Template Context:

Every control template receives:

{
    "name": str,           # Variable name
    "label": str,          # Display label
    "value": Any,          # Current value
    "default": Any,        # Default value
    "options": [           # For select/multiselect
        {"value": str, "label": str, "selected": bool}
    ],
    "min": Optional[float],    # For number/slider
    "max": Optional[float],
    "step": Optional[float],
    "config": dict,        # Full variable config
}

Custom Jinja Filters: - {{ name | js_escape }} - Escape for JavaScript string context - {{ value | tojson }} - Convert to JSON - {{ label | e }} - HTML escape (automatic with autoescape)

See also: plans/features/VARIABLE_CONTROLS_REFACTOR.md for detailed implementation

---

3. Data Sources (Adapters)

Purpose: Add support for querying non-dbt data sources

Use cases: - REST APIs - Cloud services (Salesforce, Stripe, etc.) - Time-series databases (InfluxDB, Prometheus) - Custom data warehouses

Example:

queries:
  q_stripe_revenue:
    adapter: stripe  # Custom adapter
    metrics: [mrr, arr, churn_rate]
    filters:
      date_range: "{{ date_range }}"

Plugin structure:

# dataface_plugin_stripe/__init__.py
from dataface.extensions import AdapterExtension
import stripe

class StripeAdapter(AdapterExtension):
    """Query Stripe API"""

    adapter_name = "stripe"

    def __init__(self):
        self.api_key = os.getenv("STRIPE_API_KEY")
        stripe.api_key = self.api_key

    async def execute_query(self, query, variables):
        """Execute query against Stripe API"""
        # Fetch charges
        charges = stripe.Charge.list(
            created={
                'gte': variables['date_range']['start'],
                'lte': variables['date_range']['end']
            },
            limit=query.limit
        )

        # Transform to standard format
        rows = []
        for charge in charges.data:
            rows.append({
                'date': charge.created,
                'amount': charge.amount / 100,
                'currency': charge.currency,
                'customer': charge.customer
            })

        return rows

    def validate_query(self, query):
        """Validate query structure"""
        # Stripe-specific validation
        pass

def register_extension():
    return StripeAdapter()

---

4. Actions (Behaviors)

Purpose: Add custom behaviors for chart interactions and form submissions

Use cases: - Write to databases - Trigger workflows - Send notifications - Call external APIs

Example:

charts:
  clickable_chart:
    query: q_products
    type: bar
    interactions:
      click:
        action: custom:add_to_cart  # Custom action
        plugin: dataface-plugin-ecommerce

sections:
  - title: "Order Form"
    items:
      order_form:
        type: form
        fields: [...]
        on_submit:
          action: custom:create_order  # Custom action
          plugin: dataface-plugin-ecommerce

Plugin structure:

# dataface_plugin_ecommerce/__init__.py
from dataface.extensions import ActionExtension

class AddToCartAction(ActionExtension):
    """Add clicked product to cart"""

    action_name = "add_to_cart"

    async def execute(self, context, data):
        """Execute action with clicked data"""
        product_id = data['product_id']
        user_id = context.user_id

        # Insert to database
        await context.db.execute(
            "INSERT INTO cart_items (user_id, product_id) VALUES (?, ?)",
            (user_id, product_id)
        )

        return {"success": True, "message": "Added to cart"}

class CreateOrderAction(ActionExtension):
    """Create order from form submission"""

    action_name = "create_order"

    async def execute(self, context, form_data):
        """Execute action with form data"""
        # Validate
        if not form_data.get('customer'):
            raise ValueError("Customer is required")

        # Insert order
        order_id = await context.db.execute(
            "INSERT INTO orders (...) VALUES (...)",
            form_data
        )

        # Send notification
        await send_email(form_data['customer'], f"Order {order_id} created")

        return {"success": True, "order_id": order_id}

def register_extensions():
    return [AddToCartAction(), CreateOrderAction()]

---

5. Data Transforms (Processing)

Purpose: Add custom data transformations between query and visualization

Use cases: - Pivot/unpivot data - Calculate derived metrics - Aggregate across time periods - Apply business logic

Example:

charts:
  cohort_chart:
    query: q_users
    type: custom:cohort_matrix
    transform: custom:cohortify  # Custom transform
    plugin: dataface-plugin-cohort
    config:
      cohort_field: signup_date
      event_field: purchase_date
      period: month

Plugin structure:

# dataface_plugin_cohort/__init__.py
from dataface.extensions import TransformExtension
import pandas as pd

class CohortifyTransform(TransformExtension):
    """Transform user events into cohort analysis"""

    transform_name = "cohortify"

    def transform(self, data, config):
        """Transform raw data into cohort matrix"""
        df = pd.DataFrame(data)

        # Calculate cohort month
        df['cohort'] = df[config['cohort_field']].dt.to_period('M')
        df['event'] = df[config['event_field']].dt.to_period('M')
        df['period'] = (df['event'] - df['cohort']).apply(lambda x: x.n)

        # Build cohort matrix
        cohort_data = df.groupby(['cohort', 'period']).size().reset_index()

        return cohort_data.to_dict('records')

def register_extension():
    return CohortifyTransform()

---

Extension Discovery & Loading

Auto-discovery (Python Entry Points)

Dataface automatically discovers extensions via Python entry points:

# Plugin's pyproject.toml
[project.entry-points."dataface.charts"]
my_chart = "my_plugin:register_chart_extension"

[project.entry-points."dataface.adapters"]
my_adapter = "my_plugin:register_adapter_extension"

[project.entry-points."dataface.actions"]
my_action = "my_plugin:register_action_extension"

[project.entry-points."dataface.transforms"]
my_transform = "my_plugin:register_transform_extension"

Note: Variable controls use file-based discovery (HTML/CSS/JS in controls/ directory), not Python entry points. This makes them much easier to create.

Manual Registration

# In Python code
from dataface.extensions import register_extension
from my_plugin import MyCustomChart

register_extension("charts", "my_chart", MyCustomChart())

Declaration in Dashboard YAML

extensions:
  - id: dataface-plugin-funnel
    version: "1.0.0"
    source: pypi  # or github, local
  - id: dataface-plugin-stripe
    version: "2.1.0"
    source: pypi

---

Extension Base Classes

ChartExtension

from abc import ABC, abstractmethod
from typing import Dict, List

class ChartExtension(ABC):
    """Base class for custom chart types"""

    @property
    @abstractmethod
    def type_name(self) -> str:
        """Unique identifier for this chart type (e.g., 'funnel')"""
        pass

    @abstractmethod
    def generate_vega_spec(self, chart_config: Dict, data: List[Dict]) -> Dict:
        """Generate Vega-Lite JSON spec from chart config and data"""
        pass

    def validate_config(self, chart_config: Dict) -> None:
        """Validate chart configuration (optional)"""
        pass

    def transform_data(self, data: List[Dict], chart_config: Dict) -> List[Dict]:
        """Optional data transformation before rendering"""
        return data

Variable Controls (File-Based - No Python Class)

Note: Variable controls use a file-based approach (HTML/CSS/JS) instead of Python classes. This dramatically lowers the barrier to entry - no Python knowledge required.

To add a custom control:

1. Create controls/{name}/{name}.html in your project
2. Optionally add {name}.css and {name}.js (auto-loaded)
3. Use input: {name} in your YAML

Built-in controls are at dataface/render/templates/controls/: - select.html, checkbox.html, slider.html, text.html - number.html, date.html, daterange.html, readonly.html

See: plans/features/VARIABLE_CONTROLS_REFACTOR.md for full details

AdapterExtension

class AdapterExtension(ABC):
    """Base class for custom data adapters"""

    @property
    @abstractmethod
    def adapter_name(self) -> str:
        """Unique identifier for this adapter"""
        pass

    @abstractmethod
    async def execute_query(self, query: Dict, variables: Dict) -> List[Dict]:
        """Execute query and return rows"""
        pass

    def validate_query(self, query: Dict) -> None:
        """Validate query structure (optional)"""
        pass

ActionExtension

class ActionExtension(ABC):
    """Base class for custom actions"""

    @property
    @abstractmethod
    def action_name(self) -> str:
        """Unique identifier for this action"""
        pass

    @abstractmethod
    async def execute(self, context: Dict, data: Dict) -> Dict:
        """Execute action with context and data"""
        pass

TransformExtension

class TransformExtension(ABC):
    """Base class for custom transforms"""

    @property
    @abstractmethod
    def transform_name(self) -> str:
        """Unique identifier for this transform"""
        pass

    @abstractmethod
    def transform(self, data: List[Dict], config: Dict) -> List[Dict]:
        """Transform data"""
        pass

---

Data Apps: Full Example

Goal: Build an order management data app with forms, charts, and custom actions.

name: order_management
title: "Order Management App"

extensions:
  - id: dataface-plugin-ecommerce
    version: "1.0.0"

variables:
  order_status:
    data_type: string
    input: select
    options:
      static: ["pending", "confirmed", "shipped", "delivered"]
    default: "pending"

queries:
  q_orders:
    model: ref('orders')
    dimensions: [order_id, customer, product, quantity, status, created_at]
    filters:
      status: order_status

  q_products:
    model: ref('products')
    dimensions: [product_id, name, price, stock]

cols:
  # Section 1: Create Order Form
  - title: "Create New Order"
    items:
      order_form:
        type: form  # Custom component
        fields:
          - name: customer
            type: text
            required: true
          - name: product
            type: select
            options:
              query: q_products
              value_field: product_id
              display_field: name
          - name: quantity
            type: number
            min: 1
            default: 1
        on_submit:
          action: custom:create_order  # Custom action
          table: orders
          refresh_queries: [q_orders]

  # Section 2: Orders Dashboard
  - title: "Recent Orders"
    charts:
      orders_table:
        title: "Orders by Status"
        query: q_orders
        type: table
        interactions:
          click:
            action: custom:view_order_details  # Custom action

This creates a full data application where users can: 1. Fill out a form to create orders 2. View orders in a table 3. Click orders to see details 4. Filter by status

All extensible through plugins!

---

Security Considerations

Sandboxing

Extensions run in the same Python process, so trust is required. Future versions may add:

• Permission system (what data can extension access?)
• Code review process for public plugins
• Isolated execution (containers, separate processes)

Data Access

Extensions should not have direct database access unless explicitly granted:

# In action extension
async def execute(self, context, data):
    # context.db is provided by Dataface (controlled access)
    # Cannot import arbitrary database libraries
    await context.db.execute(query)

---

Plugin Distribution

PyPI (Recommended)

pip install dataface-plugin-funnel
pip install dataface-plugin-stripe

GitHub

extensions:
  - id: my-custom-plugin
    source: github:username/dataface-plugin-custom
    version: main

Local Development

extensions:
  - id: my-dev-plugin
    source: local:./plugins/my-dev-plugin

---

Extension Registry (Future)

Planned: Official extension registry at https://extensions.dataface.io

• Browse extensions by category
• See usage stats, ratings
• Security reviews
• Version compatibility matrix

---

Summary

Dataface's extension system enables:

1. ✅ Custom visualizations - Any chart type you need (Python + Vega-Lite)
2. ✅ Custom inputs - Advanced variable controls (HTML/CSS/JS only - no Python!)
3. ✅ Custom data sources - Query anything (Python adapters)
4. ✅ Custom actions - Write data, trigger workflows (Python)
5. ✅ Custom transforms - Process data your way (Python)

This makes Dataface a platform for building full data applications, not just dashboards.

Extension points and complexity:

Extension Type	Files Required	Python?	Barrier
Variable Controls	HTML + CSS + JS	No	Very Low
Charts	Python + Vega spec	Yes	Medium
Adapters	Python class	Yes	Medium
Actions	Python class	Yes	Medium
Transforms	Python class	Yes	Medium

Key insight: Variable controls are the most commonly extended, so we made them the easiest. Just drop HTML/CSS/JS files in a controls/ directory - no Python, no registration, no plugins.

For complex extensions (charts, adapters, actions, transforms), Python plugins with entry points provide power and flexibility.

The future: A rich ecosystem of community extensions turning Dataface into the "WordPress of data apps."