Skip to main content

Data Service API

The Data Service API provides a powerful, flexible system for managing dynamic data structures using Frictionless Table Schema. It follows a hierarchical structure: App → DataSource → DataTable.

Overview​

The Data Service enables you to:

  • Create Apps: Top-level containers for organizing data
  • Define DataSources: Collections of tables with specific provider types (PostgreSQL, JSONB, MongoDB)
  • Manage DataTables: Define table structures using Frictionless Table Schema
  • Validate Schemas: Automatic validation of table schemas
  • Track Operations: Audit trail for all schema operations

Architecture​

graph TD
    A[App] -->|has many| B[DataSource]
    B -->|has many| C[DataTable]
    C -->|defines| D[Frictionless Schema]
    C -->|materializes to| E[Physical Table]
    B -->|uses| F[Provider: PostgreSQL/JSONB/MongoDB]
    C -->|tracked by| G[SchemaOperation]

Key Concepts​

App​

A top-level container that groups related data sources. Each app has:

  • Name & Slug: Human-readable and URL-safe identifiers
  • Description: Optional documentation
  • Multiple DataSources: Can contain different provider types

DataSource​

A collection of tables using a single provider type:

  • Provider Types:
    • flat_table: Traditional PostgreSQL tables with full schema enforcement and Alembic migrations
    • jsonb: PostgreSQL JSONB storage with flexible schema-less structure (id + data JSONB)
    • mongodb: MongoDB collections (planned - not yet implemented)
  • Provider Config: JSON configuration for provider-specific settings
  • Isolated: Each datasource is independent
  • Schema Management: Each provider handles schema changes differently:
    • flat_table: Uses Alembic to generate ALTER TABLE migrations
    • jsonb: Metadata-only updates (physical table structure never changes)
    • mongodb: Schema-less (no physical migrations needed)

DataTable​

Defines a table structure using Frictionless Table Schema:

  • Name: Used in API URLs (e.g., /datatables/users)
  • Schema: Frictionless Table Schema (JSON format)
  • Physical Table Name: Auto-generated (e.g., myapp_users)
  • Materialization: Can be created as actual database tables
  • Schema Hash: SHA256 hash for cache invalidation

Base URL Structure​

All Data Service endpoints follow this app-centric pattern:

/api/apps/                                                    # App management
/api/apps/{app_slug}/datasources/                            # DataSource management
/api/apps/{app_slug}/datasources/{datasource_slug}/datatables/   # DataTable management
/api/apps/{app_slug}/datatables/                             # Cross-datasource queries

Quick Start​

1. Create an App​

POST /api/apps/
Content-Type: application/json
Authorization: Bearer YOUR_ACCESS_TOKEN

{
  "name": "My Application",
  "description": "My awesome application"
}

Response:

{
  "id": 1,
  "name": "My Application",
  "slug": "my-application",
  "description": "My awesome application",
  "datasource_count": 0,
  "table_count": 0,
  "created_at": "2024-10-30T12:00:00Z"
}

2. Create a DataSource​

POST /api/apps/my-application/datasources/
Content-Type: application/json
Authorization: Bearer YOUR_ACCESS_TOKEN

{
  "name": "Users Database",
  "slug": "users-db",
  "provider_type": "flat_table",
  "description": "User data storage"
}

3. Create a DataTable​

POST /api/apps/my-application/datasources/users-db/datatables/
Content-Type: application/json
Authorization: Bearer YOUR_ACCESS_TOKEN

{
  "name": "users",
  "description": "User accounts table",
  "json_schema": {
    "fields": [
      {
        "name": "id",
        "type": "integer",
        "constraints": {
          "required": true,
          "unique": true
        }
      },
      {
        "name": "email",
        "type": "string",
        "format": "email",
        "constraints": {
          "required": true
        }
      },
      {
        "name": "name",
        "type": "string",
        "constraints": {
          "required": true
        }
      },
      {
        "name": "created_at",
        "type": "datetime"
      }
    ],
    "primaryKey": ["id"]
  }
}

Features​

Schema Validation​

All DataTable schemas are validated using Frictionless Table Schema specification:

  • Field Types: string, integer, number, boolean, datetime, date, time, year, etc.
  • Constraints: required, unique, minimum, maximum, pattern, enum
  • Foreign Keys: Reference other tables
  • Primary Keys: Define unique identifiers

See Frictionless Table Schema Guide for details.

Cross-DataSource Queries​

Query all tables across all datasources for an app:

GET /api/apps/my-application/datatables/

Filter by datasource:

GET /api/apps/my-application/datatables/?datasource=users-db

Filter by materialization status:

GET /api/apps/my-application/datatables/?is_materialized=true

Statistics & Analytics​

Get comprehensive statistics:

# App-level stats
GET /api/apps/my-application/stats/

# DataSource-level stats
GET /api/apps/my-application/datasources/users-db/stats/

# Cross-datasource stats
GET /api/apps/my-application/datatables/stats/

Soft Delete​

All resources support soft deletion (marked as inactive):

# Delete app (also inactivates all datasources and datatables)
DELETE /api/apps/my-application/

# Delete datasource
DELETE /api/apps/my-application/datasources/users-db/

# Delete datatable
DELETE /api/apps/my-application/datasources/users-db/datatables/users/

Schema-Only Operations​

Get just the Frictionless schema without full table details:

GET /api/apps/my-application/datasources/users-db/datatables/users/schema/

Rate Limiting​

All endpoints are rate-limited for security:

  • Standard Limit: 100 requests per minute
  • Burst Limit: 10 requests for write operations
  • Authentication: Rate limiting per authenticated user

Tenant Isolation​

All Data Service operations are tenant-scoped:

  • Data is completely isolated per tenant
  • No cross-tenant data access
  • Automatic tenant detection from domain/subdomain

Next Steps​

Examples​

Complete Workflow​

See the API Endpoint Reference for comprehensive examples of:

  • CRUD operations for all resources
  • Schema updates and validation
  • Data validation against schemas
  • Error handling and responses
  • Filtering and pagination