Data Service API

The Data Service API provides a powerful, flexible system for managing dynamic data structures using Frictionless Table Schema. It follows a hierarchical structure: App → DataSource → DataTable.

Overview

The Data Service enables you to:

• Create Apps: Top-level containers for organizing data
• Define DataSources: Collections of tables with specific provider types (PostgreSQL, JSONB, MongoDB)
• Manage DataTables: Define table structures using Frictionless Table Schema
• Validate Schemas: Automatic validation of table schemas
• Track Operations: Audit trail for all schema operations

Architecture

graph TD
    A[App] -->|has many| B[DataSource]
    B -->|has many| C[DataTable]
    C -->|defines| D[Frictionless Schema]
    C -->|materializes to| E[Physical Table]
    B -->|uses| F[Provider: PostgreSQL/JSONB/MongoDB]
    C -->|tracked by| G[SchemaOperation]

Key Features

The Data Service provides enterprise-grade capabilities for data management:

• ✅ CRUD Operations: Complete Create, Read, Update, Delete API for all tables
• ✅ Advanced Querying: 20+ filter operators, full-text search, sorting, pagination
• ✅ Aggregations: SQL-like GROUP BY, HAVING, and aggregate functions (SUM, AVG, COUNT, etc.)
• ✅ Relationships: Foreign keys, JOIN support, automatic relationship population
• ✅ Hierarchies & Graphs: Organizational charts, network traversal with BFS/DFS algorithms
• ✅ Schema Migrations: Automatic schema evolution with Alembic, zero-downtime updates
• ✅ Multiple Index Types: B-tree, GIN, Hash, BRIN, unique, composite, partial indexes
• ✅ Authorization: Policy-based access control with Cerbos, row-level security
• ✅ Multiple Providers: Choose between FlatTable (normalized), JSONB (flexible), or MongoDB (planned)
• ✅ Full-Text Search: PostgreSQL text search with ranking and relevance
• ✅ Bulk Operations: Efficient batch inserts, updates, and deletes
• ✅ Soft Delete Support: Mark records as deleted without removing them
• ✅ Audit Trail: Track all schema and data changes
• ✅ Tenant Isolation: Complete data separation per organization

Key Concepts

App

A top-level container that groups related data sources. Each app has:

• Name & Slug: Human-readable and URL-safe identifiers
• Description: Optional documentation
• Multiple DataSources: Can contain different provider types

DataSource

A collection of tables using a single provider type:

• Provider Types:
  • flat_table: Traditional PostgreSQL tables with full schema enforcement and Alembic migrations
  • jsonb: PostgreSQL JSONB storage with flexible schema-less structure (id + data JSONB)
  • mongodb: MongoDB collections (planned - not yet implemented)
• Provider Config: JSON configuration for provider-specific settings
• Isolated: Each datasource is independent
• Schema Management: Each provider handles schema changes differently:
  • flat_table: Uses Alembic to generate ALTER TABLE migrations
  • jsonb: Metadata-only updates (physical table structure never changes)
  • mongodb: Schema-less (no physical migrations needed)

DataTable

Defines a table structure using Frictionless Table Schema:

• Name: Used in API URLs (e.g., /datatables/users)
• Schema: Frictionless Table Schema (JSON format)
• Physical Table Name: Auto-generated (e.g., myapp_users)
• Materialization: Can be created as actual database tables
• Schema Hash: SHA256 hash for cache invalidation

Base URL Structure

All Data Service endpoints follow this app-centric pattern:

/api/apps/                                                    # App management
/api/apps/{app_slug}/datasources/                            # DataSource management
/api/apps/{app_slug}/datasources/{datasource_slug}/datatables/   # DataTable management
/api/apps/{app_slug}/datatables/                             # Cross-datasource queries

Quick Start

Admin Operations

Steps 1-3 (creating apps, datasources, and datatables) are admin operations typically performed once during setup. These are REST API only - no SDK methods available.

The SDK is designed for data operations (CRUD, queries, relationships) after your tables are set up.

1. Create an App

POST /api/apps/
Content-Type: application/json
Authorization: Bearer YOUR_ACCESS_TOKEN

{
  "name": "My Application",
  "description": "My awesome application"
}

Response:

{
  "id": 1,
  "name": "My Application",
  "slug": "my-application",
  "description": "My awesome application",
  "datasource_count": 0,
  "table_count": 0,
  "created_at": "2024-10-30T12:00:00Z"
}

2. Create a DataSource

POST /api/apps/my-application/datasources/
Content-Type: application/json
Authorization: Bearer YOUR_ACCESS_TOKEN

{
  "name": "Users Database",
  "slug": "users-db",
  "provider_type": "flat_table",
  "description": "User data storage"
}

3. Create a DataTable

POST /api/apps/my-application/datasources/users-db/datatables/
Content-Type: application/json
Authorization: Bearer YOUR_ACCESS_TOKEN

{
  "name": "users",
  "description": "User accounts table",
  "json_schema": {
    "fields": [
      {
        "name": "id",
        "type": "integer",
        "constraints": {
          "required": true,
          "unique": true
        }
      },
      {
        "name": "email",
        "type": "string",
        "format": "email",
        "constraints": {
          "required": true
        }
      },
      {
        "name": "name",
        "type": "string",
        "constraints": {
          "required": true
        }
      },
      {
        "name": "created_at",
        "type": "datetime"
      }
    ],
    "primaryKey": ["id"]
  }
}

4. Working with Data (SDK)

Once your tables are created, use the SDK for data operations:

Python

from taruvi import TaruviClient

# Initialize client
auth_client = TaruviClient(
    base_url="https://your-domain.com",
    jwt_token="YOUR_JWT_TOKEN"
)

# Create users
user = auth_client.database.create('users', {
    'id': 1,
    'email': 'john@example.com',
    'name': 'John Doe',
    'created_at': '2024-01-15T10:00:00Z'
}, app_slug='my-application')

# Query users
users = auth_client.database.query('users', app_slug='my-application') \
    .filter('email', 'contains', 'example.com') \
    .sort('created_at', 'desc') \
    .get()

# Update user
updated = auth_client.database.update('users', 1, {
    'name': 'John Smith'
}, app_slug='my-application')

# Delete user
auth_client.database.delete('users', 1, app_slug='my-application')

JavaScript

import { TaruviClient } from '@taruvi/sdk'

// Initialize client
const client = new TaruviClient({
    baseUrl: 'https://your-domain.com',
    jwtToken: 'YOUR_JWT_TOKEN'
})

// Create users
const user = await client.database
    .from('users')
    .create({
        id: 1,
        email: 'john@example.com',
        name: 'John Doe',
        created_at: '2024-01-15T10:00:00Z'
    })
    .execute()

// Query users
const users = await client.database
    .from('users')
    .filter('email', 'contains', 'example.com')
    .sort('created_at', 'desc')
    .execute()

// Update user
const updated = await client.database
    .from('users')
    .get('1')
    .update({ name: 'John Smith' })
    .execute()

// Delete user
await client.database
    .from('users')
    .delete('1')
    .execute()

See the Data Service Quickstart for a complete tutorial.

Features

Schema Validation

All DataTable schemas are validated using Frictionless Table Schema specification:

• Field Types: string, integer, number, boolean, datetime, date, time, year, etc.
• Constraints: required, unique, minimum, maximum, pattern, enum
• Foreign Keys: Reference other tables
• Primary Keys: Define unique identifiers

See Frictionless Table Schema Guide for details.

Cross-DataSource Queries

Query all tables across all datasources for an app:

GET /api/apps/my-application/datatables/

Filter by datasource:

GET /api/apps/my-application/datatables/?datasource=users-db

Filter by materialization status:

GET /api/apps/my-application/datatables/?is_materialized=true

Statistics & Analytics

Get comprehensive statistics:

# App-level stats
GET /api/apps/my-application/stats/

# DataSource-level stats
GET /api/apps/my-application/datasources/users-db/stats/

# Cross-datasource stats
GET /api/apps/my-application/datatables/stats/

Soft Delete

All resources support soft deletion (marked as inactive):

# Delete app (also inactivates all datasources and datatables)
DELETE /api/apps/my-application/

# Delete datasource
DELETE /api/apps/my-application/datasources/users-db/

# Delete datatable
DELETE /api/apps/my-application/datasources/users-db/datatables/users/

Schema-Only Operations

Get just the Frictionless schema without full table details:

GET /api/apps/my-application/datasources/users-db/datatables/users/schema/

Rate Limiting

All endpoints are rate-limited for security:

• Standard Limit: 100 requests per minute
• Burst Limit: 10 requests for write operations
• Authentication: Rate limiting per authenticated user

Tenant Isolation

All Data Service operations are tenant-scoped:

• Data is completely isolated per tenant
• No cross-tenant data access
• Automatic tenant detection from domain/subdomain

Common Use Cases

The Data Service is designed for a wide range of applications:

Content Management Systems (CMS)

• Blog platforms: Posts, comments, authors, categories with relationships
• Documentation sites: Articles with hierarchical organization
• Publishing workflows: Draft → Review → Published status management
• Features: Full-text search, soft delete, audit trails

SDK Example: Publishing a Blog Post

Python

from taruvi import TaruviClient

auth_client = TaruviClient(
    base_url="https://your-cms.com",
    jwt_token="YOUR_JWT_TOKEN"
)

# Create a draft post
draft_post = auth_client.database.create('posts', {
    'title': 'Getting Started with Taruvi',
    'content': 'In this guide, we will...',
    'author_id': 1,
    'status': 'draft',
    'created_at': '2024-01-15T10:00:00Z'
}, app_slug='my-blog')

# Review and publish
published_post = auth_client.database.update('posts', draft_post['id'], {
    'status': 'published',
    'published_at': '2024-01-15T12:00:00Z'
}, app_slug='my-blog')

# Get all published posts with authors
posts = auth_client.database.query('posts', app_slug='my-blog') \
    .filter('status', 'eq', 'published') \
    .populate('author') \
    .sort('published_at', 'desc') \
    .get()

for post in posts:
    author = post.get('author', {})
    print(f"{post['title']} by {author.get('username')}")

JavaScript

import { TaruviClient } from '@taruvi/sdk'

const client = new TaruviClient({
    baseUrl: 'https://your-cms.com',
    jwtToken: 'YOUR_JWT_TOKEN'
})

// Create a draft post
const draftPost = await client.database
    .from('posts')
    .create({
        title: 'Getting Started with Taruvi',
        content: 'In this guide, we will...',
        author_id: 1,
        status: 'draft',
        created_at: '2024-01-15T10:00:00Z'
    })
    .execute()

// Review and publish
const publishedPost = await client.database
    .from('posts')
    .get(draftPost.id.toString())
    .update({
        status: 'published',
        published_at: '2024-01-15T12:00:00Z'
    })
    .execute()

// Get all published posts with authors
const posts = await client.database
    .from('posts')
    .filter('status', 'eq', 'published')
    .populate(['author'])
    .sort('published_at', 'desc')
    .execute()

posts.data.forEach(post => {
    const author = post.author || {}
    console.log(`${post.title} by ${author.username}`)
})

E-Commerce Platforms

• Product catalogs: Products, categories, inventory with JSONB for flexible attributes
• Order management: Orders, line items, fulfillment tracking
• Customer analytics: Aggregations for sales reports, customer segmentation
• Features: Complex filtering, aggregations, relationship population

SDK Example: Product Catalog with JSONB Attributes

Python

from taruvi import TaruviClient

auth_client = TaruviClient(
    base_url="https://your-shop.com",
    jwt_token="YOUR_JWT_TOKEN"
)

# Create product with flexible JSONB attributes
product = auth_client.database.create('products', {
    'name': 'Wireless Headphones Pro',
    'sku': 'WHP-PRO-001',
    'price': 199.99,
    'category_id': 5,
    'inventory': 150,
    'attributes': {
        'color': 'black',
        'bluetooth': '5.0',
        'battery_life': '30 hours',
        'noise_cancelling': True,
        'weight': '250g'
    },
    'status': 'active'
}, app_slug='shop-app')

# Search products by JSONB attributes
black_headphones = auth_client.database.query('products', app_slug='shop-app') \
    .filter('attributes.color', 'eq', 'black') \
    .filter('attributes.noise_cancelling', 'eq', True) \
    .filter('price', 'lte', 250) \
    .get()

# Get products with categories populated
products = auth_client.database.query('products', app_slug='shop-app') \
    .filter('status', 'eq', 'active') \
    .populate('category') \
    .sort('price', 'asc') \
    .page_size(20) \
    .get()

for product in products:
    category = product.get('category', {})
    print(f"{product['name']} - ${product['price']} ({category.get('name')})")

JavaScript

import { TaruviClient } from '@taruvi/sdk'

const client = new TaruviClient({
    baseUrl: 'https://your-shop.com',
    jwtToken: 'YOUR_JWT_TOKEN'
})

// Create product with flexible JSONB attributes
const product = await client.database
    .from('products')
    .create({
        name: 'Wireless Headphones Pro',
        sku: 'WHP-PRO-001',
        price: 199.99,
        category_id: 5,
        inventory: 150,
        attributes: {
            color: 'black',
            bluetooth: '5.0',
            battery_life: '30 hours',
            noise_cancelling: true,
            weight: '250g'
        },
        status: 'active'
    })
    .execute()

// Search products by JSONB attributes
const blackHeadphones = await client.database
    .from('products')
    .filter('attributes.color', 'eq', 'black')
    .filter('attributes.noise_cancelling', 'eq', true)
    .filter('price', 'lte', 250)
    .execute()

// Get products with categories populated
const products = await client.database
    .from('products')
    .filter('status', 'eq', 'active')
    .populate(['category'])
    .sort('price', 'asc')
    .pageSize(20)
    .execute()

products.data.forEach(product => {
    const category = product.category || {}
    console.log(`${product.name} - $${product.price} (${category.name})`)
})

Analytics Dashboards

• Business intelligence: Real-time aggregations with GROUP BY and HAVING
• KPI tracking: Time-series data with date-based grouping
• User analytics: Activity tracking, event logging, funnel analysis
• Features: Aggregations, date functions, efficient querying

Social Networks

• User profiles: Users with flexible metadata (JSONB provider)
• Connections: Graph traversal for friend networks, followers
• Content feeds: Posts with likes, comments, shares
• Features: Graph queries, relationship population, hierarchies

SDK Example: Social Feed with Relationships

Python

from taruvi import TaruviClient

auth_client = TaruviClient(
    base_url="https://your-social-app.com",
    jwt_token="YOUR_JWT_TOKEN"
)

# Create a post
post = auth_client.database.create('posts', {
    'user_id': 1,
    'content': 'Just launched my new project! Check it out 🚀',
    'created_at': '2024-01-15T14:30:00Z',
    'likes_count': 0,
    'comments_count': 0
}, app_slug='social-app')

# Add a comment
comment = auth_client.database.create('comments', {
    'post_id': post['id'],
    'user_id': 2,
    'content': 'Congrats! This looks amazing!',
    'created_at': '2024-01-15T14:35:00Z'
}, app_slug='social-app')

# Get user's feed with all relationships
feed = auth_client.database.query('posts', app_slug='social-app') \
    .populate('user', 'comments.user') \
    .sort('created_at', 'desc') \
    .page_size(10) \
    .get()

for post in feed:
    user = post.get('user', {})
    comments = post.get('comments', [])
    print(f"{user.get('username')}: {post['content']}")
    print(f"  {post['likes_count']} likes, {len(comments)} comments")

    for comment in comments[:3]:  # Show first 3 comments
        comment_user = comment.get('user', {})
        print(f"    {comment_user.get('username')}: {comment['content']}")

JavaScript

import { TaruviClient } from '@taruvi/sdk'

const client = new TaruviClient({
    baseUrl: 'https://your-social-app.com',
    jwtToken: 'YOUR_JWT_TOKEN'
})

// Create a post
const post = await client.database
    .from('posts')
    .create({
        user_id: 1,
        content: 'Just launched my new project! Check it out 🚀',
        created_at: '2024-01-15T14:30:00Z',
        likes_count: 0,
        comments_count: 0
    })
    .execute()

// Add a comment
const comment = await client.database
    .from('comments')
    .create({
        post_id: post.id,
        user_id: 2,
        content: 'Congrats! This looks amazing!',
        created_at: '2024-01-15T14:35:00Z'
    })
    .execute()

// Get user's feed with all relationships
const feed = await client.database
    .from('posts')
    .populate(['user', 'comments.user'])
    .sort('created_at', 'desc')
    .pageSize(10)
    .execute()

feed.data.forEach(post => {
    const user = post.user || {}
    const comments = post.comments || []
    console.log(`${user.username}: ${post.content}`)
    console.log(`  ${post.likes_count} likes, ${comments.length} comments`)

    // Show first 3 comments
    comments.slice(0, 3).forEach(comment => {
        const commentUser = comment.user || {}
        console.log(`    ${commentUser.username}: ${comment.content}`)
    })
})

Organizational Management

• Employee directories: Hierarchical org charts with manager relationships
• Department structures: Tree queries for organizational hierarchies
• Access control: Row-level security based on department/role
• Features: Hierarchy queries, authorization policies, graph traversal

IoT & Time-Series Data

• Sensor data: High-volume inserts with efficient bulk operations
• Device management: Flexible schemas for various device types (JSONB)
• Aggregated metrics: Hourly/daily summaries with date_trunc aggregations
• Features: Bulk operations, BRIN indexes, efficient aggregations

Project Management

• Task tracking: Tasks with statuses, assignments, dependencies
• Milestone planning: Date-based filtering and grouping
• Team collaboration: Authorization for department-based access
• Features: Advanced filtering, authorization, relationships

Next Steps

Getting Started

• Quick Start Guide - Build your first app in 10 minutes
• Schema Guide - Learn Frictionless Table Schema format
• API Endpoints Reference - Complete endpoint listing

Core Guides

• CRUD Operations - Creating, reading, updating, deleting data
• Querying & Filtering - Advanced queries with 20+ operators
• Aggregations - SQL-like GROUP BY and aggregate functions
• Relationships - Foreign keys and relationship population

Advanced Topics

• Hierarchies - Tree structures and organizational charts
• Graph Traversal - Network queries with BFS/DFS
• Authorization - Policy-based access control with Cerbos
• Migrations - Schema evolution with Alembic
• Indexes - Creating and managing database indexes
• Data Imports - Bulk data loading from CSV/JSON

Reference

• Admin Interface - Using Django admin for management

Examples

Complete Workflow

See the API Endpoint Reference for comprehensive examples of:

• CRUD operations for all resources
• Schema updates and validation
• Data validation against schemas
• Error handling and responses
• Filtering and pagination