---
title: "Lakehouse Integration"
description: "Data inventory, connectors, recursive intelligence loop, GraphRAG, and auto-seed pipeline."
canonical: "https://docs.ainative.studio/docs/architecture/LAKEHOUSE_INTEGRATION"
last-updated: "2026-10-28T06:34:16.301Z"
---

# Lakehouse Integration

Source: https://docs.ainative.studio/docs/architecture/LAKEHOUSE_INTEGRATION

> Data inventory, connectors, recursive intelligence loop, GraphRAG, and auto-seed pipeline.

# Lakehouse Integration Architecture

The AINative Lakehouse is a MinIO/Parquet/DuckDB architecture that ingests, enriches, and serves data across ZeroDB services, the agent swarm, and the recursive intelligence loop.

## Data Inventory

| Dataset | Records | Source | Update Frequency |
|---------|---------|--------|-----------------|
| SMB Businesses | 264,632 | Socrata Open Data (6 cities) | Weekly cron |
| Federal Grants | 900 | USAspending.gov | Monthly |
| WHOIS Cache | 363 (54 enriched) | Apollo/Hunter/WHOIS | Daily DAG |
| SC County Parcels | 97,182 | Santa Cruz County | Monthly |
| Knowledge Graph Entities | 39,714 | CRM + Enrichment + Grants | Daily |
| Knowledge Graph Edges | 49,326 | works_at, connected_to, received_grant_from | Daily |
| Paul Graham Essays | 231 | paulgraham.com | On-demand |
| Video Transcripts | 1,275+ | YouTube (5 channels) | Weekly |

### SMB City Coverage

| City | Records | Source API |
|------|---------|-----------|
| Denver | 50,000 | Colorado SOS (data.colorado.gov) |
| Los Angeles | 50,000 | LA Business Tax (data.lacity.org) |
| Seattle | 50,000 | WA State (data.wa.gov) |
| San Francisco | 50,000 | SF Business Registry (data.sfgov.org) |
| New York City | 43,581 | NYC Business Licenses (data.cityofnewyork.us) |
| Austin | 21,051 | Austin Food Establishments (data.austintexas.gov) |

## Connectors

All connectors are Python scripts in `scripts/`:

| Connector | Script | What It Does |
|-----------|--------|-------------|
| SMB City | `smb_city_connector.py` | Pulls business data from Socrata APIs for 6 US cities |
| Federal Grants | `federal_grants_connector.py` | Fetches grant data from USAspending.gov |
| WHOIS Discovery | `whois_discovery_domains.py` | Enriches domains with WHOIS data |
| Montana Livestock | `montana_livestock_connector.py` | Auction market data |
| Montana Web3 | `montana_web3_connector.py` | Crypto/blockchain entities |
| SC County | `santa_cruz_county_connector.py` | Property parcel data |
| SMB Enrichment | `smb_enrichment_pipeline.py` | Apollo/Hunter/WHOIS/LinkedIn enrichment |
| Grant Graph | `build_grant_graph_edges.py` | Builds KG edges between grants and SMB businesses |
| PGWisdom | `pgwisdom_connector.py` | Paul Graham essays → lakehouse + ZeroMemory |
| Video Transcripts | `research_agent.py` | YouTube transcript ingestion (5 channels) |

## Recursive Integration Loop

The lakehouse data feeds into ZeroDB services via 4 Celery tasks:

```
Lakehouse Data
  ↓
lake.export_marketplace_data (05:00 UTC daily)
  → SMB/grants/WHOIS → Parquet lakehouse (MinIO)
  ↓
lake.sync_graph_to_zerodb (05:30 UTC daily)
  → KG entity/edge updates → ZeroDB events
  ↓
lake.emit_lakehouse_signals (06:00 UTC daily)
  → Data snapshot → ZeroMemory agent signals
  ↓
Agent Swarm (Scout, Vega, Aurora)
  → Consumes signals, triggers downstream actions
  ↓
lake.score_seed_quality (on project creation)
  → RLHF feedback on auto-seeded prospect quality
  → Feeds back into enrichment DAG adjustments
```

### Auto-Seed Pipeline

When a new project is created:
1. `POST /api/v1/projects` dispatches `seed_new_project.delay()`
2. Task pulls 500 prospects from SMB directory + WHOIS data
3. Scores each prospect with `get_lead_score()` (0-100)
4. Stores as rows in project's `prospects` table
5. Triggers `score_seed_quality` for RLHF feedback

Optional params: `industry` (saas, ecommerce, finance, healthcare, realestate) and `city`.

## GraphRAG Prospecting

`POST /api/v1/public/data/prospect` — Hybrid search combining:
- **Text relevance** (40%): Full-text search against SMB businesses
- **Graph proximity** (30%): Companies connected to CRM contacts or grant recipients rank higher
- **Lead score** (30%): Enrichment pipeline score (phone, WHOIS, tech stack, city)

## Video Transcription Channels

| Channel | Videos | Tags |
|---------|--------|------|
| Peter Diamandis | 864 | moonshots, exponential-tech |
| This Week in Startups | 149 | startups, venture-capital |
| Anthropic Official | 140 | anthropic, claude, ai-safety |
| Claude Official | 103 | claude-code, developer-tools |
| AINative AI Agents | 19 | agent-cloud, ai-agents |

Transcripts stored in `scripts/outputs/transcripts/` and ingested to ZeroMemory.

## Key Files

| File | Purpose |
|------|---------|
| `scripts/smb_city_connector.py` | SMB business data ingestion |
| `scripts/smb_enrichment_pipeline.py` | Multi-source enrichment |
| `scripts/build_grant_graph_edges.py` | Grant → company KG edges |
| `scripts/pgwisdom_connector.py` | Paul Graham essays |
| `scripts/research_agent.py` | YouTube transcript ingestion |
| `src/backend/app/celery_tasks/lakehouse_integration_tasks.py` | 4 integration Celery tasks |
| `src/backend/app/celery_tasks/seed_project_task.py` | Auto-seed on project creation |
| `src/backend/app/api/v1/endpoints/graphrag_prospect.py` | GraphRAG prospecting API |
| `src/backend/app/api/v1/endpoints/data_marketplace.py` | Data marketplace endpoints |
| `src/backend/app/services/seed_data.py` | Seed data service |
| `services/airflow/dags/smb_enrichment_dag.py` | Daily enrichment DAG |

## Multi-Tenant Isolation

The lakehouse is multi-tenant. Two distinct data planes share the MinIO
infrastructure but are kept isolated:

- **Internal platform lake** (`ainative-lake`) — AINative's own agent telemetry,
  platform metrics, and external datasets. Admin/agent access only. See the
  internal [Data Lake API](../api/data-lake).
- **Customer lakehouse** (`ainative-lakehouse`) — per-project customer data,
  exposed as the project-scoped [ZeroDB Lakehouse](../api/zerodb-lakehouse).

### Partition layout (Refs #5246)

Every object is written under a tenant prefix so storage is attributable and
isolatable per project/tenant:

```
ainative-lakehouse/
└── tenant={project_id}/
    └── {sensor_type}/
        └── date=YYYY-MM-DD/
            └── batch_*.parquet
```

Untagged internal ingestion is attributed to `INTERNAL_TENANT_ID` (AINative
bills itself "like a customer"). `LakehouseWriter.write_parquet(..., tenant_id=)`
keys batches by `(tenant_id, sensor_type)` so two tenants never share a file.

### Read scoping (Refs #5270, #5271)

The customer query path is **project-scoped end to end** — the same model as
every other ZeroDB project service:

1. **Ownership** — `_get_project_or_404(project_id, user.id, db)` on every
   endpoint (`query`, `tables`, `stats`, `connectors`). Non-owner → 404.
2. **Scoped globs** — `scoped_query.scoped_parquet_glob(project_id, sensor)`
   builds `tenant={project_id}/{sensor}/**` only. Listings pass
   `Prefix=tenant={project_id}/`.
3. **SQL sandbox** — `validate_scoped_sql` allows a single `SELECT`/`WITH` over
   *logical* table names and rejects raw path access (`s3://`, `read_parquet`,
   `read_csv`, `glob`, `ATTACH`, `COPY`, `SET`, `PRAGMA`, `INSTALL`, `LOAD`,
   multi-statement). `rewrite_tables` expands logical names to the caller's
   scoped `read_parquet()` glob; the bare name never reaches DuckDB. The DuckDB
   connection is locked (`lock_configuration`) after credentials are set.

Result: a project can only ever read its own partition. Cross-tenant reads are
impossible at both the engine and HTTP boundary (proved by
`tests/test_lakehouse_project_scoping.py` and
`tests/test_lakehouse_endpoint_scoping.py`).

### Key isolation files

| File | Purpose |
|------|---------|
| `src/backend/app/zerodb/lakehouse/writer.py` | Tenant-partitioned Parquet writer (#5246) |
| `src/backend/app/zerodb/lakehouse/config.py` | `PARTITION_TEMPLATE`, `INTERNAL_TENANT_ID` |
| `src/backend/app/zerodb/lakehouse/scoped_query.py` | Project-scoped read engine + SQL sandbox (#5270/#5271) |
| `src/backend/app/zerodb/api/lakehouse.py` | Customer-facing project-scoped endpoints |