About Data Market

A central data catalog and marketplace for discovering, exploring, and managing data assets across your organization — from Snowflake tables to Databricks dashboards.

Architecture: Federated Catalog

The Data Market uses a push-based architecture. Instead of crawling every database, lightweight catalog agents in each region/platform push metadata deltas to the central registry. This means no inbound network access is required — just outbound HTTPS.

Central Marketplace (React + Firebase)
    ├── Search, Browse, Import, Rate
    └── Catalog Backend (Firestore)
            │
    ┌───────┼───────┐
    ▼       ▼       ▼
  Agent   Agent   Agent
 (DE)    (US)   (APAC)

Features

Discover

Full-text search across asset names, descriptions, column names, and tags with faceted filtering.

Import

Bulk import metadata via JSON or YAML files. Preview before importing and validate schemas automatically.

Federated

Catalog data assets from Snowflake, Databricks, Synapse, BigQuery, and more across all regions.

Governance

Clear data classification (Public, Internal, Confidential, PII) and ownership for every asset.

Fast

Client-side fuzzy search with Fuse.js for instant results. No external search cluster needed for MVP.

Scalable

Built on Firebase Firestore for serverless scaling. Push-based agent model minimizes infrastructure overhead.

Phase 1 · MVP

This is the MVP release with manual JSON/YAML import, search, and ownership display. Phase 2 will add automated agents, ratings, access workflows, and lineage visualization.