Data Engineering
NovaSight
Unified retail data lakehouse consolidating 200+ stores into a single governed platform.
About the Project
What We Built
NovaSight is a cloud-native data lakehouse built on GCP for a national retail chain operating 200+ physical stores alongside three e-commerce channels. Siloed data across incompatible POS systems, ERP modules, and loyalty platforms made real-time inventory visibility impossible resulting in 18% overstock rates, chronic out-of-stock events during peak demand, and customer analytics that were always 48 hours stale. The new platform consolidates every data source into a governed, query-optimised lakehouse powering same-day decisions.
The Problem We Solved
Challenge, Solution & Outcome
Data lived in 12 incompatible systems across 200+ stores with no unified view. Inventory decisions were based on 48-hour-old exports, causing 18% overstock across slow-moving SKUs while popular items sold out during peak periods. Customer analytics required 3-day manual ETL runs.
A GCP-native data lakehouse using Apache Flink for real-time CDC ingestion, Delta Lake for open-format ACID storage, dbt for governed transformations, and BigQuery as the analytical serving layer with Looker providing self-serve access to unified data for every business team.
Merchandising teams now make restocking decisions on same-day data. Overstock rates dropped 18% in the first six months as demand signals became visible before inventory built up. Customer segmentation that previously took 3 days runs in 4 hours, enabling personalised campaigns at scale.
Technology
Tech Stack Used
Capabilities
Key Features
Apache Flink Change Data Capture pipelines stream events from 200+ store POS systems (4 different vendors), 3 e-commerce platforms, and 12 ERP modules eliminating overnight batch reconciliation entirely.
An open-format Delta Lake on Google Cloud Storage provides ACID transactions, time-travel queries for up to 30 days, and schema evolution without breaking downstream consumers across 15TB of retail data.
250+ dbt models implement business logic across raw, staging, intermediate, and mart layers. Column-level lineage, automated testing, and data documentation are generated on every deployment.
Materialised BigQuery views expose business-ready tables to Looker dashboards and direct SQL consumers. Query acceleration through BI Engine reduces dashboard load times from 40 seconds to under 3 seconds.
Flink streaming aggregations compute store-level stock positions and replenishment triggers within 90 seconds of a sale giving supply chain teams actionable signals before shelves empty during peak demand.
BigQuery column-level security, GCP IAM roles, and a centralised data catalogue enforce PII protection, role-based access, and GDPR compliance across all 200+ data consumers without manual provisioning.