Metadata-Driven Databricks Data Platform

I designed and implemented a distributed Azure Databricks lakehouse from initial architecture through production, and own it in production now — including the pager end of it.

The organizing idea is that adding a source should be configuration work, not another custom pipeline.

Architecture

Sources

• MES
• ·ERP / IBM i
• ·SQL Server
• ·REST APIs
• ·SharePoint
• ·IoT

Metadata-driven ingestion

Python and Spark, with connections, objects, and load behavior read from configuration

Validation, schema handling, logging

Data-quality checks, schema evolution, structured logs, retry and recovery

Delta Lake

1. Bronze

   Landed, as received

2. Silver

   Conformed and validated

3. Gold

   Modelled for consumption

Consumers

• Power BI semantic models
• ·Internal applications
• ·Downstream systems

Configuration, not pipelines

The ingestion framework is reusable Python, SQL and Spark. What varies between sources lives in configuration rather than in code:

• connections
• source objects
• jobs
• ingestion behavior, including full and incremental loads
• transformations
• validation rules
• target tables, including their creation

The payoff is not elegance, it is that the tenth source costs a fraction of the first, and that all ten behave the same way when something goes wrong. A bespoke pipeline per source means ten different logging conventions, ten different failure modes and ten things to learn before you can fix any one of them at two in the morning.

It also changes who can onboard a source. When behavior is metadata, adding one stops being a task that requires knowing the framework’s internals.

Performance: multiple hours to under one minute

Critical ingestion workloads were re-engineered from multiple hours end to end down to under a minute. That came from several changes together rather than one trick:

• redesigning the Spark transformations, including how and where work was shuffled
• moving from full reprocessing to incremental patterns
• partitioning appropriate to how the data is actually queried
• improving orchestration so stages were not waiting on each other unnecessarily
• moving work to the layer that should own it, rather than the layer it happened to start in

The last one recurs constantly. A transformation written where it was first needed is rarely where it belongs once the volume grows.

Production characteristics

A platform is not finished when the data lands once. What makes it something you can rely on:

• Schema evolution — sources change shape without asking, and the pipeline has to handle new and altered columns rather than failing or silently dropping them
• Structured logging — enough context to answer what ran, over what, and what happened without reading Spark UI archaeology
• Validation — data-quality checks at ingestion, so bad data is caught at the boundary rather than surfacing later as a wrong number in a report
• Recovery — failures are expected, so retry and restart are designed rather than manual
• Separate development and production environments, with promotion between them
• Git and Azure DevOps CI/CD, so a platform change is a reviewed, versioned deployment
• Operational ownership — I support what I build, which is the fastest feedback loop there is on whether a design was any good

What this connected to

The platform is not an island. It ingests from MES, ERP on IBM i, SQL Server, REST APIs, SharePoint and IoT sources, and serves Power BI semantic models, internal applications and downstream systems. Most of the difficulty in practice sits at those two boundaries rather than in the middle.