Runtime lineage reconstructs history — it cannot prevent it
Every major lineage tool in production today works the same way. A query executes against the warehouse. The engine logs the statement. A parser — sometimes the warehouse's own query history API, sometimes an external SQL parser — decomposes that statement into source and target columns, then writes edges to a graph.
This is runtime lineage. OpenLineage collectors, Databricks Unity Catalog's system tables, Snowflake's ACCESS_HISTORY view, dbt's manifest.json generated after dbt compile — all reconstruct the dependency graph from artifacts produced during or after execution. The graph is accurate for what ran. It tells you nothing about what will run next.
The gap matters when an upstream source changes a column type from INTEGER to VARCHAR. Runtime lineage surfaces this after the downstream model either fails or, worse, silently coerces the data and produces wrong aggregates. A finance team pulls a revenue dashboard Monday morning. The numbers look plausible. They are wrong. Nobody knows until someone manually spot-checks against the source system three days later.
Runtime lineage is not broken. For audit trails, compliance reporting, and understanding historical data flow, it remains essential. But for preventing production breakage, it has a structural limitation: it requires execution to produce its graph. The broken query must run before you learn it was broken.
Compile-time lineage resolves the graph before execution
Rocky, a Rust-based SQL control plane that surfaced on Hacker News this week with 120 points and 48 comments, takes a different approach. Its compiler parses transformation models written in a .rocky DSL, resolves column-level dependencies against declared schemas, and builds the lineage graph before any query reaches the warehouse.
The mechanism works like a type checker. Each model declares its inputs and outputs. The compiler walks the DAG, matching each referenced column against the upstream model's schema definition. If a column is missing, renamed, or has an incompatible type, compilation fails with a diagnostic code — E010 for missing columns, E013 for type mismatches. No query is submitted. No rows are written. No compute is consumed.
Running rocky compile validates the entire DAG. Running rocky lineage --column revenue traces that single column backward through aggregations, joins, and seeds to its origin. Running rocky lineage-diff compares the resolved graph between two git refs, showing exactly which columns gained or lost dependencies across a pull request.
This is not entirely new territory. SQLMesh performs a similar static analysis using its own SQL parser, resolving column lineage at plan time rather than after execution. dbt's approach sits between the two — dbt compile resolves Jinja and produces SQL, but column-level lineage in dbt relies on either the dbt Cloud metadata API or third-party tools like Elementary and SQLLineage that parse the compiled SQL after the fact.
The Rocky project adds branch isolation on top. rocky branch create experiment-v2 provisions a separate schema in the warehouse, runs the modified DAG there, and lets you diff the resulting lineage graph against main. Think of it as a staging environment for your column dependencies, not just your data.
Schema drift is the silent failure compile-time lineage actually catches
Schema drift is the most common source of lineage-invisible breakage in production warehouses. A source system adds a column. An upstream team changes a DECIMAL(18,2) to DECIMAL(38,10). A CSV seed file arrives with headers in a different order. None of these necessarily cause query failures. Many cause silent data quality degradation.
Consider a Databricks pipeline where a bronze table ingests from a SQL Server CDC feed. The source DBA changes unit_price from DECIMAL(10,2) to DECIMAL(10,4). The bronze-to-silver transformation casts to DOUBLE for aggregation — no error. The silver-to-gold model rounds to two decimal places for the finance report. The extra precision is silently dropped in some rows but causes floating-point rounding differences in others. Revenue figures shift by fractions of a cent per row, adding up to thousands across millions of transactions.
With runtime lineage, you discover this when finance flags a reconciliation mismatch. With compile-time lineage, the compiler catches the type change at the schema contract boundary. Rocky's data contracts feature lets you declare column unit_price: DECIMAL(10,2) required in the model definition. When the upstream schema changes, rocky compile fails before the pipeline runs.
The limitation is scope. Compile-time lineage only covers transformations defined in the tool's own models. If an Azure Data Factory pipeline drops data into a staging table that Rocky reads as a source, the schema contract validates against the declared schema — but the declared schema is only as current as the last time someone updated it. Automatic schema introspection from the warehouse catalog closes this gap partially. Rocky supports this through its adapter layer for Databricks (Unity Catalog), Snowflake, and BigQuery, pulling live schemas at compile time.
Still, any data that arrives through a path outside the tool's control — a manual CSV upload, a third-party connector, an unmanaged Spark job — sits outside the compile-time graph. This is where runtime monitoring fills the gap.
Branch isolation makes lineage diffing practical for teams
Column lineage as a static artifact is useful for documentation. Column lineage as a diffable artifact is useful for code review.
The traditional workflow for validating a transformation change goes like this: a developer modifies a dbt model, opens a pull request, a reviewer reads the SQL, mentally traces the column dependencies, and approves based on their understanding of the DAG. For simple models this works. For a DAG with 200+ models and cross-database references, mental tracing is unreliable.
Rocky's branch model addresses this directly. When a developer runs rocky branch create feature-x, the tool creates an isolated schema (e.g., dev_feature_x) in the target warehouse. All models in the modified DAG compile and execute against that schema. The lineage graph for the branch is stored alongside the main lineage graph. Running rocky lineage-diff main..feature-x produces a structured diff showing which columns gained new upstream dependencies, which lost them, and which changed type.
This turns lineage review from a manual SQL-reading exercise into a structured diff. A reviewer can see that fact_orders.total_revenue now depends on stg_pricing.discount_rate when it previously did not, and ask whether that dependency is intentional.
SQLMesh offers similar functionality through its sqlmesh plan command, which computes a diff of the DAG and shows affected downstream models. The granularity differs — SQLMesh diffs at the model level with column-level lineage available in its UI, while Rocky exposes column-level diffs in the CLI output.
The practical constraint is warehouse cost. Each branch creates real tables in the warehouse. For DuckDB-backed local development, this is free. For a Databricks branch with 50 models over terabyte-scale data, compute costs add up. Rocky's SHALLOW CLONE support on Databricks mitigates this by cloning metadata rather than data, but the feature is marked as planned rather than shipped for all model types.
Compile-time catches contracts, runtime catches everything else
Compile-time column lineage prevents a specific class of failures: schema drift, type mismatches, missing columns, and broken contracts within the managed DAG. These are real and common — schema drift alone accounts for a significant share of data pipeline incidents in organizations running 50+ models.
But production pipelines fail for reasons no compiler can anticipate. A Databricks cluster times out due to an under-provisioned autoscale policy. An ADF pipeline's linked service credential expires. A dbt Cloud job succeeds but produces zero rows because the source table was truncated during a maintenance window. A Power BI dataset refresh fails because the gateway service stopped responding. These are runtime failures, and they require runtime detection.
This is where compile-time and runtime observability serve complementary roles. Compile-time lineage tells you which columns will be affected if a schema changes. Runtime monitoring tells you that the pipeline actually failed, that the refresh is delayed, that row counts dropped 40% from the previous run.
MetricSign detects these runtime failures across Power BI refreshes, ADF pipelines, Databricks jobs, and dbt runs — grouping related incidents by root cause so that a single expired credential does not generate 30 separate alerts for 30 downstream datasets. The column lineage graph from a tool like Rocky tells you which columns are in the blast radius. The runtime signal from monitoring tells you the blast actually happened.
The engineering takeaway: adopt compile-time lineage to shift schema validation left into CI. Keep runtime monitoring to catch the failures that no static analysis can predict. Neither replaces the other. Together, they close the gap between 'this DAG is structurally valid' and 'this DAG actually produced correct data on time.'