Why Your dbt Tests Are Not Enough

dbt has become the backbone of the modern data stack. In a few years it went from an internal tool at Fishtown Analytics to a standard part of how data teams build, test, and document their transformation pipelines. And rightfully so. dbt brought software engineering discipline to analytics work: version control, modularity, documentation, and automated testing.

But somewhere along the way, a conflation happened. Teams started treating passing dbt tests as evidence of data quality. Green checks in the CI pipeline became the signal that the data was good. And that assumption is quietly causing problems in data teams everywhere.

dbt tests are excellent at what they do. The issue is that what they do covers only a fraction of the full data quality problem. A pipeline that passes every dbt test can still serve wrong numbers to the business, miss values that have drifted over months, fail to reconcile against the source system it was supposed to mirror, and silently produce reports that no one has verified against anything outside the pipeline itself.

This is not a criticism of dbt. It is a description of what any transformation-layer testing framework can and cannot do by design. Understanding where dbt tests end and where your data quality gaps begin is the first step to building a data quality program that actually works.

This guide covers exactly that: what dbt tests cover well, the four categories of data quality issues they do not address, and what a complete data quality stack looks like for teams that want to close those gaps.

What dbt Tests Actually Cover

To understand where dbt tests fall short, you first need to be clear about what they are genuinely good at. dbt's testing framework is designed to validate the outputs of your transformation models. That is its scope, and within that scope it does the job well.

Schema and Structural Tests

dbt's built-in generic tests cover the structural properties of model outputs. Not null checks confirm that required fields are populated. Uniqueness tests confirm that primary keys or identifiers do not contain duplicates. Accepted values tests confirm that categorical fields contain only the expected set of values. Referential integrity tests (relationships) confirm that foreign keys resolve to records in a referenced model.

These tests run against the output of a model after it has been built. They are excellent for catching transformation errors that break the structure of the output, and for enforcing contracts between models in a dependency chain.

Freshness Checks

dbt's source freshness feature checks whether source data has been loaded recently enough to be considered current. If your pipeline expects a daily load and the most recent record is from three days ago, the freshness check surfaces that. This is a genuinely useful check that catches pipeline failures and stale source loads.

Custom Data Tests

dbt's singular tests allow you to write custom SQL logic as a test. If a row returns, the test fails. This gives teams significant flexibility to encode business rules or more complex validation logic as dbt tests, which is one of the framework's most powerful capabilities.

Where dbt Tests Genuinely Excel

dbt tests are most valuable for catching regressions introduced by model changes. When a developer refactors a transformation and accidentally breaks a join, drops a column, or introduces duplicates, dbt tests catch it in CI before the change reaches production. That is a significant and real value. The pipeline discipline dbt brings to analytics engineering is not in question. The question is what category of data quality problem it was never designed to solve.

The Gaps dbt Tests Do Not Address

dbt tests run after your models are built. That single architectural fact creates four categories of data quality problems that dbt tests are structurally unable to catch, regardless of how many tests you write or how good your test coverage is.

The four gaps are: source data quality issues that exist before dbt runs, cross-system reconciliation failures that require comparing your pipeline output against an external reference, production data quality drift that happens between pipeline runs and outside the CI environment where tests run, and business rule violations that require domain knowledge to validate rather than just structural checks.

Each of these is a real and common source of data quality failures in production. And none of them are covered by dbt testing, however mature your test suite is.

Gap 1: Source Data Quality

dbt operates on data that has already landed in your warehouse. By the time dbt runs, the ingestion layer has already done its job: data has been extracted from source systems, loaded into staging tables, and is waiting to be transformed. dbt tests have no visibility into whether that ingestion was correct.

Source data quality problems are problems that exist in the raw data before any transformation touches it. They include:

Missing records from an incremental load where the ingestion job silently dropped rows due to a schema change in the source. Duplicate records introduced by a source system that emitted the same event twice. Invalid or corrupt values in fields that your transformation models assume are clean (a date column containing free-text strings, a numeric field containing nulls that your model aggregates without null handling). Encoding or character set issues that cause string values to arrive malformed. Truncated fields where the source system exported a column wider than the destination could receive.

None of these are catchable by dbt tests because dbt tests run downstream of the data. A not-null test on a transformed model will catch the symptom (a null in the output) but not the cause (a null that arrived from the source and propagated through). A uniqueness test on a final model will catch duplicate output rows, but only if the duplication survived all the transformations. A deduplication step in the model will silently mask a source duplication problem, and the dbt test will pass while the underlying source quality issue goes undetected.

Source data quality monitoring requires checking the data at the ingestion boundary, before transformation runs. That is a different layer entirely from what dbt tests cover, and it requires tooling that operates at the raw or staging layer rather than the model output layer.

This is one of the most common dbt testing limitations teams run into as their pipelines scale. The test suite grows, coverage looks strong, and then a source-side data quality issue causes a production incident that no dbt test caught because no dbt test was looking at the right place.

Gap 2: Cross-System Reconciliation

dbt tests validate the internal consistency of your pipeline. They check whether model A relates correctly to model B, whether a field in your transformed output is unique, whether a foreign key resolves within the warehouse. What they cannot do is tell you whether your pipeline output matches an external reference.

Cross-system reconciliation is the practice of comparing your pipeline output against the source of truth it was derived from, or against a parallel system that should contain the same data. It answers the question: does your data warehouse agree with your ERP, your CRM, your billing system, your bank statement? This is a fundamentally different question from anything dbt tests ask.

Consider a revenue reporting pipeline. The pipeline pulls data from a billing system, transforms it through several models, and produces a revenue table that feeds the finance dashboard. dbt tests can confirm that the revenue table has no nulls in the amount field, that every transaction has a unique ID, and that the customer foreign key resolves. What dbt tests cannot confirm is that the total revenue in the data warehouse matches the total revenue in the billing system for the same period.

That reconciliation gap is exactly where data quality issues with business impact live. A transformation bug that double-counted a revenue category, an ingestion job that dropped a day of transactions because of a timezone boundary issue, a currency conversion that used the wrong rate for one market: all of these could pass every dbt test in the suite while producing revenue figures that do not match the source system.

Cross-system reconciliation requires comparing totals, record counts, and key metrics between your pipeline output and an external reference on a scheduled basis. It is a row-count and aggregate check across system boundaries, not an internal consistency check within the pipeline. dbt's architecture has no mechanism for this, and no amount of dbt test coverage fills this gap.

For teams where the data warehouse is a system of record for financial, operational, or compliance reporting, this gap is not theoretical. It is the difference between a pipeline that passes all its tests and a pipeline that is actually trustworthy.

Gap 3: Production Data Monitoring

dbt tests run in a CI environment when code changes are deployed. They also run on a schedule when the pipeline runs. But data quality issues do not only happen when models are rebuilt. They happen continuously, in production, between pipeline runs.

Production data quality monitoring is the practice of watching the data in your warehouse tables over time and alerting when something changes unexpectedly: a metric that drops 40 percent overnight, a table that stops updating, a distribution that shifts, a column that suddenly has 20 percent more nulls than it did last week. This category of data quality monitoring is what the data observability space has grown up around, and it is structurally different from what dbt tests do.

dbt tests are point-in-time checks. When a dbt run completes, the tests run against the current state of the models. If everything passes, the run is green. But what happens to the data in that table over the next 24 hours? If a source system sends corrupt data in the next incremental load and your pipeline is incremental, the data lands in the table without triggering a dbt test run. The corruption sits in production until the next scheduled run, or until someone notices a wrong number in a report.

The difference between dbt testing and production data monitoring is the difference between quality gates at deployment and continuous quality surveillance in production. Both matter. A team that has strong dbt test coverage but no production monitoring is protecting the pipeline at build time and leaving it unmonitored at runtime.

Production monitoring for data quality typically covers volume anomalies (the number of rows in a table changing unexpectedly), distribution drift (the statistical distribution of values in a column changing over time), freshness monitoring beyond what dbt source freshness covers, and null rate and format change tracking at the column level. These checks run continuously against the live tables, not against model builds.

For data teams whose pipelines serve operational dashboards, financial reports, or customer-facing features, this gap is a significant production risk. A pipeline with green dbt tests and no production monitoring has half the data quality infrastructure it needs.

Gap 4: Business Rule and Semantic Validation

dbt's generic tests (not null, unique, accepted values, relationships) validate technical properties of data. They answer questions like: is this field populated, is this key unique, does this value appear in a known list? These are structural checks, and they are the right starting point.

But data quality for a business is not just about structure. It is about whether the data makes sense in the context of the domain it describes. Business rule and semantic validation is the fourth gap: the category of checks that require understanding what the data means, not just what shape it has.

Some examples of business rule violations that dbt tests cannot catch by default:

An order total that is lower than the sum of its line items. A customer record with a signup date that is after their first transaction date. A subscription that is marked active but has an end date in the past. A revenue figure that is negative for a product category that cannot logically have negative revenue. A user activity record with a session duration of eight hours for an application where sessions time out at 30 minutes. A shipment record with a delivery date before the order date.

None of these are structural violations. The fields are populated, the types are correct, the keys are unique. The data is technically valid but semantically wrong. Standard dbt tests will not catch them. Custom dbt singular tests could catch some of them if someone writes the specific logic for each case, but that requires a data engineer to know the domain rule exists, translate it into SQL, maintain it as the business logic evolves, and cover every table where the violation could appear.

Business rule validation at scale requires a systematic approach to encoding domain knowledge as data quality rules across the full pipeline, with alerting when rules are violated in production. This is a data quality management challenge that sits adjacent to what dbt was designed for, and it scales much better when it lives in a dedicated data quality layer rather than as a growing collection of custom dbt tests.

What a Complete Data Quality Stack Looks Like

The answer to dbt testing limitations is not to replace dbt tests or write more of them. It is to recognize dbt tests as one layer in a larger data quality stack, and to build the other layers deliberately.

A complete data quality stack for a modern data team has four layers that map directly to the four gaps covered in this guide:

Layer 1: Source quality monitoring. Checks that run at the ingestion boundary, before dbt transforms anything. This layer validates that the data arriving from source systems has the expected volume, schema, format, and value distributions. It catches ingestion failures, source-side data quality degradation, and schema drift before it propagates downstream.

Layer 2: Transformation testing (dbt). This is where dbt tests live and where they excel. Schema validation, uniqueness, referential integrity, and custom transformation logic tests that run when models are built. The existing strength of your dbt testing belongs here.

Layer 3: Cross-system reconciliation. Scheduled checks that compare warehouse output against external reference systems. Record count reconciliation, aggregate total comparison, and key metric validation between the pipeline output and the source systems it was built from. This layer is what gives your data warehouse the right to be called a system of record.

Layer 4: Production data quality monitoring. Continuous monitoring of live warehouse tables for volume anomalies, distribution drift, freshness degradation, and null rate changes. This layer watches the data between pipeline runs and surfaces quality issues that happen in production, not just at build time.

Most data teams have layer 2 built out reasonably well (the dbt layer). Layer 1 is often partially covered by freshness checks but not systematically. Layer 3 is frequently missing entirely. Layer 4 ranges from ad hoc alerting on key dashboards to fully automated observability, depending on team maturity.

Building out the missing layers does not require replacing dbt or rebuilding your pipeline. It requires adding purpose-built tooling or systematic checks at each layer, integrated with the alerting and documentation processes your team already has.

The data teams that have the fewest production data quality incidents are not necessarily the ones with the most dbt tests. They are the ones who have deliberately covered all four layers and know which layer is responsible for catching which category of issue.

Stop Mistaking Test Coverage for Data Quality

The dbt testing limitations described in this guide are not edge cases or hypothetical scenarios. They are the lived experience of data teams who built strong test suites, deployed with confidence, and still had production data quality incidents.

Green tests mean your transformations are working as written. They do not mean your source data is clean, your pipeline output reconciles against the systems it derives from, your production tables are healthy between runs, or your data obeys the business rules your stakeholders assume it does.

All four of those things require a data quality layer that sits outside dbt. Building that layer is not a criticism of dbt. It is what a mature data quality practice looks like.

If your team is running dbt and wants to close the gaps this guide describes, Vexdata adds source quality monitoring, cross-system reconciliation, and continuous production monitoring to your existing dbt pipeline without replacing any of it. See how data teams are using it at vexdata.io.