Schema changes are inevitable. This is not a pessimistic take — it is just the reality of building data pipelines on top of systems that are actively being developed by other teams. The application developers adding a new feature ship a new column. The SaaS vendor updates their API and renames a field. The upstream team decides that a string field should actually be an integer, or that a nullable column should now be required, or that three columns should be collapsed into one. None of these changes are malicious. They are the normal output of software development. But if your pipeline is not designed to handle them, each one is a potential incident.
The teams that handle schema changes gracefully are not the ones who prevent them — that is not possible — but the ones who have thought carefully about where schema assumptions live in their systems and how to isolate the blast radius when those assumptions break.
Why Schema Changes Break Pipelines
To understand how to handle schema changes, it helps to understand exactly why they cause problems. Most pipeline failures from schema changes fall into one of a small number of categories.
A new column appears in the source that the pipeline does not know about. Depending on how the pipeline is built, this might be harmless — the column is simply ignored — or it might cause a failure if the pipeline is doing explicit schema validation or writing to a strongly typed destination that does not tolerate unexpected fields.
An existing column is renamed. The pipeline looks for the old name, finds nothing, and either errors out or silently loads nulls where real data should be. This is one of the most dangerous failure modes because it can produce incorrect data without raising an obvious error — downstream models may run successfully while operating on a null column they believe contains real values.
A column’s data type changes. A field that was a string becomes an integer, or a date becomes a timestamp. If the destination table has the old type enforced and the pipeline tries to load the new type, you get a cast error. If the pipeline silently coerces the type, you may get data loss or precision errors that are difficult to detect.
A column is removed. The pipeline expects it, the source no longer provides it, and the result depends entirely on how your pipeline handles missing fields. Silent nulls are a common and dangerous outcome.
Each of these failure modes has a different signature and a different fix, which is why “handle schema changes” is not a single problem but a family of related problems that require different approaches at different layers of the pipeline.
The Raw Layer as a Buffer
The most effective architectural defense against schema change failures is a raw layer that accepts data without enforcing a fixed schema. When your extraction layer loads data into a raw staging area without transformation, schema changes in the source affect only the raw layer. Downstream models built on top of the raw layer are insulated from immediate breakage — they continue to run on the existing raw data while you assess the change and update your transformation logic deliberately.
This is one of the strongest arguments for the ELT pattern. In an ETL system where transformation happens before loading, a schema change in the source can break the pipeline before any data reaches the destination. In an ELT system, raw data lands regardless, and the transformation layer can be updated independently. The change is still work, but it is controlled work rather than an emergency.
Practically, this means designing your raw tables to be as permissive as possible. In BigQuery, JSON columns can absorb semi-structured data with evolving schemas. In Snowflake, the VARIANT type serves the same purpose. In a data lake on object storage, Parquet files written with schema evolution support — available in Delta Lake, Iceberg, and Hudi — can accommodate new columns without requiring a full table rewrite. The key principle is that the raw layer should never reject data because the schema changed. It should accept the data and surface the change for human review.
Schema Evolution Strategies
Once you have accepted that schemas will change, the next question is how to handle those changes in your transformation and serving layers. There are four main strategies, each appropriate for different scenarios.
Backward compatible evolution covers the safest class of changes: adding new nullable columns, adding new values to an enum, or relaxing constraints. These changes do not break existing consumers. A new nullable column appears as null in downstream models that do not explicitly select it, and models that do want it can be updated at their own pace. Design your source systems and APIs to prefer backward compatible changes wherever possible, and make sure your pipeline treats them as non-breaking.
Forward compatible evolution is the practice of writing transformation logic that is tolerant of missing fields. Instead of assuming a column exists and failing if it does not, use conditional logic — COALESCE, CASE WHEN, or equivalent — to handle the case where a field is absent. This requires more defensive coding but produces pipelines that degrade gracefully rather than failing hard when a source changes.
Schema versioning is appropriate when source schemas change in ways that are not backward compatible and where historical data under the old schema needs to remain accessible alongside new data under the new schema. The approach is to maintain versioned raw tables — raw.salesforce_accounts_v1, raw.salesforce_accounts_v2 — and build transformation logic that unions across versions with appropriate mappings. This is operationally heavier but preserves full historical fidelity. It is most commonly needed when a source system undergoes a major restructuring.
Full reload with schema migration is the blunt instrument: when a schema change is significant enough, you truncate the destination table, update the pipeline to reflect the new schema, and reload from scratch. This is the simplest approach when history under the old schema is not valuable or when the dataset is small enough that a full reload is fast. The cost is losing historical data in its original form, which is sometimes acceptable and sometimes not.
Detection: Knowing Before Things Break
The best schema change handling is proactive rather than reactive. If you know a schema has changed before your downstream models run, you can assess the impact and make updates before anything breaks in production. This requires schema change detection built into your pipeline.
The approach is straightforward: at extraction time, compare the schema of the incoming data against the schema you observed on the previous run. If they differ — new columns, missing columns, type changes — emit an alert before proceeding. The alert should describe exactly what changed: which columns are new, which are missing, and which have changed type. This gives the on-call engineer the information they need to assess impact and decide whether to proceed, pause downstream processing, or route data to a quarantine area while the change is investigated.
Several tools in the modern data stack support schema change detection natively. Fivetran alerts on schema changes and allows you to configure whether new columns are automatically included or blocked. Great Expectations and Soda can run schema validation as part of a data quality suite. dbt’s source freshness and schema tests catch structural changes in the data that reaches the transformation layer. None of these tools eliminate the need to handle schema changes — they just ensure you find out about them quickly rather than hours later when a stakeholder notices a broken dashboard.
Contracts: The Upstream Conversation
Schema changes that break pipelines are often symptoms of a communication gap between the team that owns the source system and the team that owns the pipeline. Application developers do not always know that a column rename in their user table will cascade into six broken downstream models. Data engineers do not always know that a major feature launch is coming that will restructure a core table.
Data contracts are an increasingly popular approach to formalizing this relationship. A data contract is an explicit agreement between a data producer and one or more data consumers about the structure, semantics, and reliability of a dataset. It specifies what columns exist, what types they are, what values are valid, and how much notice will be given before breaking changes are made.
The tooling around data contracts is still maturing, but the practice does not require sophisticated tooling to start. A shared document, a Slack channel where schema changes are announced, and a lightweight review process for changes to high-impact tables can dramatically reduce the number of surprise schema changes that hit production pipelines. The cultural change — getting source system owners to think about their downstream consumers before shipping changes — is more important than any specific tool.
Building for the Inevitable
The engineering mindset that produces resilient pipelines treats schema changes not as exceptional events to be prevented but as normal operational conditions to be handled. This means building raw layers that accept evolving schemas without breaking. It means writing transformation logic that degrades gracefully rather than failing hard. It means instrumenting extraction to detect changes and alert on them before they cascade into downstream failures. And it means investing in the upstream relationships and contracts that give you advance warning when changes are coming.
None of this eliminates the work that schema changes create. Pipelines need to be updated, models need to be revised, and sometimes historical data needs to be reconciled. But the difference between a schema change that is a minor planned update and one that is a 2am production incident is almost entirely a function of how well the pipeline was designed to handle it.