Schema drift is when upstream data changes its structure without warning. A column gets renamed, a type changes, a new field appears — and your pipeline breaks or, worse, silently produces wrong results.
user_name becomes username. Your transforms SELECT the old name → NULLs everywhere, silently.
price goes from DECIMAL to STRING. Aggregations break or produce garbage.
A "deprecated" field removed without notice. Any downstream model depending on it fails.
Usually safe but can break SELECT * patterns, change column ordering, or exceed schema limits.
"I detect schema drift by comparing incoming schemas against an expected contract — column names, types, and nullability. Additive changes can be auto-allowed; breaking changes fail fast."
Not all schema changes are equal. Your policy should distinguish between safe and dangerous changes.
Type widening (INT → BIGINT) looks safe but can break Avro/Protobuf consumers that hardcode types. Always check compatibility at the serialization level, not just SQL.
Data contracts are explicit agreements between producers and consumers about schema, semantics, SLAs, and ownership. They shift quality left — problems are caught before production.
A contract isn't just a schema. It includes SLAs (freshness, completeness), semantic definitions (what "amount" means), ownership, and change policies.
Contracts only work if they're enforced. Here's how they integrate into the development workflow.
Confluent Schema Registry, AWS Glue Schema Registry. Store versioned schemas, enforce compatibility rules (BACKWARD, FORWARD, FULL).
dbt 1.5+ supports contract: {enforced: true} on models. Columns, types, and constraints validated at build time.
Serialization formats with built-in schema evolution rules. Field additions are safe; removals require deprecation.
Data contracts create clear boundaries. Know who owns what — interviewers test this.
"Data contracts shift accountability to producers. Without contracts, the data team owns every downstream break. With contracts, producers own stability and consumers own adaptation."