The Skills Gap in Data Engineering Nobody Talks About
When people talk about the skills gap in data engineering, they usually mean the technical one. There are not enough […]
All, Opinion
All, Opinion
Why Data Engineers Need to Think Like Software Engineers
There is a version of data engineering that looks like this: a skilled analyst who learned Python, discovered that they
All, Opinion
The Hype Around Real-Time Data — Is It Worth It?
Real-time data is the most consistently overhyped concept in the data engineering industry. It has been overhyped for the better
All, Opinion
Why Most Data Teams Are Over-Engineering Their Stacks
There is a pattern that repeats itself across data teams with enough regularity that it deserves to be named. A
All, Pipelines
Data Quality Checks: How to Bake Them Into Your Pipeline
Bad data is one of the most insidious problems in data engineering. Unlike a pipeline failure — which is loud,
All, Pipelines
Handling Schema Changes Without Breaking Your Pipeline
Schema changes are inevitable. This is not a pessimistic take — it is just the reality of building data pipelines
All, Pipelines
Incremental Loads vs. Full Loads: When to Use Which
If you have spent any time building data pipelines, you have encountered this decision more times than you can count.
All, Pipelines
Building Reliable ELT Pipelines from Scratch
There is a meaningful difference between a pipeline that works and a pipeline that is reliable. A pipeline that works
All, Architecture
Event-Driven Architecture in Data Engineering
Most data pipelines are built around a simple assumption: data moves on a schedule. A job runs every hour, every
All, Architecture
Designing for Scale: Partitioning and Clustering Strategies
There is a particular kind of pain that data engineers know well. A query that runs fine in development —