I wrote the first post in this series in March 2017, when Great Expectations was a new project and the idea of applying unit-testing discipline to data pipelines was still enough of a novelty that I had to explain why you'd want to do it. Seven years later, data quality has become a recognized engineering discipline with dedicated tooling, dedicated job titles, and dedicated conference tracks. Worth looking back at what GE got right and what the field still hasn't solved.
What Great Expectations Got Right
The expectation as the primitive. Every data quality framework that's come along since GE has adopted some version of the expectation model — a named, parameterized assertion that runs against data and returns a structured result. Before GE, data quality "checks" were one-off scripts with no common interface, no standard output format, and no reusability. The expectation model gave the field a shared vocabulary.
Expectations as code. Storing expectations as JSON files that live in source control, are diffable in pull requests, and travel with the data pipeline they protect — this was the right choice in 2017 and it's still the right choice. Configuration-as-code for data quality has the same benefits as infrastructure-as-code: auditability, versioning, reviewability, reproducibility.
Separating definition from execution. The DataContext/suite/checkpoint architecture — define expectations in a suite, run them via a checkpoint, store results separately — gives you the flexibility to validate the same suite against multiple data sources, in multiple environments, on different schedules. That separation was not obvious from the early simple API, but it was the right architecture for production use.
Data Docs. Automatically generated documentation from expectation definitions and validation results made data quality visible to people who don't read JSON. The ability to say "here's a link to the validation report for this dataset" rather than "let me explain what the pipeline checks" changed how I communicate data quality to non-engineering stakeholders.
What the Field Still Hasn't Solved
The blank suite problem. Profiling helps, but generating a high-quality expectation suite for a new dataset still requires domain knowledge that can't be automated. This is inherent to the problem — you can't automatically know what "valid" means for your data without understanding the domain. Better tooling helps you bootstrap faster; it doesn't eliminate the domain knowledge requirement.
Cross-batch expectations at scale. Detecting that this batch's distribution looks meaningfully different from last week's batch — anomaly detection on data quality metrics — is still awkward with GE's native capabilities. The framework gives you the primitives but not the higher-level drift detection. You build it on top, or you use a separate monitoring tool alongside GE.
The maintenance overhead of large suite collections. When your data quality project has 50 suites covering 50 data assets, the question of "which suites need to be updated when the upstream schema changes?" is still a manual exercise. Lineage-aware suite management — automatically identifying which suites are affected by an upstream change — doesn't exist out of the box.
The Lasting Contribution
What Great Expectations gave the data engineering community is harder to measure than features: it established that data quality deserves engineering rigor. The same way tSQLt established in 2011 that database logic deserves unit tests, GE established in 2017 that data pipelines deserve data quality gates. Both were non-obvious claims when they were made. Both are obvious now — which means the frameworks did their job. As always, I'm here to help.