Data & Intelligence Engineer

An executive asks why the dashboard says 40,000 signups when finance's number is 38,200. Three days later nobody has an answer, because the discrepancy is not in any one system — it is in the seams between four of them. Understanding how an event travels from a tap to a chart is what turns that three-day investigation into a twenty-minute one.

The four stages, and which one actually breaks

Every modern data stack does the same four things: ingestion moves data from source systems, storage holds it, transformation turns it into business concepts, and serving exposes it to dashboards and applications.

The non-obvious part is where incidents actually come from. Transformation code is the part engineers worry about, and it is the part least likely to break — it is version-controlled, reviewed, and tested. The overwhelming majority of data incidents originate at the ingestion boundary, where an upstream team owns the schema and has no idea you depend on it. The layer with the least clear ownership is the layer that fails.

Warehouse, lake, lakehouse

A data warehouse enforces structure on the way in: you define the schema first, and the system rejects anything that does not fit. A data lake stores raw files in object storage and applies structure only when you read them. A lakehouse puts a table format over object storage so you get lake economics with warehouse guarantees — atomic writes, schema evolution, and time travel.

The insight that matters for a real decision: the choice between them is far less important than whether you have a catalog and clear ownership. A lake without a catalog becomes a swamp, and the cost is never storage — storage is cheap. The cost is that nobody can find anything, and nobody trusts what they find.

Why data breaks quietly

Software failures are loud. A broken service returns errors, alerts fire, someone gets paged. Data failures are silent: the pipeline runs, the dashboard renders, and the number is simply wrong. Nothing crashes. This single property explains why data engineering invests so heavily in testing and observability compared to equivalent application work — there is no equivalent of a stack trace.

Failure modes

Silent schema drift — an upstream team renames a column. Ingestion maps the missing field to null, and the metric decays toward zero as old rows age out. Signal: a metric that slopes smoothly downward instead of dropping sharply. Real outages are step functions; drift is a ramp.

Timezone split-brain — events are stamped in UTC but aggregated in local time somewhere downstream. Signal: daily numbers that disagree by a few percent while weekly totals reconcile perfectly.

Late-arriving data — a mobile client buffers events offline and delivers them two days later. Signal: historical numbers that change when you re-run the same query, especially for the most recent few days.

Duplicate delivery — the pipeline guarantees at-least-once delivery and nothing de-duplicates on a stable key. Signal: counts inflated by a small percentage, worse on days with infrastructure incidents, because retries are what produce the duplicates.

Explaining this to stakeholders

On warehouse versus lake: a warehouse is a library where a librarian catalogues and shelves every book before it enters. A lake is a storage unit where you can throw anything through the door. A lakehouse is the storage unit with a librarian who indexes things as they arrive. All three hold your stuff; only two let you find it.

On why data-quality work looks like nothing: it is plumbing. Nobody admires the pipes, and the only time anyone thinks about them is when the ceiling is wet.

On late-arriving data: it works like expense reports. The month is not final the moment the month ends — it is final once everyone has submitted, and until then any total you quote is provisional.