How did we use DBT and BigQuery to manage late arriving web events?

How we process half a billion web records a day, without spending half a billion everyday. This blog explains a key mechanism in our data platform infrastructure which enables our web data asset to be refreshed hourly and process late arriving web events.

Here’s a basic architecture diagram of our data platform:

As you can see we use DBT as a workflow and data transformation layer, and BigQuery for data storing and querying.