This post is cross-posted from Ladder’s engineering blog. Shout out Ladder for forcing me to write something!
At Ladder, we use Datomic as our primary data store and Google BigQuery as our data warehouse. We’ve iterated on how we send data from Datomic to BigQuery - starting from a nightly job dumping every entity and ending with a streaming solution with just seconds of latency. In this post, we’ll walk through bucketing Datomic entities into BigQuery tables, transforming entities into rows, and using one of Datomic’s superpowers to stream changed entities into BigQuery.