Modern Data Stack

The “Modern Data Stack” is the term often used for a combination of tools often built around dbt in the middle. Most frequently it’s used to mean a combination of Fivetran, Snowflake & dbt.

Conceptually it’s a drive to use pre-built components to minimise the amount of custom software required to achieve a fairly capable and robust data & analytics stack. That involves:

1. A modular ingestion layer. Historically this was often Fivetran, but many other tools exist in this space. The principle however is that no organisation (especially small ones) shouldn’t build their own custom integrations with common platforms (for example, don’t build your own integration with Facebook - it’s a solved problem so you should just use the existing solutions). Some of these solutions are also free and open source like Airbyte or Meltano.

2. A scalable data warehouse solution, which doesn’t require significant development work to set up. In particular that you can access very large data storage options, with powerful compute - and not employ anyone additional in your organisation to run and operate it. Historically this has often been a choice between AWS Redshift, Snowflake and Google BigQuery, but since 2020 there have been many new entrants into this space. The most interesting of which are Trino, DataBricks & Materialize.

   Snowflake

   59 words·1 min

   AI Operations Infrastructure Development

   Scalable data warehouse deployment for small organisations.

3. A robust transformation framework, which enables great collaboration. In almost all cases, this means dbt, but other solutions like SQLMesh exist as alternatives.

   dbt

   343 words·2 mins

   Infrastructure Operations

   Operating and running a streamlined dbt operation.

At it’s core - the “Modern Data Stack” isn’t something you just buy, but an approach to data infrastructure which has become very common in organisations of all sizes in the last 5 years. Compared to previous approaches which involve a lot more proprietary software it can reduce maintenance burdens in teams and also make it much easier to find people who are familiar with your tooling (and so drastically improve team onboarding and hiring options).

If you’d like to find out more about how this can help you, let’s chat.