About 4.5 years ago I learned about dbt. I was at Data Council in SF at the Holiday Inn on Van Ness. I watched several modern data ecosystem tools and technologies make their debut such as Dagster and Admundsen, but one company that was not on the agenda grabbed most of my attention.
I was eating a box lunch at a hightop, when an energetic and eager man walked up with a laptop open displaying a terminal and an IDE. This was Drew Banin, Co-Founder of dbt. He quickly ran through everything that was possible with dbt with thoughtful finesse and blew me away.
Could transforming data be this much easier?
Well now there are over 90,000 of us who think so and it was incredible coming together last week to share and learn how everyone is doing more with data.
The Modern Data Stack has gotten complex
dbt rightly highlighted the biggest challenge of using the tool today: complexity. In the early days of dbt you had to convince your team that doing transformations in the warehouse made sense and that SQL + Jinja was all that you needed.
Today teams and organizations have so fully adopted dbt that in some cases the number of models now exceeds what a reasonable person can reason about.
Even when adopting dbt best practices around staging and production tables the DAGs start to get unwieldy after you have a couple hundred models, especially when there are multiple teams working on it at the same time. Once you have a couple thousand models it becomes unclear what the impact of a change in one model will do to another or to some downstream app for analytics or automation.
It also becomes challenging to find where an issue actually originated — which upstream table has the bug? And thus many products in the dbt ecosystem have popped up the past few years to make this more manageable such as Monte Carlo, Datafold, and Metaplane.
But now dbt is starting to provide their own tools to better navigate these large projects. During the keynote they announced the dbt Mesh and many feature to support this model of working with data:
I expect them to continue to invest here and I hope they also guide people towards creating fewer models in the first place that are more efficient.
Maturing Ecosystem
The dbt community has grown dramatically but the better measure here of how well the community is doing is how many active members have stuck with it over the years. I ran into familiar faces at Coalesce and 2/3 of this years sponsors sponsored last year too.
Another trend I kept hearing around the rare water coolers (this would be my one recommendation for next year, less sodas more water) was around shifting from PLG focus to more of a sales focus. Data tools can gain traction with technical practitioners but the allure of high ACV contracts complete with SSO & BAAs beckons us all up the food chain.
There also seemed to be a pretty healthy relationship with AI at the conference. No one was going too off the rails about how it would change everything. But many teams were creating thoughtful experiences around how AI could help navigate the complexity that is data at every company.
Love this community
Hats off to whoever at dbt Labs spearheaded funding more creative booths for Coalesce. Last year had a several notable examples that took full advantage of it and Hex had a blast creating a 1950s diner to hang with all of you.
This year we upped our pho-retail game and were quite pleased how the 90s Hex 3.0 computer software store came together.
…but I don't think anyone was prepared for puppies. Congrats to Secoda for winning the game 🙇. Not sure how anyone tops that but somehow I’m sure someone will next year.
There were also a ton of great talks that are now all on Youtube for you to check out including one by me and Izzy on the metadata architecture behind Hex Magic (recording) and one by our own analytics engineer Amanda about how Hex does marketing attribution (recording).
Coalesce 2024
The parties were fun but not too rowdy, the talks were stimulating & amusing, and competing companies all felt like friends. This is a testament to the community norms and culture dbt Labs has created from day one. I am proud to be part of this community and look forward to seeing everyone next year where every big conference ends up: Vegas.
If you didn’t make your way over to the Coalesce freebies section, you can sign up for a free extended trial of Hex here: 30-day trial
And if you are a dbt Cloud user check out the new dbt Semantic Layer integration in Hex!