Daniel Sternberg: How data drives craft at Notion

Daniel Sternberg leads the data and AI teams at Notion which just surpassed 100M users. Last week, he shared the philosophies that inform his data team strategy at our first Friends of Data event.

Daniel shared perspectives on data team structure and ROI, to the balance of data vs. craft-based decisions, and one of his most influential books. It was a wide-ranging conversation with valuable nuggets no matter what kind of data person you are!

This transcript has been edited for brevity; for the full conversation, watch the recording above.

So Daniel, can you share how the data team is structured at Notion and walk us through how data fits in?

Yeah, for sure. So, Notion was the first place I've ever been where the data team started in a very centralized way. And I think the reason for that is there's a culture around respecting craft which leads the company toward more centralized structures; so, for example, we want data experts to have a team that works together.

I’m not saying that it scales forever like that necessarily, but that’s the set up. At its very core, Notion is all about being one holistic product that can do many things and so being together is actually really important.

Where does data fit in? We are a purely functionally organized company. I've been overseeing the overall data org for the last two years which includes data platform, data science, data engineering, and BI engineering, which is part of our Data Eng team. Now, I’m also overseeing AI and we report to the CTO. So we're part of the technical org, but parts of the team support some of the business side teams as well, like sales, marketing, and finance — on the data-end side in particular.

Notion is known for a sense of craft — are there times when the company drops the data to go with what feels right for the product?

You’re exactly right, craft and taste are very core to Notion and are taken into account of how things get built. Notion is a SaaS subscription business and you actually can't rely on data alone to make product feature decisions in a SaaS subscription business. This is because you can’t go from: “if i build this feature, it will drive this usage, which will drive this revenue” in this very clean, clear way.

And so, we use data to help us have intuitions about what some of the problems are and get customer feedback and qualitative feedback as well. We then have designers create different versions of how they could improve on that and dog food it internally for a while. And then we have data scientists think about: how can I measure if this was successful or not? And in some cases, we’re just looking to measure that we didn't screw things up too badly.

In my experience, a useful framework that I’ve used with my data teams has been: Is the job of the team I'm working with optimization? or Is their job to build new stuff that solves problems for customers?

If their job is optimization, it's going to be very data-driven. Growth teams tend to look more like this. Search at Notion also falls into this category.

But other teams, like our Docs team, can have high-level goals of how they’re improving experience in Notion, but can't measure every feature they launched to see if it drove the product forward.

How do you measure your data team’s ROI?

I mean, to be very clear, I do not have a dashboard where I can see: these are our success metrics for the data org. We have outcomes or OKRs we want to drive in a given quarter, but I don't have like the, “I can prove to you that data drove $16 million in revenue last month.”

There's a combination of an intuition and some data. There are exciting cases where there is some insight that you had that drives a very clear outcome. For example, a big value prop of Notion is using it for collaboration. So, we had set up a paywall, such that for a certain segment of users, when they created a new Notion workspace and added a second person, they got paywalled. And we saw that in the segment, not surprisingly, it had a big impact on how many of them added more people.

We also had lower-than-expected monetization rates from those people. And the insight was: we should instead set a limit around how much content they put into Notion after they add people. And once they hit a certain amount of content in Notion, then we'll put the paywall in, because at that point, they will have actually gotten value out of the collaboration. And that's a great insight, but that doesn't happen all the time.

There’s a lot of things you do on data and it’s worth differentiating between these types of outcomes: there are exploratory findings and there are the nuts and bolts.

Exploratory findings really shape how people think and show, over time, that we know how to measure our product. Or they’re ideas about behaviors we want to drive in the product that will help us drive more usage of Notion and ideally more revenue over time. They take a lot of work but are awesome.

“You will get better ROI if you don't think of the job as a glamorous job.”

Then, on the other side there are the nuts and bolts that just have to get done. For example, we need to be able to count how much money we're making accurately. Doing it well is really important. To that point, what I've learned over time is: you will get better ROI if you don't think of the job as a glamorous job. And that's okay.

How are you thinking about AI and how it will or won't change data work?

The only place where we've started using AI meaningfully to make decisions is in using an LLM to help with a first-pass of documenting all of our data models and then we have a human in the loop review it.

It's one of the reasons I actually feel like we can invest more in self-service on our teams now because our documentation is an order of magnitude better than it was a year ago. But we don't do a lot with AI in data work otherwise right now.

I think the concept that AI can help accelerate the work of people with some basic data skills is uncontroversial in the same way that it helps you as a co-pilot when you're writing code.

Do I think that we're gonna get to a place where I can just ask a question and it’s going to answer anything mildly complicated for me? It's not that it's not theoretically possible, but it misattributes where the difficulty is in the work — which is in the data work.

"[AI] misattributes where the difficulty is in the work — which is in the data work."

For example, for our documentation work, we had to have people who have thought about what the right data models to build are and then they had to build those models. After that, you need to go through the process of getting to a decent state of governance and documentation, which requires humans. And that's most of the work.

It also then needs to be a really well-formed question, which data teams spend a lot of time, I think, helping people ask better questions. Maybe AI can help you ask better questions over time.

To me, the company that is able to automate out a lot of that work is a company that already had a sizable data organization that helped bootstrap their way to building all of that underlying stuff. So until you can get AI to build all of that for you, you're not gonna be able to do it.

Cool, last question. Do you have a book, article, anything that you recommend to people who are in and around data that has informed your thinking or worldview?

Wow. That's great. I’ll be honest, I don't read a lot of like just data books these days. But Philosophical Investigations by Ludwig Wittgenstein has influenced my life and thinking in a lot of weird, deep ways. It is not a thing that you should pick up if you're just casually hanging out with your friends, though, you can read it in bits and pieces because it's short sections.

It's influenced me in catering how I interact with people who have different backgrounds about similar problems with this concept of language is just all about use and the meaning of words is not an intrinsic thing.

Thank you, Daniel. It was great having you.

Thank you for having me Barry.

Listen to the bonus Q&A questions in our full video:

• Has Notion needed to generate data? (31:47)

• How has the data team evolved at Notion since you’ve been there? (33:58)

• Running an experiment takes time, how do you assess what needs to run through an experiment and what doesn’t? (42:44)