BI Tools & Hex

Why advanced data teams need both BI tools and flexible data workspaces

We sometimes talk to data scientists who use Hex for a couple of days and go "Wow! I'm going to cancel our {{ BI tool name }} contract and just use Hex for everything!" We always love to hear the passion, but 99% of the time recommend that they please not do that. Obviously, we still recommend they use Hex as well! Sometimes, of course, we also hear the opposite sentiment: "Why should we use Hex, if we already have {{ BI tool name }}?".

There can be some confusion around this, so we're going to explain once and for all how Hex compares to standard BI tools— and why they usually work best in tandem, rather than instead of one another.

Just want the punchline? BI tools are great at self-serve analytics for the entire company via point-and-click exploration and tiled dashboards. Hex is built for data analysts and scientists to do deeper exploration and modeling— The things that literally cannot be done in BI tools. It's also much more efficient to prototype new analytical projects in Hex, rather than commit to the overhead of ETL'ing data and building a new BI tool model.

What's BI?

Business Intelligence, or BI, is a pretty broad term that theoretically encompasses any use of data to drive better business outcomes. In practice, it usually refers to the work done with point-and-click tools like Looker, Tableau, Metabase, and their kin. These "BI tools" sit atop your database and let the entire company self-serve simple insights without writing code.

Business Intelligence is all about knowing what's going on with the day-to-day of your organization, and specifically about granting that power to non-technical users. You probably hear a lot about "self-serve" and "data for everyone" when researching BI tools.

Looker: "Give everyone the power to discover the answers they need to sophisticated questions."

Metabase: "The easy, open source way for everyone in your company to ask questions and learn from data."

Tableau: "Trusted data for everyone with Tableau"

PowerBI: "Enable everyone at every level of your organization to make confident decisions using up-to-the-minute analytics."

Having this self-serve ability is an important part of a data-driven organization, and can relieve a ton of pressure on a data team. Data teams typically field a huge number of "simple" requests that actually take an inordinate amount of time to understand, scope, create, deliver, and inevitably iterate on. With a good BI tool deployed, users can do most of that themselves, and just reach out to the data team when they have questions or get stuck. This opens up a whole world of impactful non-reactive data work that the team can now focus on.

Hopefully now you have put down the phone, or deleted that email draft canceling your BI subscription (BI vendors: kickback checks can be directed to [email protected]. Crypto accepted.)

Okay, so why do I also need Hex?

There's two reasons:

  1. BI tools are great for simpler analysis, but peter out when things get complex and are often not enough to satisfy the needs of technical data practitioners. Hex allows for analytical depth and power that goes far beyond what's possible with a BI tool.

  2. BI tools require big operational and organizational lifts to deploy. Even if already deployed, they require a significant time investment to begin working with a new dataset. Hex provides increased operational speed and efficiency, letting data practitioners get straight into analysis without overhead or unnecessary time investment.

1. Analytical Depth & Power

There is a ton of important data work that simply cannot be done inside of BI tools. Predictive modeling, complex statistical analyses, geospatial work, unstructured data, machine learning— The list of important analysis that can't be done in a SQL based BI tool is actually pretty long. A nascent data team may not notice this at first, but as a well deployed BI tool takes the day-to-day reactive pressure off of a data team (see above), their time is freed up to spend more time doing this deeper work. Any strong data organization will have an appetite for doing these kinds of projects.

BI is the tip of the iceberg of data work
BI is the tip of the iceberg of data work

To be clear, it's not just that BI tools are inefficient or kind of clunky for these purposes— They literally were not designed for this kind of data work and cannot do it, full stop. As a data team ventures more and more into complex data projects, they'll have to expand into using Python, R, and other tools like notebooks and SQL IDEs. The data chaos that a BI tool was intended to extinguish will begin to rear its head again, with analysis being done in local environments, csv files being emailed back and forth, and nobody ever sure which version is the final one that should be used in the board meeting.

This is where Hex comes in. Hex is a data workspace that empowers the technical data practitioners on your team to rapidly collaborate on powerful and complex analysis— and then easily share those results with stakeholders in a similar way to a BI tool. Hex doesn't aim to replace self-serve BI, rather allow a strong data team to do complex and deep data work without regressing back into data chaos.

2. Operational Speed & Efficiency

The blessing and curse of BI tools is their strong governance layers. An organization can expose more direct interfaces to data because the data team has taken the time to "model" it, defining metrics and preparing it for non-technical consumption. This modeling pays dividends, but takes time. It also requires data to be readily available in a database attached to the BI tool, which can mean requests to the data engineering team and new ETL pipelines. If datasets ultimately don't become widely used, or really were only ever going to be used for a point analysis by the data team, this can become a huge waste of time.

The hidden labor of a BI interface
The hidden labor of a BI interface

In this case, a flexible data workspace like Hex is an enormous productivity boost and frustration reliever for exploratory analysis and initial forays into data. The ability to combine data from databases, files, and APIs in the same workspace means analysts can get started prototyping without organizational or technical blockers. Being able to seamlessly intertwine Python, SQL, and no-code charts means they can use the right tool for the right job without resorting to hacky workarounds. Despite this flexibility and speed of exploration, all the work is still collaborative, version controlled, and ultimately easy to share.

If a Hex exploration shows promise as something that would be a valuable addition to the core data dictionary, it can be off-loaded and codified into the BI tool model. If the exploration hits a dead end, or simply served its decision-making purpose and doesn't need to be available for self-serve, it's no big deal. There's no tech debt, nothing to spin down, and no time was wasted on modeling the data for a wide audience or to fit tight pull request guidelines.

This might sound contrived; It's really not. Having personally built hundreds of LookML models, I can confidently say that few (work-related) things are as enjoyable as exploring well-modeled data in Looker for the first time. However, it can also be phenomenally clunky and frustrating to try and figure out a new and unmodeled dataset with Looker as the only tool in your arsenal.

A real life* scenario

*Ok fine, it's not real life. Based On A True Story, though.

Vandelay Industries, looking to democratize data across their organization (again, commission checks to [email protected]), purchases Looker. They build a great LookML model, roll it out widely, and after a few months the entire company is self-serving on simple reports. This frees up the data team from the constant requests for data dumps they were fielding. Finally, they can focus on what they'd hoped to do when they were hired! They dive eagerly into predictive analysis and statistical modeling, exploring unstructured telemetry data, and other more complex data science and analytics.

Looker's SQL & LookML interface can't support most of these projects, so the team starts using notebooks. They're finding really great insights, but are getting frustrated by the developer experience and quirks of trying to collaborate in their local environments. It's tough to connect to a database like they could with Looker, so there's a lot of copying and pasting of queries into SQL editors and loading of .csv files into notebooks. This is slow and sometimes even impossible with large datasets, so some avenues of exploration are unfortunately discarded. Lots of different versions of notebooks start flying around, and compared to Looker, it’s become way harder to share a finished result with stakeholders. Having failed at trying to help their users install Jupyter, the data team ends up doing a lot of excel exporting, downloading of static .pngs, and creating of pdf reports.

The Vandelay Industries data team
The Vandelay Industries data team

It’s really hard to reproduce and collaborate on this work, so progress is slow and the stakeholders aren’t excited about it either. They start to lose trust, and the whole system feels chaotic, inefficient, and just not that impactful. The data team is really excited about the work they're doing, but frustrated with the clunky development experience and the feeling that nobody's really looking at their results.

Hex comes in and gives the data team a collaborative environment where they can easily work on powerful analyses in Python and SQL with no limitations on complexity. Like in Looker, they can connect Hex straight to the data warehouse without using another tool or downloading data. Unlike in Looker, they can also blend in data from other sources, or even mock out and change values. Once completed, projects can be rapidly turned into data apps and shared with nontechnical stakeholders just as easily as Looker dashboards. The data team’s work is both easier to do, and gets seen by more end users. It's actually being used to make decisions and feels more efficient and more impactful.

End users continue to use Looker heavily for self-serve and simple BI, but rely on Hex apps delivered by the data team for answering complex questions and consuming more flexible reports. After a few months, the data team tends to begin a lot of their new analyses in Hex, rapidly collaborating and exploring new datasets together with more flexibility and speed. As projects develop, they're mindful to move work to Looker where appropriate. Some SQL heavy Hex projects are ultimately spun off and codified into the Looker model, but more complex analyses stay in Hex.

And everyone lived happily ever after.

Bottom Line

Not only do these tools coexist nicely, many organizations actually need both in order to build a powerful and efficient data function.

Already have a BI tool, but relegate anything that doesn't fit in it to a spiderweb of other tools and services? You should probably try Hex. Don't have a BI tool yet, but are looking to be more data driven? Hex can get you off the ground ASAP while you go through the process of spinning up a complicated BI tool deployment.

This is something we think a lot about at Hex, where we're creating a platform that makes it easy to build and share interactive data products which can help teams be more impactful.

If this is is interesting, click below to get started, or to check out opportunities to join our team.