Healthcare data teams: How to handle endless requests without burning out

Overseeing data at a healthcare or life sciences company is complex task — and yet the work is quite literally changing our lives (🙏). From advancing cancer treatment, to getting real-world evidence to clinicians, to improving patient treatment plans, this work is transforming what’s possible with our health every day.

But meeting the demand for insights in this field is exceptionally challenging. At the heart of these company’s life-saving innovations lies intricate data ecosystems. Massive amounts of healthcare data (sometimes petabytes!) needs to be piped from multiple sources and then cleaned to be readily accessible, operational, and presented in human-readable formats.

In addition to setting up business partners for success with data, these data teams are also responsible for experimenting, prototyping, and building data products to drive the bottom line.

This creates an interesting balancing act: how much time should be spent on complex analysis versus setting up stakeholders with self-service analytics? With new data products, the data teams at Recursion, OM1, and Charlie Health are doing both more easily — changing our lives, and theirs.

1. Making drug discovery more cost-effective

with stakeholder self-serve and data team prototyping happening all in the same place.

Recursion is a leader in drug discovery for cancer and other rare diseases. What sets them apart is their dedication to speed. In the drug discovery industry, it can take up to seven years(!) to advance ideas to molecules, but at Recursion, the pipeline is moving faster — and their data team is central to maintaining this speed.

To deliver drugs to the market efficiently, their data team needs to support multi-disciplinary teams with diverse data needs; chemists, software engineers, biologists, and leadership all need to work with data, but have varying technical abilities. This was causing data processes to happen in different tools, across notebooks and spreadsheets, with results displayed in PowerPoints. The fragmentation began to slow down decisions.

To build a more cohesive data experience, Recursion brought on new tools that would let them bring disparate data together. First, they brought on a cloud data warehouse to centralize the data and then added Hex on top to use as a centralized front-end to work with the data.

All of these teams can now query different types of data (cell types, different genes, proteins, chemical perturbations) in one tool using SQL or Python or no-code — the modality that they are most familiar with — while the data team can do advanced prototyping, combining, and visualization, and work with raw outputs. “It’s kind of like a watering hole where everybody can go and explore what other people are up to,” said Genevieve Roberts, a Senior Data Scientist at Recursion.

By adopting tools that let teams centralize and work with data from multiple sources, Recursion has developed a new culture of collaboration that drives more wholistic knowledge more quickly. It also frees up time for experimental design. The shared understanding of data processes and unified data architecture allows different experts to work together smoothly, helping the company have a more cost-effective data flow, bringing in human expertise exactly where it's needed most.

Read more about how Recursion’s data team accelerated its drug discovery loop.

2. Enabling self-serve on 330 million data points

with a complex data app that has a user-friendly interface for sales and marketing.

OM1’s data team was getting bombarded with requests from their sales and marketing teams. They sell massive healthcare datasets (from hundreds of billions of data points) to clinicians and researchers. But before buying a dataset, providers and physicians often need to know if a specific type of data is included in it.

With data from more than 330 anonymized million patient records, it was impossible for the sales and marketing teams to sort through the datasets on their own and respond back to buyers quickly without the help of the data team. And for the data team, it was impossible to anticipate every request.

“Our business users know what they want, they just can’t articulate it into the database or into the data structures,” Zachary Bryant, Associate Director of Data and Analytics said.

For every request, the data team would need to talk to the product team, then get an analytics engineer to make updates to a seed file that would feed into a dbt model (or series of them), that would characterize the data. A ticket would then be sent to the data engineering team to do a PR review to merge the change into the pipeline, and finally the pipeline would be reran. After seeing the result, the sales or marketing would have another follow-up question from a customer and the process would start all over again.

To reduce this ticket-train and time spent on this repetitive cycle, OM1 onboarded Hex and built a data application that let business stakeholders explore data and manage mappings that for categories like medications and conditions through a point-and-click UI.

The app, which sits on top of Snowflake, does the technical work of a data engineer, but was easy for the data team to build with the ability to switch between SQL and Python and with Hex’s Snowpark integration. “Under the hood, there is a bunch of business logic and heuristics using SQL, Python, dbt models, and some UI components to pull this all together," Zachary explained.

Now, sales and marketing can get clearer answers on their own to quickly respond to clinicians, researchers, and scientists before they buy a dataset. And internally, OM1’s data app is saving the data team hours of time every week by delivering consistent, standardized, governed answers that are backed by business logic.

Read OM1’s full story here and see their open-source Mapping Manager.

3. Operationalizing data to match patients and therapists

with a smart data app that delivers real-time, data-driven recommendations for human-vetted placement.

Charlie Health has provided virtual therapy to more than 40,000 adolescents, young adults, and families facing serious mental health issues. Part of what makes Charlie Health’s patient outcomes so successful is the placement of new clients in therapy groups with people who are experiencing extremely similar mental health challenges and life circumstances — both of which involve many nuances.

With rapid growth in the last year (they now see as many new patients each week as they did in all of 2021), Mark Megerian, Charlie Health’s Technology Fellow, wondered if there was a way to use the power of data analytics to support the personalized, clinical, and nuanced process of client placement and group dynamics at scale.

As an experiment, the data team built a powerful data app which uses a mix of patient clustering, cosine similarity, ideal match analysis, and cohort selection to surface the best therapy group for every patient, during an initial consultation call, within minutes.

Within the data app, clinical admissions therapists, who are responsible for client placement, are presented with an assortment of therapists and therapy groups available ranked by a Fit Score that is designed to give each patient the best chances of treatment success and the best chance to form meaningful bonds with other members of their group.

The interactive app can filter the open sessions for weekdays and times on top of the Fit Score rankings and can be modified quickly to incorporate any new criteria.

Under the hood, data is pulled from Snowflake, Qualtrics, and TigerGraph into Hex. “The innovation is using Hex to prototype and scale a previously manual matching process that results in a much better experience for people,” said Mark. “We can now automate this process without the data team having to use an engineering team to build a full UI from scratch.”

Using Hex for rapid prototyping and UI development allowed the team at Charlie Health to quickly turn ideas into action, and to respond immediately to feedback gathered from their beta users. Most critically, however, is that their patients are getting even better treatment plans with data-driven decisions.

The key to healthcare data success: balancing self-service and ad hoc data analysis

Finding the sweet spot between self-service analytics and hands-on data work doesn't have to be complicated. With the right tools, healthcare and life sciences data teams can give their stakeholders the independence they need while continuing advanced analytics work in the same platform.