Announcing: Orchestration integrations and the Hex public API
Today we’re launching two new capabilities for Hex:
First-class integrations with popular orchestration tools Airflow and Dagster
A public API, allowing you to programmatically run Hex projects from other systems
Together, these are an incredibly customizable set of tools for integrating Hex into the rest of your stacks, workflows, and systems. We can’t wait to see where you take it!
Hex + Orchestration
While of course everyone thinks of Hex as the indispensable cornerstone of any data workflow, the reality is that we are but one piece in a larger stack. ETL, reverse ETL, custom internal services, model inference – all of these things get thrown in together, and need to work together seamlessly to provide high impact and great experiences to users.
This is why Orchestration tools like Airflow and Dagster exist. These tools let you build Directed Acyclic Graphs (or, DAGs), with nodes of jobs and dependencies that can run in scheduled, sequenced orders. This lets you, for instance, trigger a data transformation, and then a model inference run based on that data, and then update a visualization to show those outputs.
Historically, many people would also include legacy Python notebooks as part of DAGs, as a way to trigger logic or refresh outputs.
We've been hard at work modernizing every piece of the notebook interface, and now that extends to orchestration support as well: we’re announcing two new integrations with Hex for Dagster and Airflow. These enable you to add Hex projects to your DAGs (yes, a DAG in your DAG), programmatically triggering project runs.
This lets you do things like:
Trigger a Hex app to update cached values after an upstream transformation completes
Run a Hex project with passed-in input parameters based on a form submission in another app
Automatically kick off a model re-run through Hex when new source data lands
And, honestly, probably a million things we haven’t thought of, because you all are creative, crazy, beautiful people who continually blow our minds with what you do in Hex.
Note: this feature is available on Hex Teams plans and above.
Powered by Hex’s public API
These awesome integrations are both powered by our new public API. If you’re using another orchestration solution, or want to be able to do other kinds of automations with Hex, this new endpoint lets you run projects programmatically.
Here's a quick embedded demo of a Hex app that uses the Hex API to trigger a custom run of another app (which uses another powerful API, The Cat API).
There are four endpoints in v1.0.0: RunProject, GetRunStatus, CancelRun, and GetProjectRuns. Like the orchestration integrations, RunProject can be used both to trigger runs that set the user-facing cache of an app, and to generate custom one-off runs with specific input parameters.
See the full API docs here for more information and examples.
Note: this feature is available on Hex Teams plans and above.
Get building!
We hope you’re as excited about these features as we are, and we can’t wait to see what you build with them. There's no demo too small or janky, and no integration proposal too outlandish— we want to hear about it all. Hit us up on Twitter or send an old fashioned email to [email protected].