Deepening Our Snowpark Integration

Building upon our blog Introducing Snowpark for Python in Hex, we are excited to introduce new enhancements to our Snowpark integration! Now in Hex data scientists can move between SQL, Python, and visual cells while leveraging a Snowpark dataframe. More on this later!

Before we delve into the details, let's take a moment to recap what Snowpark is all about. At a high level, Snowpark extends Snowflake’s capabilities by empowering developers to write custom code against their data in various non-SQL programming languages including Java, Scala, and Python. Hex’s Snowpark integration specifically supports Snowpark for Python, which is what we will be focusing on in this post.

Hex’s Existing Snowpark Integration

Our existing integration brings enhanced security and an incredibly straightforward way to create a Snowpark session. Set up is a breeze - all you need to do is toggle the Snowpark option within your Snowflake data connection in Hex. Hex will then leverage this Snowflake connection to create a Snowpark session. We affectionately call this Hex’s Snowpark Easy Button!

With the Easy Button, users don't need to expose their credentials in plain text or manage hidden files just to create a Snowpark session. We piggyback off the existing Snowflake connection that has been created. This connection is created and managed by a Hex Admin. For those interested in how this works, it’s all seamlessly managed by our trusty built-in package, hextoolkit.

Here is an example of what is required to create a Snowpark session outside of Hex vs in Hex:

vs.

In terms of simplicity, Hex takes the hassle out of pre-installing packages in your local environment. We've already pre-baked Hex projects with all the essential packages needed to run Snowpark, and we diligently keep them up-to-date with the latest stable releases.

Moreover, we've allowed users to use a first-class SQL cell and still return a Snowpark dataframe. This provides auto-complete, syntax highlighting, and formatting in SQL while working with a Snowpark dataframe. If you're wondering how this works, don't worry; we'll review it below!

Exploring What’s New

Snowpark employs lazy evaluation, making it ideal for working with big data. Under the hood, it actually translates Snowpark code into SQL to leverage Snowflake’s SQL Engine. That’s how we get the distributed compute - let's see this in action!

Here is a link to the notebook above!

SQL, Python, and visual cells with Snowpark

If you're new to Hex one thing you'll learn to love is the ability to switch between different cell types. Imagine you're working on a SQL cell, performing complex table joins, and suddenly, you want to switch to a Python cell, and later, perhaps to a visual cell. You can do this effortlessly because, under the hood, you're working with a Pandas dataframe.

You may have seen this ability referred to as a polyglot workflow. Wouldn't it be great to have this flexibility between SQL, Python, and visual cells while leveraging a Snowpark dataframe? Well, as it turns out, we thought so too!

Here is a link to the notebook above!

In addition to our deepened integration with Snowpark, we also wanted to account for the users that are a bit more security conscious. With the Snowpark toggle enabled, you get all the benefits of our Snowpark integration with the security benefits of OAuth. You can read more about our Snowflake OAuth configuration in our documentation.

When To Use Snowpark?

You might be wondering when to use Snowpark. Let’s first start with a basic use case. You've likely encountered memory limits in any notebook environment, including Hex. With our Snowpark integration, you can now perform traditional analyses without worrying about memory consumption in the kernel. Say you perform a select * from my_large_table and your kernel runs out of memory. You can simply select a Snowpark dataframe as your return mode. Then, leverage that same dataframe in any other SQL, Python, or visual cells and enjoy all the Hex features you love.

Once you have a solid understanding of how to work with Snowpark dataframes, explore more advanced use cases of Snowpark, such as data engineering and machine learning workloads:

• Develop comprehensive data pipelines using Snowpark dataframes.

  • Explore Snowflake Streams for a robust Change Data Capture (CDC) pipeline.

  • Schedule your pipeline using Hex’s built-in scheduling or integrate with orchestration tools like Airflow or Dagster through our integrations.

• Leverage Snowpark ML for a streamlined approach to machine learning in Snowflake.

  • For a detailed demonstration, explore the Snowpark ML + Stored Procedure Tutorial.

In summary, Hex has deepened our Snowpark integration. We now provide users the flexibility to switch between SQL, Python, and visual cells all while leveraging your Snowflake warehouse. Under the hood, this is done using Snowpark dataframes. In addition, we support OAuth while working with a Snowpark session. If you want to take Snowpark + Hex for test drive, you can start a free trial!