DevOps Management for Azure Synapse Analytics - Part 1

Over the past few months, I have been "playing" with Azure Synapse Analytics. It is a Platform as a Service (PaaS) offering from Microsoft Azure that provides managed data warehouse capabilities with analytical service. In short, the user can not only ingest and manage data, but also run queries and perform analysis – whose results can be then fed to a machine learning model or a visualization model in Power BI. This analysis can be done using either using on-demand or dedicated resources.

An Azure Synapse Analytics resource can be deployed using the Azure Portal, PowerShell commands, Command-Line Interface, ARM templates and even Terraform modules. However, to deploy at an enterprise-scale, there are several factors that one needs to consider and the most important being DevOps.

What to expect?

• In this two-part series, I will explain how can we create and manage an Azure Synapse Analytics service across non-production and production environments.

• Part One will focus on setting up CI/CD best practices to promote subsequent synapse services such as pipelines, scripts, notebooks, linked services, triggers etc. to other synapse workspaces.

• Part Two will focus on promoting database objects such as tables, stored procedures, views, logins etc., to other synapse workspaces.

Prerequisites

• The organization has a synapse workspace within a development environment, where developers are actively creating pipelines, database objects, analyzing data etc. In the example, I have the development workspace named as synapsedemo121.

• The organization has another synapse workspace created in a test/production environment, which will be our target environment, whose artifacts would be created from the development workspace using CI/CD best practices. In the example, I have the target workspace named as synapsedemo121test.

• The organization also has a project created within an Azure DevOps account.

Part One - Promoting Synapse Pipeline Artifacts using CI/CD

Step 1: Create an Azure Repo within Azure DevOps project

Log into Azure DevOps account and create a new repository using the Repos option on the left menu.

Step 2: Configuring Git Repository for Synapse Workspace

Under the Manage option in Synapse Studio, configure git repository and associate the synapse workspace.

Provide necessary organization details for configuring the repository.

On successfully configuring git, the repository information would look like the illustration below.

Step 3: Creating a Synapse Pipeline

Let's first create new feature branch within the synapse workspace as shown below.

Within the feature branch, I have created a pipeline, that copies a sample invoice bill data from one data lake storage to another. The following shows two linked services that were created for the pipeline.

Execute the pipeline for testing. Commit to save the changes.

Now once the pipeline functions well, create a pull request to merge the new pipeline and subsequent changes to the Azure Repo.

On successful completion of the pull request.

This will create the resource configuration files in json format within the Azure Repo.

Now to deploy or promote the synapse workspace to another environment, we will have to first publish the changes.

On publishing, the Azure Resource Manager templates (template and parameters json files) are created in the publish_branch specified, while configuring the git repository. By default, this is workspace_publish.

Let's merge this to our main branch using a pull request.

On successful completion of the pull request.

Once the ARM template files are merged along with other workspace configuration files, the file structure in Azure Repo would look something like below.

Step 4: Creating a Build Pipeline

To create a build pipeline, select Pipelines from the left menu in Azure DevOps.

We will be using the classic editor option as highlighted in the figure below.

Select the Azure Repo which includes our ARM template files.

Next, select an empty job template.

Select the + option and search for Publish build artifacts task.

Provide the path to the ARM template and parameters files that resides within the Synapse Resource repo.

For automatically triggering the build pipeline whenever changes are made to the Azure Repo, enable the continuous integration option.

On successfully triggering the build pipeline, the artifacts are created in a drop location. These files created as part of the build can be accessed from the option highlighted in the illustration below.

Step 5: Creating a Release Pipeline

To create a release pipeline, select + New from the Releases option. Let’s name the pipeline as Synapse Workspace Release.

Provide the build pipeline details at the Artifacts section. Name the stage as Test, denoting that we are promoting the synapse workspace from the development environment.

Add a Synapse deployment task. Provide all the required parameters and the target synapse workspace name (synapsedemo121test is the test synapse workspace – can be created in any resource group or region). Click Save and Create release.

On successful completion of the release pipeline, the workspace elements such as scripts, notebooks, triggers, linked services, datasets, pipelines etc., gets promoted and created within the test synapse workspace (synapsedemo121test).

On successful completion of the release pipeline.

Let's check whether the pipeline was created within the test synapse workspace.

Looks like the pipeline was created successfully! Now let's check the linked services.

Nice! Since we have both the synapse pipeline and linked services, let's execute the pipeline.

Wonderful! The pipeline seems to execute successfully as well.

Pro Tip: Verify the connection for linked services in the target environment before executing any pipelines.

Inference

Although this method allows user to deploy synapse pipelines & subsequent artifacts from one environment to another, it does not promote any database objects. In the next part, we will exactly learn how to promote these objects and in no time one can master the DevOps implementation for Azure Synapse Analytics.

Thanks for reading!