Azure Projects

I have included examples from a range of different Azure resources for a general overview of my experience. A more detailed example of one of my Azure function apps can be seen in the Python page.

Data Factory Pipelines

execute_notebook

This pipeline executes a durable Azure function app, monitors its status until the run completes, and then returns the appropriate output. This was used to execute a jupyter notebook through an Azure function app.

mis_summary

This pipeline executes a calculation stored procedure to populate an aggregate table. There is retry logic in place to account for a resource lock error which sometimes occurs when too many pipelines are querying the transaction table that this stored procedures pulls data from.

Data Factory Dataflow

dflow_service_users

This dataflow loads a json file retrieved from an API, and then performs a series of transformations before loading the data into a SQL table. The transformations include flattening the json data, validating the data, performing lookups and deriving values.

Function App

The below function app was created to execute jupyter notebooks. The function has an HTTP trigger, which executes an orchestrator function that monitors the status of the notebook execution. The orchestrator function is a durable function, which allows for long-running operations. The orchestrator executes the activity function, which executes the notebook using papermill. Below are the three main functions used.

Loading code...

Loading code...

Loading code...

Logic App

FetchSourceFiles

This logic app was used as a file listener to copy a list of source files from an SFTP into an Azure blob storage container.

GetPipelineAnalytics

This logic app was used to get pipeline log data from an analytics workspace and save it into azure blob storage. This would subsequently be used to load the data into a SQL table for reporting.