Azure Databricks Engineer - banff

Job Summary We are seeking a skilled Azure Databricks Engineer with strong experience in Azure Data Factory (ADF) to design, develop, and maintain scalable data pipelines and analytics solutions on the Azure cloud platform. The ideal candidate will have expertise in big data processing, ETL/ELT workflows, and distributed computing using Databricks.

Key Responsibilities

Design and implement scalable data pipelines using

Azure Data Factory (ADF)

Develop and optimize data processing workflows using

Azure Databricks (PySpark/Scala)

Build ETL/ELT processes for ingesting, transforming, and loading large datasets

Integrate data from multiple sources such as APIs, databases, and data lakes

Work with

Azure Data Lake Storage (ADLS)

and

Azure SQL Database

Monitor, troubleshoot, and optimize pipeline performance and data quality

Implement data security, governance, and compliance st...