Azure Data Engineer – Data Pipelines & Databricks

Azure Data Engineer Data Pipelines & Databricks Location: Toronto, ON Key Responsibilities: Develop and maintain ETL/ELT pipelines using Azure Data Factory (ADF) and Azure Databricks. Implement data ingestion flows from diverse sources including Azure Blob Storage, Azure Data Lake, On-Prem SQL, and SFTP. Design and optimize data models and transformations using Oracle, Spark SQL, PySpark, SQL Server, Progress DB SQL. Build orchestration workflows in ADF using activities like Lookup, ForEach, Execute Pipeline, and Set Variable. Perform root cause analysis and resolve production issues in pipelines and notebooks. Collaborate on CI/CD pipeline creation using Azure DevOps or Jenkins. Apply performance tuning techniques to Azure Synapse Analytics and SQL Data Warehouse. Maintain documentation including runbooks, technical design specs, and QA test cases. Data Pipeline Engineering: Design and implement scalable, fault-tolerant data pipelines using Azure Synapse and Databricks. Ingest data f...