Lead I

Role Description
Key Responsibilities:
Design, build, and maintain robust data pipelines using Azure Data Factory (ADF) to ingest data from diverse sources including APIs, Excel files, Azure Data Lake Storage Gen2 (ADLS Gen2), and SharePoint, ensuring secure access and efficient data flow.
Configure and manage SharePoint integration within ADF pipelines, handling authentication and access criteria effectively.
Develop complex data transformations using ADF Mapping Data Flows, applying best practices to create scalable, maintainable ETL/ELT workflows.
Implement and champion metadata-driven architecture for enhanced scalability, modularity, and reusability of data pipelines.
Perform rigorous data validation and testing to maintain high data quality and accuracy across ingestion and transformation stages.
Build dimensional models and apply data warehousing concepts to structure data optimally for analytics and reporting.
Leverage Azure Databricks for big data proce...