Databricks Architect

Architect - Databricks (Cloud Data)
Job Title:

Architect - Databricks (Cloud Data)
Primary skills:

Databricks, PySpark, and architecture solutions.
Secondary skills:

Unity Catalog, StarBurst, Azure DevOps, CICD, and orchestration layers.
Mode of work:

Work from office
Experience:

13+ Years of Experience

Responsibilities:
· Lead migration from Native Spark to Databricks.
· Architect migration to Databricks.
· Build Data Governance solutions using tools like Unity Catalog and StarBurst.
· Develop orchestration workflows using Databricks and ADF.
· Implement CICD pipelines for Databricks in Azure DevOps.
· Process near-real-time data through Auto Loader and DLT pipelines.
· Design secure, scalable infrastructure for Delta Lake and Spark SQL.
· Optimize Databricks environments for cost-effectiveness.
· Extract complex business logic from on-prem solutions like SSIS, Informatica, Vertica, etc., i...