Senior Data Engineer – Azure Databricks Migration & Optimization

We’re migrating complex SQL Server logic (stored procedures, views, SSIS packages) from legacy systems straight into Azure Databricks + ADF — lift-and-shift style , no full redesign. The real work starts after migration: aggressively optimize for performance and cost, understand Spark physical plans inside-out, avoid hidden cross-joins or skew surprises, master Delta Lake retention pitfalls, and make smart compute choices that balance speed and DBUs.

If you love diving deep into Spark execution models, reading physical plans to fix subtle regressions, knowing exactly what VACUUM + OPTIMIZE does to time travel, and delivering production-grade cost savings without breaking business logic — this is your role.

Key Responsibilities

• Migrate complex SQL Server stored procs/views/SSIS to Databricks notebooks (PySpark / Spark SQL) with zero functional breakage
• Analyze & optimize Spark physical plans — elim...