Job Details

Role Overview:

We are seeking a highly skilled Senior Databricks Engineer to design, architect, and implement metadata-driven, scalable, and cost-efficient Lakehouse solutions on Azure. The ideal candidate will have strong expertise in Databricks, Delta Lake, and Azure Data Services with the ability to lead best practices, frameworks, and automation for enterprise-scale data platforms.

Key Responsibilities:

• Architect & Design Lakehouse Solutions
  • Build scalable, secure, and cost-optimized Lakehouse solutions using Azure Databricks and Delta Lake.
  • Leverage Azure Data Lake Storage (ADLS) for structured, semi-structured, and unstructured data integration.
  • Ensure support for ACID transactions, schema enforcement, time travel, governance, lineage, and extensibility.
• Data Modeling & Analytics
  • Design Lakehouse-based dimensional data models that align with modern data warehousing principles.
  • Optimize for scalable, high-performance analytical workloads and advanced analytics use cases.
• Best Practices & Frameworks
  • Define and establish best practices for job scheduling, cluster configuration, modular notebook design, and code reusability.
  • Lead development of metadata-driven onboarding templates, pipeline frameworks, and automation tools for accelerated source integration and processing.
• Collaboration & Compliance
  • Partner closely with Enterprise Data Management (EDM), IT Infrastructure, and IT Security teams to ensure architectural, operational, and compliance standards are met.
  • Provide technical leadership and mentorship to junior engineers.

• Strong hands-on expertise with Azure Databricks, Delta Lake, and Azure Data Lake Storage (ADLS).
• Proven experience in Lakehouse architecture, data modeling, and modern data warehousing principles.
• Knowledge of Azure cloud services (e.g., ADF, Synapse, Key Vault, Event Hub).
• Proficiency in Python, PySpark, and SQL for data engineering and analytics.
• Experience with cluster optimization, performance tuning, and cost management.
• Strong understanding of metadata-driven pipelines, automation frameworks, and CI/CD practices.
• Excellent collaboration, communication, and leadership skills.