Position: Lead Azure Data Engineer, Hyderabad

Department: Information Technology | Role: Full-time | Experience: 7 to 10 Years | Number of Positions: 1 | Location: Hyderabad

Skillset:

Azure Data Bricks, Apache Spark, Delta Lake, ML Pipeline, ADF, ADLS, Azure DevOps, Medallion Architecture, Python, Pyspark, SQL, Scala/Java, Data Governance, Performance Optimization, Project Ownership, Leadership, Excellent English communication skills

Job Description:

About Us:

We provide companies with innovative technology solutions for everyday business problems. Our passion is to help clients become intelligent, information-driven organizations, where fact-based decision-making is embedded into daily operations, which leads to better processes and outcomes. Our team combines strategic consulting services with growth-enabling technologies to evaluate risk, manage data, and leverage AI and automated processes more effectively. With deep, big four consulting experience in business transformation and efficient processes, we are a game-changer in any operations strategy.

We are looking for an accomplished and dynamic Data Engineering Lead to join our team and drive the design, development, and delivery of cutting-edge data solutions. This role requires a balance of strong technical expertise, strategic leadership, and a consulting mindset. As the Lead Data Engineer, you will oversee the design and development of robust data pipelines and systems, manage and mentor a team of 5 to 7 engineers, and play a critical role in architecting innovative solutions tailored to client needs.

You will lead by example, fostering a culture of accountability, ownership, and continuous improvement while delivering impactful, scalable data solutions in a fast-paced, consulting environment.

Key Responsibilities:

1. Architecture & Delivery:

• Lead the end-to-end architecture of data pipelines leveraging Azure Databricks, Delta Lake, and Apache Spark.

• Design and implement the medallion architecture (Bronze, Silver, Gold) to structure scalable data lakes

• Oversee data ingestion from diverse sources, transformation pipelines, and delivery of analytics-ready datasets

• Define best practices for real-time and batch data processing patterns

1. Technical Leadership:

• Mentor and guide a team of data engineers; perform code reviews and enforce engineering best practices

• Collaborate with solution architects, DevOps, data scientists, and business stakeholders

• Set and maintain coding standards, data governance policies, and deployment strategies

1. Platform Engineering:

• Architect CI/CD pipelines for data projects using Azure DevOps and Git

• Optimize Spark jobs, Delta Lake performance, and storage cost management

• Implement observability, data quality checks, and job orchestration across the stack

1. Governance & Security:

• Implement RBAC, Key Vault, managed identities, and other security controls

• Ensure compliance with data privacy regulations (GDPR, HIPAA, etc.)

• Promote data cataloging and lineage through Azure Purview or equivalent tools.

Required Qualifications:

• 7 to 10 years of experience in data engineering with hands-on expertise in data pipeline development, architecture, and system optimization.

• 4+ years of hands-on experience with Azure Databricks, Apache Spark, and Delta Lake.

Core Competency:

o Azure Databricks Mastery:

• Expert in Apache Spark (via PySpark, SQL, and optionally Scala).

• Deep experience with Delta Lake: schema evolution, time travel, merge operations, optimization.

• Develop complex notebooks, jobs, and ML pipelines in Databricks.

• Hands-on with Databricks Job Workflows, cluster configurations, and job orchestration.

o Advanced Azure Ecosystem Proficiency:

• Azure Data Factory (ADF): building robust and reusable pipelines, parameterization, integration with Databricks.

• Azure Synapse Analytics: working knowledge of data movement, analytical workloads.

• Azure Data Lake Storage Gen2 (ADLS): hierarchical namespace, lifecycle management, access policies.

• Azure DevOps: Implementing CI/CD pipelines for notebooks, configuration-as-code.

• Azure Key Vault, Managed Identities, and RBAC: for securing pipeline access and secrets.

o Data Architecture & Design:

• Designing and implementing Medallion Architecture (Bronze, Silver, Gold layers).

• Dimensional modelling and denormalization for BI/reporting-ready datasets.

• Implementing data lakehouse principles using Delta.

• Designing for scalability, incremental ingestion, and idempotent pipelines.

Engineering & Development Skills:

o Programming & Frameworks:

• Strong with Python and PySpark.

• Skilled in SQL (including analytical functions, optimization).

• Optional: Scala or Java for Spark-based enhancements. o Data Quality & Observability:

• Implement data validation frameworks.

• Monitoring with logs, custom alerts, data profiling.

• Experience with data observability tools.

o Performance Optimization:

• Partitioning, Z-ordering, and caching techniques in Delta.

• Spark tuning and Cost-aware design (e.g., autoscaling, spot instances).

• Security & Governance:

o Enterprise Data Governance:

• Metadata cataloging and lineage using Azure Purview.

• Data masking, row-level security in Gold layer.

• Implementing audit trails for access and usage.

Leadership & Strategic Thinking:

o Project Ownership & Team Leadership:

• Driving data strategy and platform choices.

• Mentoring junior engineers; enforcing code quality & best practices.

• Conducting code reviews and architecture walkthroughs.

• Balancing delivery speed with data reliability and governance.

o Cross-functional Collaboration:

• Working closely with Product Managers, Architects, Data Scientists.

• Translating business requirements into scalable data solutions.

• Strong documentation and communication skills.

Nice to Have Skills:

• MLflow for model management and experiment tracking.

• Familiarity with BI tools like Power BI, Tableau.

• DataOps and test automation for pipelines.

Education:

• Bachelor’s or Master’s degree in computer science, Data Engineering, or a related field.

• Certifications such as Azure Data Engineer, Databricks Certified Data Engineer Professional is a plus.

Additional Information:

Why Join Us?

• Lead mission-critical data projects with enterprise impact.

• Collaborate with a high-performing team in a cloud-native environment.

• Influence data architecture and engineering strategy across the organization.

• Access to continuous learning, leadership programs, and certification support.

Required Qualification:

Bachelor of Engineering - Bachelor of Technology (B.E./B.Tech.)- IT/CS/E&CE/ Data Engineering/MCA

With a fast-growing analytics, business intelligence, and automation company