Databricks, Pyspark

Join a high-impact data engineering team where you’ll shape modern analytics platforms and help turn raw data into trusted, actionable insights. In this role, you’ll lead the design and delivery of scalable data pipelines on Databricks using PySpark, partnering closely with analysts, data scientists, and platform teams to enable faster decision-making across the business. You’ll bring strong engineering discipline—clean code, performance tuning, and reliable operations—while guiding best practices and mentoring teammates. If you enjoy solving complex data challenges, optimizing distributed workloads, and building systems that are resilient, secure, and easy to evolve, this is a great opportunity to drive meaningful outcomes in a collaborative, growth-focused environment.

Responsibilities

Key Responsibilities

• Lead the development of end-to-end data pipelines on Databricks using PySpark for batch and incremental processing. • Design scalable data models and curated datasets to support analytics and downstream consumption. • Write and optimize advanced SQL for transformations, validations, and performance-critical queries. • Implement robust data quality checks, reconciliation logic, and monitoring to ensure trusted datasets. • Tune Spark jobs for performance and cost efficiency (partitioning, caching, file formats, cluster sizing). • Establish coding standards, reusable frameworks, and review practices to improve maintainability. • Collaborate with stakeholders to translate requirements into technical designs and delivery plans. • Troubleshoot production issues, perform root-cause analysis, and drive preventive improvements. • Mentor team members and provide technical guidance across design, implementation, and optimization.

Minimum Qualifications

• BTECH, MTECH, MCA, or MSC in Computer Science, Information Technology, or a related field. • 6–8 years of overall experience in data engineering or large-scale data processing roles. • Strong hands-on experience with PySpark for distributed data processing and transformation logic. • Strong hands-on experience with Databricks for building, running, and managing data workloads. • Proficiency in Advanced SQL including complex joins, window functions, and query optimization. • Experience building reliable pipelines with strong focus on data quality, performance, and stability

Requirements

Technology->Analytics - Solutions->SQL Server - Analytics Technology->Big Data - Data Processing->PySpark Technology->Data Engineering->Databricks

Additional:

Preferred Qualifications

• Experience designing lakehouse-style architectures and organizing curated layers for analytics readiness. • Strong experience with Spark optimization techniques and handling large-scale datasets efficiently. • Ability to build reusable PySpark utilities/frameworks for ingestion, transformation, and validation patterns. • Experience with orchestration and scheduling approaches for dependable pipeline execution and recovery. • Proven track record of technical leadership: mentoring, conducting reviews, and driving engineering best practices.