Job description
Job Title: Py. Spark Data Engineer Experience: 4+ Years Location: Hyderabad Job Summary: We are seeking a Senior Spark Engineer to design and implement high-performance Spark execution patterns inside x. Flows, supporting batch and streaming pipelines with built-in data quality, observability, and governance.
Requirements
Key Responsibilities Design and implement Spark-based execution frameworks for x. Flows pipelines. Build reusable Spark components for: Readers (JDBC, Files, Kafka, CDC) Transformers (Join, Filter, Aggregate, Window, Union) Writers (Iceberg, Delta, Parquet, Snowflake) Optimize Spark performance (partitioning, caching, shuffles, memory).
Implement Data Quality & Reconciliation execution patterns. Handle schema evolution, CDC, watermarking, and checkpoints. Integrate Spark jobs with EMR Serverless / Data bricks / Kubernetes. Publish execution metrics, logs, and lineage for observability. Work closely with platform & UI teams to support no-code execution.
Required Skills
4+ years of experience with Apache Spark Strong expertise in Py. Spark (preferred) or Spark Scala Deep understanding of Spark internals (DAGs, stages, shuffles, caching) Experience with Iceberg / Delta Lake Strong SQL skills Experience with batch and streaming pipeline Benefits Comprehensive Medical Coverage: Health insurance of INR 7.
0 Lakhs for you and your family (up to 6 members), ensuring complete peace of mind. Robust Protection Plans: Group Personal Accident Insurance and Group Term Life Insurance to safeguard you and your loved ones. Retirement Benefits: PF and Gratuity provided as per standard government regulations. Flexible Work Options: Enjoy hybrid work arrangements & flexible working hours Generous Leave Policy: 21 days of annual leave, in addition to 10 company-declared holidays.
Employee Well-being Spaces: Access to a dedicated break-out area with round-the-clock refreshments for relaxation and rejuvenation.