Data Engineer - Databricks

Job Description – Data Engineer (AWS + Databricks) Experience: 5+ Years Location: Flexible / Hybrid Key Responsibilities Build scalable ETL/ELT pipelines using Databricks (PySpark, Spark SQL, Delta Lake). Design and implement Medallion Architecture (Bronze, Silver, Gold) for Lakehouse. Develop batch and real-time pipelines using Spark Streaming, AWS Kinesis, or Glue Streaming.

Manage data ingestion from diverse sources – RDBMS, APIs, S3, Kafka. Perform data profiling, validation, and quality checks. Optimize Spark jobs, partitions, and cluster configurations for performance. Implement and manage data infrastructure using AWS S3, Glue, Athena, Lambda, Step Functions, Redshift, EMR. Set up CI/CD pipelines (GitHub Actions, CodePipeline, Jenkins).

Collaborate with analysts and data scientists to deliver analytical datasets. Technical Skills Databricks: PySpark, Spark SQL, Delta Lake, Unity Catalog, Job Workflows. AWS: S3, Glue, Athena, Kinesis, Lambda, Step Functions, EMR, Redshift, CloudWatch. Programming: Python (pandas), SQL (window functions, optimization).

IaC & DevOps: Terraform / CloudFormation, Git, Jenkins, GitHub Actions. Performance Tuning: Spark optimizations, Z-ordering, partitioning, caching.