Job description
Key Responsibilities
- Design, develop, and optimize data pipelines using Databricks, Py. Spark, Delta Lake, and related Azure components.
- Work across pipeline development, data modeling, and production code optimization, ensuring scalability and performance.
- Build new pipelines from scratch as well as enhance and maintain existing ones.
- Apply strong understanding of ETL design principles, data modeling concepts, and data governance standards.
- Collaborate within a scrum team, taking ownership of assigned stories while independently delivering high-quality, production-ready code.
- Demonstrate proficiency across Py. Spark, SQL, Delta Lake, Unity Catalog, and Databricks Workflows, with solid understanding of logic and data flow.
- Work in Azure environments, leveraging tools like ADLS, ADF, and Synapse (as applicable).
- Contribute inputs to architecture and design discussions where appropriate, while primarily focusing on hands-on development.
- Troubleshoot data issues, perform root-cause analysis, and optimize performance for existing pipelines.
Requirements
- 2-3 years of experience in data engineering, preferably with Azure Databricks.
- Strong technical expertise in Py. Spark, SQL, and Delta Lake.
- Familiarity with Unity Catalog, data governance, and DevOps practices (Git, CI/CD).
- Ability to work independently as well as collaboratively within an agile delivery team.
- Excellent problem-solving, debugging, and communication skills.