Lead Data Software Engineer (Databricks)

We are searching for a Lead Data Software Engineer to join our team and oversee the development of our cutting-edge supply-chain data analytics platform. This platform delivers a comprehensive view of suppliers, products, material categories, shipments, and compliance activities while employing artificial intelligence to generate insights and recommendations tailored to our clients’ needs.

If this role excites you – apply!

Responsibilities

Design and develop data architectures and ETL pipelines utilizing Databricks and external orchestrators such as Airflow Collaborate with machine learning teams to incorporate AI-driven insights into the platform Develop system-level improvements and components for data engineering workflows Optimize the performance and scalability of data-intensive applications Adhere to software engineering standards, including containerization, unit testing, linting, and code style reviews Maintain and improve data workflow, Delta Lake, and Delta Live Tables within Databricks Provide technical mentorship and leadership to the data engineering team Engage with clients and stakeholders to understand their requirements and offer customized solutions Tackle challenges independently and ensure successful project delivery Requirements Experience leading data engineering efforts on projects utilizing Databricks Solid understanding of data architectures and skills in data modeling Background in designing and building ETL pipelines in Databricks with external orchestrators such as Airflow Hands-on expertise in Databricks (Delta Lake, workflows, Delta Live Tables, deployment, and versioning) Proficiency in Python and cloud-native technologies Competency in Spark/Py.

Spark Engineering experience with either AWS or Azure Knowledge of big data and techniques for optimizing data-intensive applications Strong communication skills with a proactive attitude and client-facing experience Ability to navigate ambiguity and complete projects independently Comfort working in a dynamic and transparent startup environment Fluent English communication skills at a C1 level Nice to have Proficiency in configuring and managing CI/CD pipelines in Azure DevOps Familiarity with Data Observability and Data Quality Monitoring practices Capability to integrate data quality checks into data pipelines