Lead Data Software Engineer with Databricks with Apache Kafka, Apache Spark, Kubernetes

We are looking for a Lead Data Software Engineer to drive the migration of an existing data analytics platform, originally built on Azure resources such as Data Factory, Databricks, Event Hub, and Cassandra, to a cloud-agnostic platform deployable on-premises.

Responsibilities

Implement streaming and batch Spark pipelines running on Kubernetes Lead the implementation of objects within the data generator framework Oversee deployment and testing across local and development environments Drive improvement of the current, actively changing solution Conduct bug investigation and resolution Perform unit testing to ensure solution quality Participate in refinement, planning, and demo sessions Guide and mentor team members throughout the implementation phase Requirements 5+ years of experience with Apache Spark, Confluent/Apache Kafka, and Kubernetes Proficiency in Python Familiarity with the data engineering domain and cloud-agnostic platform migrations Capability to dive deeper into new technologies in a short time during active implementation phases Ability to work independently with syncs with a team lead Strong communication skills and proactiveness Showcase of being a hands-on team player and technical leader Proficiency in English at a B2+ level Nice to have Knowledge of Java Familiarity with Kafka Connect, KSQL, and Kafka Streams Skills in Ansible and Argo CD Understanding of TDD