Databricks Engineer

This is a remote position.

Overview

Data Pipeline Development: Design, build, and optimize robust ETL/ELT pipelines using Azure Databricks, Spark, and SQL. Data Processing & Transformation: Utilize Python, Py. Spark, and SQL to clean, transform, and aggregate complex data for analytics. Azure Data Lake Management: Manage and optimize data storage and retrieval in Azure Data Lake Storage (ADLS) Gen2.

Delta Lake Implementation: Implement Delta Lake for ACID compliance, data versioning, and high-performance data lake operations. Integration with Azure Services: Integrate Databricks with Azure Data Factory for orchestration, Azure Synapse Analytics for warehousing, and Azure Key Vault for security. Performance Optimization: Monitor, troubleshoot, and optimize Databricks clusters and spark jobs to manage costs and performance.

Data Security & Governance: Implement Role-Based Access Control (RBAC), data encryption, and data lineage tracking.

Requirements

Strong data engineering skills on Azure, ADF, Synapse and Databricks Strong experience of Python/Pysprak, SQL Benefits Diversity Inclusion: Exavalu also promotes flexibility, adapting to the needs of employees, customers, and the business. It might be part-time work, working outside normal 9-5 business hours or working remotely.

We also offer a welcome back program to help individuals return to the mainstream after a prolonged absence due to health or family reasons. At Exavalu, we are committed to building a diverse and inclusive workforce. We welcome applications for employment from all qualified candidates, regardless of race, colour, gender, national or ethnic origin, age, disability, religion, sexual orientation, gender identity or any other status protected by applicable law.

We nurture a culture that embraces all individuals and promotes diverse perspectives, where you can make an impact and grow your career