Lead I - Data Engineering

Design and develop scalable data pipelines using Azure Data Factory (ADF). Develop data processing solutions using Azure Databricks and PySpark. Build and maintain data warehouses and data lakes using Azure services. Develop ETL/ELT processes for structured and unstructured data. Write complex SQL queries, stored procedures, and performance-optimized data transformations.

Implement data ingestion from databases, APIs, files, and cloud-based sources. Work with Azure Data Lake Storage Gen2 and cloud storage solutions. Design and implement data models for analytical and reporting requirements. Monitor, troubleshoot, and optimize data pipelines and workflows. Implement data quality, security, governance, and access-control practices.

Collaborate with data scientists, BI developers, software engineers, and business stakeholders. Participate in code reviews, testing, deployment, and production support.

Required Skills

Strong experience with Azure Data Factory (ADF). Hands-on experience with Azure Databricks, PySpark, and Python. Strong knowledge of SQL and relational databases such as SQL Server, PostgreSQL, or Oracle. Experience with Azure Data Lake Storage Gen2. Knowledge of data warehousing concepts, dimensional modeling, and ETL/ELT processes.

Experience working with large datasets and distributed data processing. Familiarity with Git and CI/CD pipelines. Understanding of Azure security, identity, and access management concepts. Good understanding of data engineering best practices and SDLC.

Preferred Skills

Experience with Azure Synapse Analytics. Knowledge of Microsoft Fabric. Experience with Azure Key Vault and Azure Monitor. Knowledge of Kafka/Event Hubs and real-time data processing. Experience with Power BI and BI/reporting environments. Familiarity with Terraform or other Infrastructure-as-Code tools. Azure certifications such as Azure Data Engineer Associate (DP-203) are a plu