Job description
Principal Databricks Data Engineer Experience Required: 12–18 Years Key Requirements
- 12–18 years of overall Data Engineering experience
- 8+ years of experience with enterprise Data Warehouse and Data Lake platforms
- 5+ years of hands-on experience with Databricks and Apache Spark at scale
- Strong experience modernizing legacy Cloudera platforms including:
- CDH/CDP
- Hive
- HBase
- Impala
- Spark
- Modernize Cloudera platforms to Databricks Lakehouse architecture
- Redesign ingestion, transformation, and consumption patterns from HDFS-based architecture to Cloud Object Storage and Delta Lake
- Refactor legacy Hive and Impala logic into Py. Spark and Spark SQL ELT pipelines
- Ensure data reconciliation, audit integrity, and consistency during migration
- Design and govern enterprise Data Warehouse and Data Lake/Lakehouse architectures
- Implement layered data architecture including:
- Raw / Landing Layer
- Curated / Conformed Layer
- Semantic / Consumption Layer
- Modernize traditional Enterprise Data Warehouse platforms into scalable Lakehouse architectures
- Strong experience with finance and risk data models including:
- General Ledger
- Sub-ledger
- Financial Hierarchies
- Credit Risk
- Liquidity Risk
- Market Risk
- Enable reporting capabilities including:
- Aggregation
- Drill-down
- Drill-back
- Build and manage semantic and consumption layers for BI, reporting, and analytics
- Define business metrics, dimensions, hierarchies, and KPIs
- Experience with:
- Databricks SQL
- Delta Tables
- dbt or similar frameworks
- Develop and optimize large-scale data pipelines using:
- Py. Spark
- Spark SQL
- Delta Lake
- Implement Medallion Architecture including:
- Bronze Layer
- Silver Layer
- Gold Layer
- Optimize workloads using:
- Z-ORDER
- OPTIMIZE
- Caching
- Cluster Configuration Tuning
- Implement:
- Data Governance
- Data Quality Frameworks
- Reconciliation Controls
- Exception Handling
- Establish data lineage and metadata management
- Ensure data security, access control, and compliance standards
- Experience with AWS or Azure cloud platforms
- Experience with CI/CD pipelines using:
- Git
- Terraform
- Jenkins
- Azure DevOps
- Familiarity with:
- Apache Airflow
- Databricks Workflows
- Experience with dbt is an advantage
- Act as a technical authority and lead enterprise architecture decisions
- Mentor senior engineers and establish engineering standards
- Collaborate with finance, risk, analytics, and governance stakeholders
- Translate complex data structures into business-ready insights Nice-to-Have Skills
- Experience in BFSI, Capital Markets, or Regulatory Reporting
- Exposure to:
- SAP Finance
- Oracle Financials
- SAP S/4HANA
- Experience supporting AI/ML workloads
- Databricks and Cloud Certifications Key Responsibilities
- Lead Cloudera to Databricks transformation initiatives
- Shape enterprise finance and risk data platforms
- Support regulatory, management, and analytical reporting systems Essential Skills
- Databricks
- Apache Spark
- Py. Spark
- Spark SQL
- Delta Lake
- Lakehouse Architecture
- Cloudera (CDH/CDP)
- Hive
- HBase
- Impala
- Data Warehouse & Data Lake
- Medallion Architecture
- Databricks SQL
- dbt
- AWS / Azure
- Airflow / Databricks Workflows
- CI/CD
- Terraform
- Jenkins
- Git
- Finance & Risk Data Models
- Enterprise Data Architecture