Big Data Principal

Senior / Lead Data Engineer Engineering –Enterprise Data Platform Level Lead/Principal Data Engineer / Technical Lead– hands-on Experience 12+ years in data engineering (Senior); 5+ years including team leadership (Lead) Reports to Enterprise Data Architect Domain Financial Services / Banking (strongly preferred) Location Hybrid – to be confirmed Engagement Permanent / Long-term Contract About the Role We are looking for a Senior / Lead Data Engineer to join the engineering organisation a multi-year banking transformation programme delivering capabilities across Accounts (Real-Time Ledger), Payments, Foreign Exchange and Client Lifecycle Management, underpinned by an Enterprise Data Platform built on Snowflake.

This is a hands-on build role. Working closely with the Enterprise Data Architect, you will turn the target-state data architecture into a production platform: standing up the platform foundations, engineering data acquisition and transformation, and implementing canonical data models and data products. You will act as the engineering counterpart to the Data Architect – implementing the design, feeding back on buildability, and setting the engineering standards others follow.

At Lead level, you will also shape and lead the data engineering team.

Key Responsibilities

Platform foundations Build Snowflake environment structures – layered databases and schemas (Raw / Conformed / Consumption), environment separation (development / test / production) and naming conventions Implement role-based access control, service accounts, virtual warehouse design, resource monitors and cost guardrails Manage the platform as code – Snowflake objects, roles, grants and warehouses through infrastructure-as-code and version control Establish Git workflows and CI/CD pipelines for data code, with automated build, test and deployment across environments Data acquisition and CDC Assess source systems and recommend fit-for-purpose acquisition patterns – full vs incremental, batch vs micro-batch vs event, and file vs API vs direct database vs message hub Advise on where change data capture should be applied and owned – log-based CDC at source, application-emitted change events, or warehouse-side change detection – and implement the chosen patterns Engineer CDC and replication from relational, document and time-series application databases, handling initial loads, deletes, schema evolution, lag and backfills Ingest event streams from messaging platforms into Snowflake, covering ordering, idempotency, deduplication, replay and error handling Configure and optimise managed ELT connectors (e.

g. Fivetran), and build file- and API-based ingestion where required Design and build a metadata-driven ingestion framework – sources, objects, load types (full, incremental, CDC), schedules and target mappings held as configuration, with control tables for watermarks, run status, audit and reprocessing – so new feeds are onboarded through metadata rather than new code Make the framework reusable across ingestion channels (managed ELT, file, API, database CDC and event feeds), with generic, parameterised loading, schema-drift handling and standard error handling Prove patterns through technical spikes, and record decisions as Architecture Decision Records Transformation and modelling Design and build dbt projects from the ground up – structure, layering, conventions, macros, packages, tests and documentation Run dbt both natively in Snowflake (dbt Projects on Snowflake) and on dbt Cloud, building portable, convention-led projects that move between the two with minimal rework Implement canonical data models – including industry-aligned models such as BIAN-based business objects – from source-to-canonical mapping specifications and governed reference data Build dimensional and domain-oriented data products for reporting, regulatory, analytics and AI consumers Engineer identity resolution, key management and cross-source reconciliation, and feed implementation findings back into the models and mappings Quality, governance and operations Embed data quality as code – tests, freshness checks, reconciliations and data contract enforcement Implement Snowflake governance controls – tagging and classification, dynamic data masking, row access policies and PII handling aligned to GDPR – with lineage that supports BCBS 239 Build monitoring and alerting for pipeline failures, freshness, data quality and cost, and define run-books and support processes Orchestration and scheduling Design and implement orchestration and scheduling natively in Snowflake using Tasks and task graphs (DAGs) – dependencies, root and finaliser tasks, serverless vs warehouse-backed compute, schedules and event-triggered runs Orchestrate ingestion, dbt execution (e.

g. EXECUTE DBT PROJECT), data quality checks and downstream refreshes end to end within Snowflake, with retries, failure handling, alerting and notifications Use Dynamic Tables where declarative, target-lag-driven pipelines are a better fit than task-based scheduling, choosing between incremental and full refresh modes and managing their cost Build operational visibility over task and Dynamic Table runs – run history, latency, failures and credit consumption – and keep orchestration designs portable to external orchestrators where needed AI-assisted engineering Use Snowflake Co.

Co (formerly Cortex Code) to accelerate development, with generated code reviewed, tested and deployed through standard CI/CD Apply Snowflake Cortex AI capabilities – AISQL functions, Cortex Search, semantic views, Cortex Analyst and Cortex Agents – where they add value, and design data products to be AI-ready Leadership and collaboration Work as the engineering counterpart to the Enterprise Data Architect, contributing to design decisions and ADRs Collaborate with Integration, Solution and Application teams on source extracts, event contracts and data contracts Define and own engineering standards, templates and reusable patterns Lead level: lead, mentor and help recruit data engineers, own code review, and plan and report delivery Primary Skills (Essential) Candidates are expected to have strong, hands-on depth in all of the following: Skill area What we expect SQL Advanced, production-grade SQL – complex joins, window functions, set-based transformation, MERGE / upsert patterns, query profiling and performance tuning Data Warehousing Solid grounding in warehouse principles – layered / medallion architectures, ELT, incremental processing, slowly changing dimensions, history and audit, reconciliation, partitioning and performance Data Modelling Dimensional modelling (Kimball – facts, dimensions, conformed dimensions); implementing canonical / enterprise data models; normalised vs denormalised trade-offs; SCD Types 1/2; working from logical models and mapping specifications Snowflake Hands-on platform engineering – schema and layer design, RBAC, virtual warehouses, Snowpipe, Streams & Tasks, zero-copy cloning, Time Travel, masking and row access policies, resource monitors, cost optimisation dbt Building dbt projects from scratch – layering, incremental models, snapshots, macros, packages, tests, docs; dbt Projects on Snowflake and dbt Cloud Data Acquisition & CDC Designing full, incremental and CDC-based acquisition; log-based CDC tooling (e.

g. Fivetran / HVR, Debezium, Kafka Connect, Snowflake Openflow); batch, micro-batch and event-driven ingestion Managed ELT Fivetran or comparable – connector configuration, schema drift, history mode, re-syncs, consumption cost management Metadata-driven Ingestion Designing and building configuration-led ingestion frameworks – metadata / control tables, parameterised and reusable load patterns, watermarking, audit, restartability and reprocessing Snowflake Orchestration Snowflake Tasks and task graphs (DAGs), Streams & Tasks patterns, Dynamic Tables (target lag, incremental vs full refresh), serverless tasks, alerts and notifications, monitoring of task history and cost Engineering Practice Python for data engineering and automation; Git; CI/CD (e.

g. GitHub Actions,); infrastructure-as-code (e.g. Terraform, schemachange); automated testing Snowflake AI Good working knowledge of Snowflake Co. Co (formerly Cortex Code) and Snowflake Cortex AI, and a disciplined approach to AI-assisted development in a regulated environment Secondary Skills (Desirable) Experience in any of the following will strengthen an application: Skill area What we value Cloud (AWS) S3, IAM, KMS, Private.

Link and networking basics, secrets management Application Databases Sourcing safely from PostgreSQL, Oracle, MongoDB, TimescaleDB and Redis – logical replication, redo logs, change streams, impact on source performance Event Streaming NATS Jet. Stream or Kafka; Snowpipe Streaming; event schema contracts and replay strategies Advanced Snowflake Snowpark (Python), Snowflake Horizon catalog, data sharing Modelling Standards BIAN, ISO 20022 or other industry reference models; awareness Operations Data observability tooling, Fin.

Ops for Snowflake and ELT consumption, external orchestration tools (e.g. Airflow, Dagster) Banking Domain Core banking / ledger, payments, FX, client onboarding / KYC or regulatory reporting data; GDPR and BCBS 239 Certifications Snow. Pro Core or Snow. Pro Advanced: Data Engineer; dbt Analytics Engineering; AWS certification Experience & Profile Senior: 10+ years in data engineering, including at least 3 years of hands-on Snowflake delivery in production.

Lead: 12+ years, including leading a data engineering team or workstream Strong track record in data warehousing and data modelling, with experience implementing models designed by architects as well as designing their own Has stood up a new data platform or major platform workstream – environments, access control, automation – rather than only extending an existing one Has built a dbt project from the ground up, including standards, testing and CI/CD Has designed and built a metadata-driven ingestion framework that has been used to onboard multiple sources in production Strong hands-on experience scheduling and orchestrating production pipelines natively in Snowflake with Tasks / task graphs and Dynamic Tables Has designed and implemented data acquisition across heterogeneous sources, and can clearly explain the trade-offs between full, incremental and CDC approaches Comfortable reading and implementing architecture artefacts – canonical models, mapping specifications, ADRs – and offering well-reasoned implementation feedback Clear communicator, able to work with architects, application teams and business stakeholders Senior vs Lead – Role Distinction Aspect Senior Data Engineer Lead Data Engineer Focus Hands-on build of pipelines, models and platform components Hands-on build plus technical direction of the engineering workstream Standards Applies and contributes to engineering standards Defines and owns engineering standards and reusable patterns People Peer reviews and supports other engineers Leads, mentors and helps recruit the team; owns code review Planning Estimates and delivers own work Plans, estimates and reports delivery Architecture Implements the design; raises issues Engineering counterpart to the Data Architect; co-authors ADRs