Data Engineer

Data Engineer Location: New York City, NY About the Role A Data Engineer designs, builds, and maintains the systems (data pipelines, databases, warehouses) that collect, process, and store large volumes of data, making it accessible and reliable for data scientists and analysts to derive insights. This role focuses on infrastructure, automation, performance, and security to ensure data quality and availability for business intelligence and machine learning.

Core Responsibilities Data Pipeline Development: Build and maintain ETL/ELT pipelines for data ingestion, transformation, and loading from various sources (APIs, databases). Data Architecture: Design and manage data models, schemas, and infrastructure (data lakes, warehouses like Snowflake, Redshift). Data Quality & Governance: Implement data validation, security (encryption, access control), and compliance measures.

Performance Optimization: Monitor system performance, tune databases, and optimize data processing for efficiency. Collaboration: Work with analysts, scientists, and business stakeholders to understand data needs and deliver solutions. Key Skills & Technologies Programming: Python, SQL Cloud Platforms: AWS, Azure, GCP Big Data Tools: Spark, Kafka, Hadoop Databases/Warehouses: Snowflake, Redshift, Big.

Query, PostgreSQL, MongoDB DevOps/Tools: Docker, Kubernetes, Airflow (for orchestration) What They Do Data engineers are the plumbers of the data world, ensuring clean, structured data flows smoothly from raw sources to analytical tools, enabling data-driven decisions and powering machine learning models. ```