Job description
Role: We are seeking a highly skilled and motivated Data Engineer with expertise in Python or Py. Spark, and it's good to have experience with DBT (Data Build Tool), to join our team. As a Data Engineer, you will play a crucial role in designing, building, and maintaining our data infrastructure in the cloud. You will collaborate with cross-functional teams to ensure data is collected, processed, and made available for analysis and reporting.
If you are passionate about data engineering, cloud technologies, and have strong programming skills, we invite you to apply for this exciting position.
Requirements
Design, develop, and maintain scalable data pipelines that collect, process, and transform data from various sources into usable formats for analysis and reporting. Cloud Integration: Leverage cloud platforms such as AWS, Azure, or Google Cloud to build and optimize data solutions, ensuring efficient data storage, access, and security.
Python/Py. Spark Expertise: Utilize Python and/or Py. Spark for data transformation, manipulation, and ETL processes. Write clean, efficient, and maintainable code. Data Modeling: Create and maintain data models that align with business requirements, ensuring data accuracy, consistency, and reliability. Data Quality: Implement data quality checks and validation processes to ensure the integrity of the data, troubleshooting and resolving issues as they arise.
Performance Optimization: Identify and implement performance optimizations in data pipelines and queries to ensure fast and efficient data processing. Collaboration: Collaborate with data scientists, analysts, and other stakeholders to understand their data requirements and provide them with reliable data sets. Documentation: Maintain thorough documentation of data pipelines, workflows, and processes to ensure knowledge sharing and team efficiency.
Security and Compliance: Implement security best practices and ensure data compliance with relevant regulations and company policies. Good to Have (Preferred Skills) DBT (Data Build Tool): Experience with DBT for managing and orchestrating data transformations. Containerization and Orchestration: Experience with containerization and orchestration tools (e.
g., Docker, Kubernetes). Data Streaming: Knowledge of data streaming technologies (e.g., Kafka, Apache Spark Streaming). Workflow Management: Familiarity with data orchestration and workflow management tools (e.g., Apache Airflow). Cloud Certification: Certification in cloud services (e.g., AWS Certified Data Analytics, Azure Data Engineer).
Data Governance: Understanding of data governance and data cataloging