Big Data Engineer

Description: Responsibilities:

• Development and Maintain Data Pipelines: Design, implement, and optimize end-to-end ETL/ELT pipelines for ingesting, processing, and transforming large volumes of structured and unstructured data.
• Utilize Python and Pyspark: Write efficient, scalable and maintainable code in Python and leverage Pyspark for large-scale data processing in distributed computing environments. Also be able to review existing code and identify areas of improvement.
• Ensure Data Quality and Integrity: Implement data validation, cleansing, transformation and reconciliation processes to ensure data accuracy and consistency throughout the data lifecycle.
• Collaborate with Stakeholders: Work closely with IT teams and business stakeholders to gather data requirements and translate them to technical solutions.
• Troubleshoot and Optimize: Monitor job performance, troubleshoot complex data issues and fine-tune for performance and scalability.
• Adhere to Best Practices: Participate in code reviews, establish coding standards, and implement CI/CD pipelines for automated testing and deployment. Skills:
• Strong hands-on coding proficiency in Python, Pyspark and SQL (Microsoft SQL Server preferred)
• Experience with big data frameworks (Hadoop, Spark).
• Experience with cloud platforms ( AWS, Azure or GCP)
• Experience with Code versioning tools ( Bitbucket, Github )
• Experience with CI/CD and setting up pipelines.
• Solid understanding of database design principles, data modelling, schemas and data warehousing solutions.
• Excellent problem-solving and analytical skills to troubleshoot complex data issues independently.