Technical Lead

Interaction with business users/stakeholders and requirement gathering, Involved Functional Requirement Specifications from customer and understand the same. Good understanding and real time architecture experience on Hadoop ecosystem (HDFS, Map Reduce, YARN, Hive, Sqoop, Spark) and Azure cloud (ADLS, ADFS, ADF) Should have real time experience by using various file formats (csv, parquet, text file, avro, sequential file) Should have Hands-on experience with hive internal, external tables, hive join techniques, partitions, bucketing and hive optimization techniques Should have hands-on experience in developing Spark applications using Data frame API, Spark-SQL in Data bricks for data extraction, transformation, and aggregation from multiple file formats for Analyzing & transforming the data using Scala and Python to uncover insights into the customer usage patterns.

Should have hands-on experience Extract Transform and Load data from sources Systems (On-prem) to Azure Data Storage services using a combination of Azure Data factory, T-SQL, Spark SQL. Should have hands-on Experience metadata driven ingestion pipeline framework by using Azure Data Factory Should have real time experience on implementing spark optimization techniques and adaptive query execution configuration in data bricks Should have good analytics and writing SQL queries to consume data and generate reports using Power BI Added advantage bash script experience with text file processing skills using Regex, Sed, Awk, Grep, curl, etc.

Cloudera Manager

• Monitor jobs and query performance. customizable dashboard, with the ability to create advanced charts for historical monitoring.
• Query on Impala nodes to find all the error (Failed queries, Missing Stats, out of memory exception)
• Monitoring entire cluster and find memory utilization based on resource level. Workload XM - monitor the workload, memory, resource utilization, slow job and failed job, queries in spark jobs and impala queries Involved in conducting the code review and Results with the Developers Have good exposure to Healthcare domain