Senior Data Engineer - ETL/Spark

Job Description : Responsibilities : - Ability to design and architect complex data lake on the cloud - Build scalable data pipeline and orchestration leveraging state of the art big data technologies - Work with data to solve business problems, building and maintaining the infrastructure to answer questions and improve processes - Work closely with the data science and decision science teams to develop data stores, orchestration workflows and pipelines for research, reporting, and machine learning - Be an advocate for best practices and continued learning - Model front end and back end data sources to help draw a more comprehensive picture of user flows throughout our system and enable powerful data analysis - Build data pipelines that clean, transform, and aggregate data from disparate sources - Build and maintain MLOps platform to deploy models that can be used to make predictions and answer questions for the overall business What does it take to be in this role : - Strong Big data engineering (Batch & Streaming) experience in cloud leveraging tech like EMR, Spark, Spark streaming - 5 or more years of experience with Big data and data lake technology stack like EMR , Spark and SQL - Familiarity with the AWS ecosystem specifically EMR, Athena, messaging brokers and RDS - Experience building scalable data pipelines or ETL processes - Experience building RESTful APIs, architecting complex systems with batch and real-time workloads, monitoring and instrumenting production systems (ref:hirist.tech)

Location: mumbai, IN

Posted Date: 12/22/2024