AU Small Finance Bank - Spark Engineer - Big Data

Job Description : 1. Design and Develop Spark Applications : - Understand the current Apache Spark platform scala code and nuances related to the same - Optimize and fine-tune Spark jobs for performance and efficiency. 2. Python Programming : - Utilize Python for scripting, data manipulation, and integration with Spark applications. - Collaborate with cross-functional teams to integrate Python-based solutions into Spark workflows. 3. Big Data Processing : - Work with large-scale distributed systems for processing and analyzing big data. - Implement best practices for data processing, storage, and retrieval. 4. Performance Optimization : - Identify bottlenecks and optimize Spark applications for enhanced performance. - Implement caching and partitioning strategies to improve data processing speed. 5. Collaboration and Communication : - Collaborate with data scientists, analysts, and other engineering teams to understand business requirements and translate them into technical solutions. - Communicate effectively with team members and stakeholders. 6. Monitoring and Troubleshooting : - Implement monitoring solutions for Spark jobs and proactively address any issues. - Troubleshoot and resolve performance and operational issues. 7. Documentation : - Maintain comprehensive documentation for Spark applications, workflows, and processes. - Provide training and knowledge transfer to junior team : - Bachelor's or Master's degree in Computer Science, Engineering, or a related field. - 5 to 8 years of experience working with Apache Spark, Scala. - Familiarity with AWS Cloud platform - Strong proficiency in Python programming. - Experience with big data processing frameworks and distributed computing. - Solid understanding of data structures, algorithms, and data modeling. - Excellent problem-solving and analytical skills. - Effective communication and collaboration skills (ref:hirist.tech)

Location: jaipur, IN

Posted Date: 2/5/2025