Spark/Scala Developer - HDFS

Spark Scala Developer Shift : EUR Experience : 10 years Responsibilities : - Develop and optimize Scala/Spark jobs for data transformation, aggregation, and processing. - Design and implement data processing pipelines to efficiently handle large-scale data. - Produce unit tests for Spark transformations and related helper methods. - Write comprehensive Scaladoc-style documentation for all developed code. - Ensure code quality and adherence to best practices by producing unit tests and leveraging test-driven development (TDD). - Work on Spark query tuning and performance optimization to improve the efficiency of the data processing systems. Skills & Qualifications : - Strong proficiency in Scala, with an emphasis on the functional programming paradigm. - Experience with Scalatest, JUnit, Mockito, and Embedded Cassandra for testing. - Hands-on experience with Apache Spark 2.x and its components, including : - RDD API - SQL DataFrame API - MLlib API - GraphX API - Streaming API - In-depth experience in Spark query tuning and performance optimization. - Solid understanding of Data Modeling and Data Warehousing. - Strong SQL knowledge and experience with various SQL database integrations (Microsoft SQL Server, Oracle, Postgres, and/or MySQL). - Proficient with distributed storage systems like HDFS, S3, Cassandra, and/or DynamoDB. - Deep understanding of distributed systems concepts such as CAP theorem, partitioning, replication, consistency, and consensus. - Ability to work effectively in an agile development environment and collaborate with cross-functional teams. Preferred Experience : - Experience in designing scalable and fault-tolerant systems using Apache Spark and related technologies. - Familiarity with cloud-based technologies and deployment strategies (AWS, Azure, GCP) (ref:hirist.tech)

Location: in, IN

Posted Date: 2/6/2025