Data Engineer - Big Data

Data Engineer Job Title : Data Engineer Introduction : Join our forward-thinking banking client as a Data Engineer, where you will significantly contribute to shaping our data solutions that drive customer value. This role is for those who thrive in a dynamic environment and are passionate about leveraging technology to solve real-world problems in the financial sector. What You'll Do : - Develop and optimize data architectures by performing data extraction, transformation, and loading to support scalable operations. - Utilize advanced PySpark 2and Spark SQL2 programming to manipulate and manage data effectively. - Implement and maintain real-time data processing systems using Kafka and other Big Data technologies within the Spark Framework. - Collaborate with technology and architecture teams to enhance data solutions and advocate for necessary changes to support product development. - Utilize SQL extensively to ensure data integrity and optimize queries for performance. - Manage NoSQL databases such as MongoDB, DynamoDB, Cassandra, or Elastic Search to support specific use cases. - Engage with cloud services (AWS, Azure, GCP) to implement scalable and efficient data processing solutions. - Apply CI/CD and DevOps practices to improve and streamline deployment processes. Technical Skills Required : - Expertise in PySpark, Spark SQL, and the Spark Framework. - Proficient in real-time big data processing using Kafka. - Strong command of SQL for complex query writing and data manipulation. - Experience with NoSQL databases like MongoDB, DynamoDB, Cassandra, or Elastic Search. - Familiarity with cloud platforms such as AWS, Azure, or GCP, particularly with services like EMR, Databricks, DynamoDB, or CosmosDB. - Understanding of CI/CD pipelines and DevOps practices. - Knowledge of the banking sector is beneficial but not essential. Professional Skills : - A degree in Computer Science, Software Engineering, or a related field. - Demonstrated ability to understand data usage and dependencies across different teams. - Proven experience in designing and deploying large-scale data extraction features. - Excellent problem-solving skills and the ability to think critically. (ref:hirist.tech)

Location: in, IN

Posted Date: 2/21/2025