Data Engineer - Spark/Hadoop

About the job : Overview : The Data Engineer plays a crucial role in the organization as they are responsible for designing, developing, and maintaining the company's data architecture and infrastructure. They work closely with data scientists and analysts to ensure data is properly collected, stored, and accessible for analysis and reporting. Key Responsibilities : - Design and implement data pipelines and ETL processes. - Develop and maintain databases, data warehouses, and data lakes. - Collaborate with data scientists to translate algorithms into production systems. - Create and maintain optimal data pipeline architecture. - Identify, design, and implement internal process improvements. - Build the infrastructure required for optimal extraction, transformation, and loading of data from a wide variety of data sources. - Work with stakeholders including the executive, analytics, and data engineering teams to assist with data-related technical issues. - Design and implement data models and data architecture. - Develop and optimize SQL queries. - Monitor and maintain data quality, integrity, and security. - Implement and maintain data governance and privacy measures. - Guide and mentor junior data engineering team members. - Keep up-to-date with the latest technologies and best practices in data engineering. - Collaborate with cross-functional teams to achieve business goals. - Document and communicate technical solutions and best practices to the team. Required Qualifications : - Bachelor's degree in Computer Science, Information Technology, or related field. - Proven experience as a Data Engineer or similar role. - Hands-on experience with cloud platforms such as AWS, Azure, or Google Cloud. - Proficiency in ETL tools, data modeling, and database design. - Strong programming skills in languages like Python, SQL, or Java. - Experience with big data technologies such as Hadoop, Spark, or Kafka. - Knowledge of data warehousing concepts and technologies. - Experience in data pipeline orchestration and automation. - Strong understanding of data governance and security principles. - Ability to work in a fast-paced, collaborative, and agile environment. - Excellent analytical and problem-solving skills. - Good communication and teamwork skills. - Ability to prioritize and manage multiple tasks simultaneously. - Relevant certifications in data engineering or cloud technologies are a plus. - Experience in implementing and managing machine learning pipelines is desirable. (ref:hirist.tech)

Location: ahmedabad, IN

Posted Date: 12/22/2024