Data Engineer - AWS Platform

As an AWS Data Engineer focused on our SaaS application, you will play a key role in designing, implementing, and maintaining data pipelines and infrastructure on the AWS platform. You will collaborate closely with cross-functional teams, including software engineers and data scientists, to gather requirements, design data solutions, and ensure the availability of high-quality datasets for machine-learning analysis. The ideal candidate will have a strong background in AWS services and data engineering. Responsibilities : Design, develop, and deploy scalable data pipelines on AWS to support the creation of machine-learning-ready datasets for our SaaS application. Collaborate with software engineers to integrate data pipelines into our SaaS application architecture, ensuring seamless data ingestion and processing. Implement efficient ETL processes to extract, transform, and load data from various sources, including internal databases, third-party APIs, and streaming sources. Optimize data storage and retrieval for performance, scalability, and cost-effectiveness, leveraging AWS services such as S3 Glue, Redshift, Athena, etc. Work closely with data scientists to understand modeling requirements and ensure the availability of clean, curated datasets for training and evaluation. Monitor and troubleshoot data pipelines to ensure reliability, availability, and data quality, implementing automated testing and alerting mechanisms. Implement security best practices for data storage, transmission, and access control, ensuring compliance with relevant regulations and standards. Document design specifications, deployment procedures, and operational processes, facilitating knowledge sharing and collaboration within the team. Requirements : Bachelor's degree in Computer Science, Engineering, or a related field. Proven experience as a Data Engineer or similar role, with at least 4 years of experience. Experience with Test-Driven Development (TDD), writing unit and integration tests to ensure data quality, reliability, and maintainability. Strong proficiency in AWS services such as S3 Glue, Redshift, Athena, EMR, Lambda, etc. Hands-on experience with data modeling, ETL development, and SQL. Excellent problem-solving skills and attention to detail. Strong communication and problem-solving skills, with the ability to collaborate effectively in an agile, fast-paced environment. Preferred Qualifications : AWS certification (e. g., AWS Certified Data Analytics - Specialty, AWS Certified Solutions Architect Associate). Experience with programming languages such as Python, Scala, or Java. Experience with containerization and orchestration tools such as Docker and Kubernetes. Familiarity with DevOps practices and tools for CI/CD. Familiarity with machine learning concepts and workflows, including data preprocessing, feature engineering, and model evaluation. Experience with real-time data processing and streaming technologies. Solid knowledge of trunk-based development practices, including frequent integration with the main codebase, feature toggles, and continuous delivery workflows. (ref:hirist.tech)

Location: mumbai, IN

Posted Date: 4/18/2025