AWS Data Engineer - ETL

Job Summary : We are seeking a highly skilled and experienced AWS Data Engineer to join our data team. The AWS Data Engineer will be responsible for designing, developing, and optimizing data pipelines, integrating various data sources, and implementing AWS cloud solutions. The ideal candidate should have a strong background in data engineering, cloud platforms (specifically AWS), and big data technologies. Key Responsibilities : Data Pipeline Development : - Design, build, and maintain scalable ETL pipelines using AWS services (e.g., Glue, Lambda, S3, Redshift, etc.). - Integrate data from multiple heterogeneous data sources into a data warehouse or data lake. - Optimize existing data pipelines for performance, reliability, and cost-effectiveness. Cloud Infrastructure : - Manage and configure AWS services such as EC2, RDS, S3, Glue, Athena, Lambda, Redshift, and EMR. - Automate infrastructure provisioning using tools like CloudFormation or Terraform. - Ensure data security and compliance with AWS best practices (IAM roles, encryption, VPC). Data Warehousing and Storage : - Design and implement scalable data warehousing solutions using Amazon Redshift, Snowflake, or other technologies. - Manage data storage in Amazon S3, ensuring efficient and cost-effective storage architectures (partitioning, compression). - Implement data versioning and backup strategies. Data Modeling and Optimization : - Perform data modeling and schema design to support data warehouses and reporting systems. - Optimize SQL queries and database performance. - Implement data governance and quality control procedures (e.g., monitoring, alerting). Collaboration : - Work closely with data scientists, analysts, and stakeholders to understand data needs. - Collaborate with DevOps and engineering teams to ensure the smooth operation of data infrastructure. Monitoring & Reporting : - Set up monitoring for ETL pipelines and cloud infrastructure (CloudWatch, X-Ray, etc.). - Provide regular reports on data pipeline performance and optimize for continuous improvement. Required Qualifications : - Bachelor's degree in Computer Science, Engineering, or a related field. - 3 years of experience in data engineering roles, with significant exposure to AWS services. - Strong proficiency in AWS services such as Glue, Redshift, S3, Lambda, and RDS. - Experience with big data tools such as Spark, Hadoop, or EMR. - Hands-on experience with ETL tools and techniques. - Proficiency in SQL and experience in data modeling and database performance tuning. - Knowledge of scripting languages such as Python or Scala. - Experience with CI/CD pipelines and infrastructure as code tools like Terraform or CloudFormation. - Familiarity with data formats such as Parquet, Avro, and JSON. - Experience in data warehousing solutions (Redshift, Snowflake, etc.). Preferred Qualifications : - AWS certifications like AWS Certified Data Analytics - Specialty or AWS Certified Solutions Architect. - Knowledge of NoSQL databases like DynamoDB or MongoDB. - Experience with DevOps and container orchestration tools (e.g., Docker, Kubernetes). - Familiarity with machine learning or data science tools on AWS (e.g., SageMaker). - Experience in real-time data processing (e.g., using Kinesis or Kafka). Soft Skills : - Strong problem-solving skills and ability to work in a fast-paced environment. - Excellent communication and teamwork skills. - Ability to multitask and manage multiple priorities effectively. (ref:hirist.tech)

Location: in, IN

Posted Date: 11/24/2024