AWS Data Engineer - Python/Apache Airflow

JD : AWS Data Engineer Job location : Hyderabad Must Have : - Design and develop efficient data pipelines using Python and orchestrate workflows with Airflow, ensuring reliable data ingestion, transformation, and storage. - Utilize Spark (Glue, EMR) for largescale data processing tasks, optimizing for performance and scalability. - Implement and manage streaming data solutions with Kafka, managing high-volume data with proficiency in SQS and Event Bridge for even- driven architectures. - Leverage AWS services extensively, including S3 for data storage, ensuring best practices in cloud-based data handling and security. - Work closely with data architects, analysts, and other stakeholders, providing technical expertise and support in data-related projects and initiatives. Good to Have : - Knowledge of Data Warehousing (DW) concepts and Data Integration (DI) methodologies. - Experience with data modelling and familiarity with ETL processes and tools. - Experience in API integration, with a solid understanding of RESTful services. - Developing and consuming APIs to facilitate seamless data exchange and system : - Bachelor's degree in computer science, Engineering, Data Science, or related field. - 4-8 years of relevant experience in data engineering, with a proven track record of developing and managing data solutions. - Strong proficiency in Python and experience with Airflow for data pipeline management. - Expertise in Spark for big data processing, with experience in Glue and/or EMR. - Demonstrated experience with Kafka for real-time data streaming and familiarity with AWS messaging services like SQS and Event Bridge. - In-depth knowledge of AWS cloud services, especially S3, and best practices in cloud security and data management. (ref:hirist.tech)

Location: hyderabad, IN

Posted Date: 11/21/2024