Data Engineer - Scala/Python

Job Description : - Develop ETL (Extract, Transform, Load) processes to move and transform data between systems. - Integrate data from various sources, ensuring consistency and accuracy. - Monitor system performance and identify areas for optimization. - Align architecture with business requirements and provide solutions which fit best to solve the business problems. - Build the infrastructure required for optimal ETL pipelines from a wide variety of data sources using SQL and AWS - big data- technologies. - Evaluate and recommend new technologies to enhance data processing capabilities. - Work closely with data scientists, analysts, and other stakeholders to understand their data needs. - Use programming language( python, spark) and tools efficiently to collate the data - Identify ways to improve data reliability, efficiency and quality. - Use data to discover tasks that can be : - Can Develop ETL (Extract, Transform, Load) processes to move and transform data between systems. - Have good hands-on experience in any coding language Python or Scala. - Good understanding of Data Engineering concepts such as data warehousing, datalakes, data modeling, data quality assurance, data lifecycle management, streaming data, metadata management etc. - Strong hold on cloud-based ecosystems(AWS preferred). - Have an understanding of data pipeline orchestration tools such as Airflow or Azkaban (Airflow Preferable). - Can work with spark, EMR and Kubernetes cluster. - Can write spark jobs, shell scripts and optimized data pipelines. - Experience with relational SQL and NoSQL databases, including Postgres, Amazon Redshift and MongoDB. - Have an understanding of OOPS-based data platforms, and can develop data platform modules. (ref:hirist.tech)

Location: noida, IN

Posted Date: 11/27/2024