Data Engineer - ETL/PySpark

Responsibilities : - Design, develop, and maintain efficient and reliable data pipelines using PySpark. - Implement ETL processes to ingest, transform, and load data from various sources. - Optimize PySpark code for performance and scalability. - Work with large datasets and distributed computing frameworks. - Develop and maintain data quality checks and monitoring processes. - Collaborate with data scientists, analysts, and other engineers to understand data requirements and deliver data solutions. - Build and deploy data pipelines to production environments. - Troubleshoot and debug data pipeline issues. - Stay up-to-date with the latest data engineering technologies and trends, particularly within the PySpark ecosystem. - Contribute to the development and maintenance of data engineering best practices. - Participate in code reviews and provide constructive feedback. - Work in an Agile/Scrum environment. - Mentor junior data engineers. Required Skills : - 5 years of experience in data engineering. - Strong proficiency in Python and PySpark. - Deep understanding of Spark architecture and its core components. - Experience with data warehousing concepts and dimensional modeling. - Proficiency in SQL and experience writing complex queries. - Experience with data ingestion tools and techniques. - Experience with data transformation and cleansing. - Experience with data quality monitoring and validation. - Experience with workflow management tools (Airflow, Luigi). - Experience with version control systems (Git) (ref:hirist.tech)

Location: in, IN

Posted Date: 2/22/2025