Senior Data Engineer - ETL/Data Pipeline

Key Responsibilities : Data Pipeline Development : - Design, develop, and maintain robust data pipelines using tools like Airflow, Databricks, and Pyspark. - Extract, transform, and load data from various sources, including databases, APIs, and cloud storage. - Ensure data quality and integrity through rigorous testing and validation procedures. Big Data Technologies : - Leverage big data technologies like Hadoop, Spark, and Snowflake to process and analyze large datasets. - Optimize data pipelines for performance and scalability. - Implement data warehousing and data lake solutions. Cloud Infrastructure : - Utilize AWS services (i.e., S3, EMR, Redshift) to build and manage cloud-based data infrastructure. - Automate infrastructure provisioning and management using tools like Terraform and CloudFormation. API Integration : - Integrate with external APIs (i.e., Facebook Graph API, GA, Segment, Google Ad Words, Pinterest, Snapchat) to extract valuable data. - Handle API rate limits, authentication, and error handling. Data Quality Assurance : - Implement data quality checks and monitoring to ensure data accuracy and consistency. - Identify and resolve data quality issues proactively. Collaboration : - Collaborate with data analysts, data scientists, and other stakeholders to understand their data needs and deliver solutions. - Contribute to a culture of data-driven decision-making. Required Qualifications : - 6 years of experience in data engineering and ETL pipeline development. - Proficiency in SQL, Python, and Pyspark. - Strong understanding of big data technologies (Hadoop, Spark, etc. - Experience with cloud platforms, especially AWS. - Proficiency in data pipeline orchestration tools like Airflow. - Experience with data warehousing and data lake architectures. - Knowledge of data modeling and data warehousing concepts. - Experience with data quality and validation techniques. - Strong problem-solving and analytical skills. - Excellent communication and collaboration skills. Preferred Qualifications : - Experience with Kubernetes and Docker. - Knowledge of machine learning and AI concepts. - Experience with data visualization tools (i.e., Tableau, PowerBI). - Certifications in AWS, Azure, or GCP (ref:hirist.tech)

Location: chennai, IN

Posted Date: 11/26/2024