Data Engineering - Google Cloud Platform

Data Engineering : - Good Experience or strong understanding of GCP (Big query) - Good Experience on Bigtable, Pub/Sub, Data proc, Data Flow - Good Experience on SQL, Python/Airflow Roles & Responsibilities : - Design, build, and maintain robust data pipelines using Google Cloud Platform (GCP) tools, including Dataflow and Dataproc, to ensure seamless data ingestion and transformation. - Data Modeling and Architecture - Create and maintain efficient data models in BigQuery and Bigtable, optimizing for performance and scalability to support analytical workloads. - Real-time Data Processing - Implement real-time data processing solutions using Pub/Sub, ensuring timely availability of data for analytics and reporting. - SQL Development - Write and optimize complex SQL queries for data extraction, transformation, and loading (ETL) processes, ensuring data accuracy and integrity. - Python and Workflow Management - Develop and manage workflows using Python and Apache Airflow, orchestrating various data processes and ensuring smooth execution of data tasks. Data Quality Assurance : - Monitor and validate data pipelines to ensure high data quality and reliability, implementing data quality checks and automated testing frameworks. - Collaboration and Communication - Work closely with data analysts, data scientists, and other stakeholders to understand data requirements and provide necessary data support. Performance Tuning : - Analyze and optimize the performance of data pipelines and queries, recommending improvements as necessary to enhance efficiency. (ref:hirist.tech)

Location: chennai, IN

Posted Date: 11/26/2024