Data Engineer - Azure Databricks

Key Responsibilities : - Framework Development & Optimization : Design and develop modular, reusable, and scalable Python-based frameworks to support data engineering and processing tasks. - Data Engineering & Transformation : Implement and maintain frameworks that facilitate the ingestion, transformation, and processing of large datasets using Azure services. - Pipeline Orchestration & Automation : Develop and optimize data engineering pipelines leveraging Azure Function Apps, Databricks (PySpark), and Apache Airflow to ensure efficient data processing. - Real-Time & Event-Driven Architectures : Design and implement real-time data streaming solutions and event-driven architectures using Azure Event Hub. - Data Warehousing & Dimensional Modeling : Work with dimensional modeling concepts to build robust data warehouse architectures, ensuring optimized storage and retrieval for analytics and reporting. - Integration with Azure Data Services : Architect and integrate frameworks with Azure Data Lake, Azure Storage, and Cosmos DB to enable seamless data flow and management. - Serverless Data Engineering Solutions : Design and deploy serverless solutions using Azure Function Apps to support scalable and cost-efficient data engineering frameworks. - CI/CD & DevOps Implementation : Develop and manage CI/CD pipelines using Azure DevOps, ensuring streamlined integration, testing, and deployment of data engineering frameworks. - Automated Testing & Monitoring : Implement automated testing, deployment, and monitoring processes to maintain high-quality code, performance optimization, and rapid iteration. Required Skills & Experience : - Strong expertise in Python, with a focus on developing scalable frameworks for data engineering. - Hands-on experience with Azure cloud services, including Azure Function Apps, Event Hub, Data Lake, Storage, and Cosmos DB. - Proficiency in PySpark and Databricks, with experience in handling large-scale data processing and transformation. - Strong background in Apache Airflow for workflow orchestration and pipeline automation. - Expertise in Data Warehousing concepts, including dimensional modeling, star schema, and optimized data storage. - Experience in real-time data streaming and event-driven architectures using Azure Event Hub. - Strong understanding of CI/CD principles with hands-on experience in Azure DevOps for deployment automation. - Knowledge of automated testing, monitoring, and logging for maintaining high-performance data pipelines. - Problem-solving mindset with a focus on scalability, efficiency, and automation in data engineering solutions. (ref:hirist.tech)

Location: in, IN

Posted Date: 3/13/2025