Azure Databricks Engineer - ETL

Key Responsibilities : - Data Pipeline Development : Design, develop, and optimize data pipelines and workflows in Azure Databricks for ETL/ELT processing, data transformation, and analytics. - Big Data Solutions : Architect and implement big data solutions using Azure Databricks, Spark, Azure Data Lake, and other Azure data services. - Data Integration : Integrate structured, semi-structured, and unstructured data from various sources like Azure SQL, Cosmos DB, Blob Storage, and external systems into the data platform. - Performance Tuning : Monitor, optimize, and troubleshoot Databricks clusters, Spark jobs, and performance bottlenecks in data pipelines. - Data Models : Design and implement data models and scalable architecture to ensure efficient and consistent data processing. - Collaboration : Work closely with Data Engineers, Data Scientists, and other stakeholders to ensure seamless integration of analytics solutions and develop new insights from data. - Security & Governance : Implement security best practices in Azure Databricks environments, ensuring proper access controls, data masking, encryption, and compliance with organizational and industry regulations. - Automation : Develop and manage CI/CD pipelines using tools such as Azure DevOps, Terraform, and automate deployment of infrastructure and Databricks jobs. - Documentation : Create and maintain comprehensive documentation of architecture, processes, and configurations for Azure Databricks solutions. Required Skills and Qualifications : Experience : 3 years of hands-on experience with Azure Databricks and big data technologies like Apache Spark. - Azure Expertise : Strong knowledge of Azure Data Lake, Azure Data Factory, Azure Synapse Analytics, Azure SQL Database, Azure Blob Storage, etc. - Programming Languages : Proficiency in Python, Scala, or SQL for data processing and pipeline development. - Data Modeling : Strong understanding of relational and non-relational data models, including experience with ETL/ELT processes. - Spark Tuning : Hands-on experience with performance tuning and optimization in Apache Spark and Databricks environments. - Version Control : Experience with Git and working within an Agile/Scrum development methodology. - CI/CD : Experience in building and maintaining CI/CD pipelines using Azure DevOps or other similar tools. - Security : Familiarity with RBAC, data encryption, and other Azure security features related to data management. - Problem Solving : Strong analytical skills, with the ability to troubleshoot and solve complex issues in data pipelines and workflows. Preferred Skills : - Certifications : Azure certifications such as Azure Data Engineer Associate, Azure Solutions Architect, or Azure Databricks certifications. - Data Science Integration : Understanding of integrating data science models into Databricks workflows using MLflow. - Infrastructure as Code (IaC) : Experience with Terraform for automating cloud infrastructure deployments. - DevOps Knowledge : Familiarity with Databricks Repos, Azure Monitor, and other DevOps tools and practices.- Education : - Bachelor's or Master's degree in Computer Science, Data Engineering, Information Systems, or a related field. (ref:hirist.tech)

Location: in, IN

Posted Date: 11/26/2024