Data Engineer - Python/Spark

Responsibilities : - Design, develop, and maintain data pipelines and solutions using Databricks. - Utilize Databricks workspaces, clusters, and notebook organization effectively. - Develop and optimize Spark jobs for data transformation and analysis. - Implement data quality checks and validation processes within Databricks. - Manage Databricks clusters for optimal performance and resource utilization. - Integrate Databricks with Azure services such as Azure Data Lake Storage (ADLS) and Azure Blob Storage. - Develop and maintain data ingestion and extraction processes from various Azure sources. - Ensure seamless data flow between Databricks and other Azure services. - Develop and maintain data transformation logic using Python, SQL, and Spark within Databricks notebooks. - Perform data analysis and generate insights to support business decision-making. - Optimize data processing pipelines for performance and scalability. - Design and implement data integrations between Databricks and Microsoft Power Platform (Power BI, Power Apps, Power Automate). - Develop data connectors and APIs for seamless data exchange. - Ensure data accuracy and consistency across platforms. - Develop and consume REST APIs for data integration. - Work with JSON data formats for data exchange. - Design and implement data models to support analytical and reporting needs. - Implement CI/CD pipelines for automated deployment of Databricks notebooks and data pipelines. - Utilize CI/CD tools and processes to ensure efficient and reliable deployments. - Automate data processing and deployment tasks. - Troubleshoot and resolve complex data engineering issues. - Perform root cause analysis and implement corrective actions. - Monitor system performance and implement optimizations. - Create and maintain technical documentation for data pipelines, processes, and configurations. - Collaborate with data scientists, analysts, and other stakeholders to understand data requirements and deliver solutions. - Communicate effectively with technical and non-technical audiences. - Required Skills and Expertise : - Extensive experience with Databricks, including workspace, clusters, and notebook organization. - Proficiency in developing and optimizing Spark jobs. - Experience integrating Databricks with Azure services (ADLS, Blob Storage). - Strong proficiency in Python and SQL. - Experience with Spark for data transformation and analysis. - Experience with Azure Data Lake Storage (ADLS) and Azure Blob Storage. - Understanding of Azure cloud services and data infrastructure. - Hands-on experience with Microsoft Power Platform (Power BI, Power Apps, Power Automate). - Experience in integrating Databricks with Power Platform. - Experience with REST APIs and JSON data formats. - Understanding of data modeling concepts and techniques. - Proficiency in CI/CD tools and processes. - Experience with automation of data pipelines and deployments. - Strong analytical and problem-solving skills. - Ability to troubleshoot and resolve complex technical issues. - - Excellent communication and interpersonal skills. - Ability to communicate technical concepts to both technical and non-technical audiences. Preferred Skills : - Experience with data warehousing and data lake architectures. - Knowledge of data governance and data quality practices. - Experience with cloud-native development practices. - Familiarity with containerization technologies (e.g., Docker, Kubernetes). - Experience with other cloud platforms (AWS, GCP). Qualifications : - Bachelor's or Master's degree in Computer Science, Data Engineering, or a related field. - 9 years of experience in data engineering, with a strong focus on Databricks. - Proven experience in designing and implementing data pipelines and solutions using Databricks and Azure services. - Strong understanding of data transformation, analysis, and integration. - Must be able to join within a reasonable timeframe. (ref:hirist.tech)

Location: bangalore, IN

Posted Date: 4/19/2025