Data Engineer - Azure Databricks/PySpark

Job Description : We are seeking a highly skilled Data Engineer with a strong focus on hands-on coding to join our growing team. The ideal candidate will have expertise in Databricks, Cloud (Azure or AWS), Pyspark, and Advanced SQL. You will play a key role in developing and optimizing data pipelines, working on large-scale data processing, and ensuring data security. This is a hands-on development role, where you will be responsible for building, improving, and securing data solutions. Key Responsibilities : - Develop and optimize data pipelines using Databricks and Pyspark in Azure or AWS cloud environments. - Write efficient and optimized SQL queries for data extraction, transformation, and loading (ETL). - Implement data security protocols on Databricks to ensure that sensitive data is protected across various pipelines. - Create and maintain data models, ensuring the structure and performance of data storage solutions are optimized for analytics. - Collaborate with teams to design and implement data workflows and integrate data from various sources into the data warehouse. - Work on Data Warehouse projects, developing solutions for data ingestion, transformation, and reporting. - Troubleshoot and resolve performance issues related to data pipelines, databases, and queries. - Ensure best practices for data management, security, and scalability within the cloud environment. - Stay updated with emerging technologies in the cloud and big data domain to ensure cutting-edge solutions are implemented. Required Skills and Qualifications : - 7-10 years of experience in data engineering or a related field with a focus on hands-on coding. - Strong expertise in Databricks, Pyspark, and Cloud (Azure or AWS) environments. - Proficiency in writing Advanced SQL queries to work with large datasets and databases. - Proven experience working on Data Warehouse projects, focusing on data ingestion, transformation, and storage solutions. - Hands-on experience in implementing data security within Databricks and cloud platforms. - Solid understanding of ETL processes and data pipeline development. - Familiarity with Hadoop and other big data processing frameworks is a plus. - Strong troubleshooting skills to optimize data pipelines and SQL performance. Preferred Skills : - Experience with Hadoop and big data technologies. - Knowledge of data governance and best practices in data management. - Experience in data integration, handling structured and unstructured data sources. - Practical experience in data modelling, with Data Vault being a plus. (ref:hirist.tech)

Location: bangalore, IN

Posted Date: 1/11/2025