MSRcosmos - Senior Data Engineer - Azure Databricks

About the Role : We are seeking a highly experienced Senior Data Engineer with over 10 years of expertise in the big data domain. The ideal candidate will have strong proficiency in Azure Databricks, Spark Streaming, Python, PySpark, and SQL, along with a proven track record in designing and implementing robust data pipelines and data lake solutions. Key Responsibilities : Data Engineering & Development : - Develop and maintain scalable data pipelines using PySpark, Python, and Spark Streaming. - Implement ETL/ELT jobs for data ingestion and processing in data lakes. Databricks Expertise : - Utilize Databricks workspace, notebooks, job clusters, Delta Lake, Databricks Lakehouse, and Unity Catalog effectively. Big Data Management : - Design and implement data lake architectures and data warehouse solutions. - Work with SQL Server, NoSQL, Spark SQL for data modeling, query optimization, and parallel processing. Streaming Data Processing : - Integrate Kafka and Pub/Sub technologies for real-time data processing workflows. Agile Methodology : - Contribute to Agile development practices, including SCRUM methodologies and Application Lifecycle Management (ALM). Key Skills & Experience : - 10 years in IT within the big data domain, specializing in Hadoop, data lakes, and data engineering. - 3-4 years of hands-on experience with Databricks. - 5 years of strong development experience in PySpark and Python. - 6 years in designing and implementing data pipelines. - Proficient in Spark Streaming, SQL Server, NoSQL, Spark SQL, and data modeling. - Familiar with identity & access management, query optimization, and parallel processing. - Strong problem-solving skills with excellent communication abilities. - Knowledge of Agile development practices, SCRUM, and ALM. (ref:hirist.tech)

Location: hyderabad, IN

Posted Date: 4/15/2025