Particle Black - Lead Data Engineer

Role : Lead Data Engineer Exp. Range : 12- 17 yrs Location : Noida Data Engineer Lead with over 12 years in the field, specializing in data engineering and Azure Data Cloud within IT Services & Consulting. This role demands a professional who combines technical expertise with strong leadership, demonstrating accountability, commitment to quality, and the ability to guide a Scrum-based data engineering team through complex project deliverables. The ideal candidate will be a skilled communicator, capable of managing timelines, ensuring accurate project reporting, and escalating issues when necessary. Key Responsibilities : - Serve as the primary technical lead, responsible for managing and mentoring a Scrum-based data engineering team, particularly in an Azure-centric environment. - Ensure team alignment with project objectives and deliverables, oversee task breakdowns, and monitor progress against milestones. - Develop accurate project estimates and comprehensive plans, utilizing advanced knowledge in data engineering to anticipate and address challenges early in the project lifecycle. - Create work breakdown structures that ensure clarity and manageable tasks for the team. - Design, build, and manage robust data pipelines that facilitate data transformation, integration, and optimization across the Azure Data Cloud environment, ensuring seamless data flow and availability. - Implement complex data transformations using Azure Synapse Spark and Python to meet business needs. Leverage knowledge in metadata-driven frameworks to create flexible, scalable, and adaptable data solutions. - Oversee the code quality through structured pull requests (PRs), code reviews, and consistent unit testing. Emphasize adherence to best practices in coding standards, error handling, and modularity. - Lead the team in performance, load, functional, and regression testing to ensure data processes are efficient, stable, and scalable. Implement best practices in ETL and pipeline optimizations to maintain high performance across the data infrastructure. - Act as a consultant and support resource for production issues, addressing complex troubleshooting, debugging, and system optimization. Provide guidance to ensure data integrity, system uptime, and minimal disruptions. - Drive the application of Agile methodologies, specifically in a data engineering context, using CI/CD practices for rapid deployment and iteration. - Collaborate cross-functionally with other Agile teams to align goals, manage dependencies, and adapt to evolving project requirements. - Ensure comprehensive and up-to-date technical documentation for all solutions, from design specifications to user guides. - Regularly update stakeholders on project status, escalate issues as needed, and maintain transparency throughout the project lifecycle. Required Skills & Technical Proficiency : - Extensive experience in Microsoft Azure, particularly with Azure Synapse Spark for large-scale data processing and Azure DevOps for CI/CD and source control. - Proficient in data engineering principles, complex SQL query design, and handling diverse file formats, including Parquet and Delta. - Advanced expertise in Python for data transformation and scripting tasks, with knowledge in metadata frameworks for optimized data management. - Skilled in testing methodologies that ensure optimal data pipeline performance and scalability in production environments. - Strong skills in creating clear, thorough technical documentation for solutions, processes, and troubleshooting guides. Education and Experience : - Education : Undergraduate degree (B.Tech/B.E., BCA) in any specialization; Postgraduate degree (M.Tech, MCA) in any specialization is preferred. Experience : 12 years of hands-on data engineering experience, with at least 3 years leading Scrum data engineering teams. This is a full-time, permanent position in the Engineering - Software & QA department. Its ideal for a candidate who thrives in a collaborative, fast-paced Microsoft Azure-focused environment and is committed to delivering reliable, scalable, and high performing data engineering solutions. About company : Software company helping government agencies achieve better outcomes through artificial intelligence and the strategic use of data. Our solutions engineered for health and human services programs deliver enhanced efficiency and improve the lives of citizens. Together with our agency customers, we achieve remarkable outcomes for families. Our built-for-government suite of case management solutions employ configurable modules, cloud-based tech, a powerful AI assistant, an intuitive interface, and data-driven workflows on a low-code platform that enables agencies to modernize faster, and with greater confidence, than custom development projects (ref:hirist.tech)

Location: noida, IN

Posted Date: 11/23/2024