Data Engineer - SQL/Python

Job Description : Key Responsibilities : - Design and develop data pipelines using tools like Apache Spark and PySpark. - Extract data from various sources (databases, APIs, files) using appropriate techniques. - Transform and clean data to ensure data quality and consistency. - Load data into data warehouses, data lakes, or data marts. - Design and implement data models that are efficient, scalable, and aligned with business requirements. - Create data pipelines that are reliable, fault-tolerant, and maintainable. - Optimize data pipelines for performance and cost-effectiveness. - Leverage cloud platforms (AWS, GCP, or Azure) to build scalable and cost-effective data solutions. - Utilize cloud services like data warehouses, data lakes, and data processing services. - Implement data security and privacy best practices in cloud environments. - Implement data governance policies and procedures to ensure data quality and security. - Monitor data pipelines and proactively identify and resolve issues. - Collaborate with data analysts and data scientists to understand their data needs and provide solutions. Required Skills and Experience : - Strong proficiency in Python and PySpark. - Experience with Apache Spark and its core components (Spark SQL, Spark Streaming, Spark MLlib). - Knowledge of SQL and database technologies. - Experience with cloud platforms (AWS, GCP, or Azure). - Understanding of data warehousing and data lake concepts. - Experience with data visualization tools (Tableau, Power BI). - Strong problem-solving and analytical skills. - Excellent communication and collaboration skills (ref:hirist.tech)

Location: hyderabad, IN

Posted Date: 1/8/2025