Job Summary
We are seeking an experienced Senior Data Engineer to design, implement, and maintain our data infrastructure and pipelines. The ideal candidate will have a strong background in data engineering, big data technologies, and cloud platforms. You will work closely with data scientists, analysts, and other stakeholders to ensure efficient and reliable data processing and storage solutions.
Key Responsibilities:
- Design, develop, and maintain scalable data pipelines and ETL processes
- Implement and optimize data storage solutions, including data warehouses and data lakes
- Collaborate with data scientists and analysts to understand data requirements and provide efficient data access
- Ensure data quality, consistency, and reliability across all data systems
- Develop and maintain data models and schemas
- Implement data security and access control measures
- Optimize query performance and data retrieval processes
- Evaluate and integrate new data technologies and tools
- Mentor junior data engineers and provide technical leadership
- Collaborate with cross-functional teams to support data-driven decision-making
Requirements
- Bachelor's or Master's degree in Computer Science, Engineering, or a related field
- 5+ years of experience in data engineering or related roles
- Strong programming skills in Python, Java, or Scala
- Extensive experience with big data technologies such as Hadoop, Spark, and Hive
- Proficiency in SQL and experience with both relational and NoSQL databases
- Experience with cloud platforms (AWS, Azure, or GCP) and their data services
- Knowledge of data modeling, data warehousing, and ETL best practices
- Familiarity with data visualization tools (e.g., Tableau, Power Bl)
- Experience with version control systems (e.g., Git) and CI/CD pipelines
- Strong problem-solving skills and attention to detail
- Excellent communication and collaboration skills
Additional Preferred Qualifications
- Experience with stream processing technologies (e.g., Kafka, Flink)
- Knowledge of machine learning and Al technologies
- Familiarity with data governance and compliance requirements
- Experience with containerization and orchestration tools (e.g., Docker, Kubernetes)
- Contributions to open-source projects or relevant certifications
- Experience in Azure big data platform (ADF, Databrick)
- Experience in PowerBl