This role is for one of Weekday’s clients
Min Experience: 12+ years
Location: Bengaluru, Karnataka, India, Bengaluru
JobType: full-time
We are looking for an experienced Senior Staff Engineer – Data Platform with 12–17 years of experience to lead the architecture, design, and evolution of our core data platform. This role will be responsible for building scalable, reliable, and high-performance data infrastructure spanning lakehouse architecture, analytical pipelines, and transactional systems. The ideal candidate will possess deep expertise in Data Engineering and Databricks, along with a strong understanding of distributed systems, cloud infrastructure, data governance, and modern data architecture. As a senior technical leader, you will drive architectural decisions, establish engineering standards, mentor technical teams, and collaborate with cross-functional stakeholders to deliver robust data solutions that support business-critical applications and advanced analytics.
Requirements
Key Responsibilities
- Data Platform Architecture: Own the end-to-end architecture and technical roadmap of the data platform, ensuring scalability, reliability, performance, and long-term maintainability.
- Databricks Engineering: Design, develop, and optimize Databricks-based data solutions using Apache Spark, Delta Lake, and Lakehouse architecture principles.
- Data Pipeline Development: Build and maintain robust batch and streaming data pipelines to support analytics, reporting, machine learning, and operational use cases.
- Distributed Systems: Architect high-throughput, fault-tolerant data processing systems capable of handling large volumes of structured and unstructured data.
- Transactional Systems Integration: Design efficient data integration patterns between transactional databases, analytical platforms, and downstream data consumers.
- Performance Optimization: Identify and resolve bottlenecks in data processing, storage, query execution, and resource utilization to improve system efficiency and control infrastructure costs.
- Data Governance and Quality: Establish standards for data quality, lineage, security, access control, metadata management, and regulatory compliance.
- Technical Leadership: Lead complex engineering initiatives, influence architectural decisions across teams, and establish best practices for data engineering and platform development.
- Cross-Functional Collaboration: Partner with product managers, analytics teams, data scientists, and engineering leaders to translate business requirements into scalable technical solutions.
- Mentorship and Innovation: Mentor senior engineers, conduct architecture reviews, promote engineering excellence, and evaluate emerging technologies to continuously improve platform capabilities.
Must-Have Skills
- Extensive experience in Data Engineering, data platform architecture, and large-scale data processing.
- Strong hands-on expertise in Databricks, Apache Spark, Delta Lake, and Lakehouse architecture.
- Proven experience designing and operating scalable ETL/ELT pipelines and distributed data systems.
- Strong understanding of data modeling, data warehousing, batch processing, and real-time data streaming.
- Experience with cloud-based data platforms, preferably AWS, Microsoft Azure, or Google Cloud Platform.
- Proficiency in SQL and at least one programming language, such as Python or Scala.
- Strong knowledge of data security, governance, reliability, observability, and performance optimization.
- Demonstrated ability to lead architecture initiatives and influence technical strategy across multiple engineering teams.
Experience and Qualifications
- 12–17 years of overall experience in software engineering, Data Engineering, or data platform development.
- Significant experience building and scaling enterprise-grade data platforms in complex environments.
- Proven track record of technical leadership, architectural ownership, and successful delivery of mission-critical data systems.
- Excellent problem-solving, communication, stakeholder management, and mentoring skills.