We are seeking an experienced Data Platform Engineer to design and implement a cloud-based data lakehouse platform that ingests engineering and security tool data, transforms it through multiple layers, and serves it to both analytics dashboards and AI agents.
Experience:
- 8+ years in data engineering roles, with at least 2 years building lakehouse architectures (Bronze/Silver/Gold or equivalent medallion patterns)
- Proven track record delivering production-grade data platforms
- Experience with graph databases (Neo4j, Amazon Neptune, TigerGraph) for relationship modeling
- Hands-on with stream processing (Kafka, Flink, Spark Streaming, Kinesis)
Technical Skills (Core):
- Cloud Platforms: Deep expertise in AWS, (S3/Blob, RDS/SQL Database, managed Kafka, serverless compute)
- SQL & Data Modeling: Expert-level SQL, dimensional modeling, SCD2, normalization vs. denormalization trade-offs
- Transformation Tools: dbt, Databricks SQL, Dataform, or custom SQL/Python frameworks
- Programming: Python or Scala for data processing, scripting, and automation
- Orchestration: Airflow, Prefect, Dagster, Step Functions, or Azure Data Factory
- IaC: Terraform, CloudFormation, Pulumi, or ARM templates
Technical Skills (Preferred):
- Search: OpenSearch, Elasticsearch, or Solr for text indexing and retrieval
- Graph: Neo4j Cypher, SPARQL, or Gremlin for graph queries; experience with graph ETL
- Data Quality: Great Expectations, dbt tests, or custom validation frameworks
- Real-time: Flink, Spark Streaming, or serverless event processing (Lambda, Cloud Functions)
- Monitoring: Grafana, Datadog, or CloudWatch for data pipeline observability
Professional Skills:
- Communication: Explain technical trade-offs (cost, performance, complexity) to non-technical stakeholders
- Problem-Solving: Debug data quality issues, optimize slow queries, resolve schema conflicts
- Collaboration: Work with data scientists, DevOps engineers, and compliance teams
- Autonomy: Manage ambiguity; propose solutions when requirements are incomplete