Platform & Pipeline Engineering
- Develop, automate, and maintain resilient data pipelines connecting on-premises
- laboratory equipment, high throughput test benches, and edge systems to Microsoft Azure.
- Investigate, unpack, and parse complex, non-standard, binary, proprietary, and undocumented file formats where standard connectors do not exist.
- Implement parsers and normalization routines capable of ingesting diverse technical data streams, for example:
- High-frequency sensor streams, time series, and bus telemetry.
- Multi-dimensional scientific arrays, spatial grids, and spectral data.
- - Hierarchical, nested, and unstructured binary logs or custom ASCII exports.
- Event logs, operational metadata, and intermittent experimental batch runs.
- Clean, validate, and transform fragmented or schema-less raw data into optimized, partitioned Lakehouse tables (Delta Lake/Parquet) ready for analytics and ML model training.
- Deploy, monitor, and maintain pipeline resources on Azure using Terraform, Databricks, and containerized workflows following established team templates.
Collaboration & Lab Support
- Work directly alongside experimental scientists, lab engineers, and technicians to decipher raw measurement files, clarify domain-specific semantics, and automate recurring data uploads.
- Collaborate closely with solution architects and senior engineers to implement data ingestion patterns, cataloguing schemas, and governance standards.
- Maintain clear technical documentation, code comments, and onboarding walkthroughs to help research teams connect their equipment seamlessly.
- Support proof-of-concept initiatives exploring metadata integration with corporate AI assistants, data discovery catalogs, and modern data tools.
Education
- BSc, MSc or Ph.D. in Data Engineering, Computer Science, Software Engineering, Applied Physics, Electrical Engineering, or a related field.
Experience
- 3+ years of professional experience in data engineering, building automated and reliable data pipelines.
- 1–2 years of practical experience handling heterogeneous, non-standard, or binary data sources (e.g., physical sensor logs, laboratory exports, hardware telemetry, or raw byte streams).
- 1+ years of hands-on experience utilizing Apache Spark (preferably Databricks) and Microsoft Azure in production environments.
- Demonstrated experience deciphering, parsing, or converting non-standard, legacy, or custom data formats into structured data structures.
Technical Skills
- Strong proficiency in Python for data manipulation (including binary data handling using modules like struct, byte manipulation, NumPy, Pandas, and PySpark) and Bash.
- Solid understanding of modern Lakehouse concepts (Delta Lake), distributed data processing, SQL, and NoSQL/document datastores.
- Proficient with Git, GNU/Linux environments, and working knowledge of Infrastructure as Code (Terraform).
- Practical experience with standard transport mechanisms (e.g., REST APIs, SFTP, Kafka/Event Hubs, or IoT/streaming protocols).
Mindset & Soft Skills
- Strong analytical mindset with a natural curiosity to open unfamiliar files, inspect headers, and solve data decoding puzzles.
- Friendly, empathetic communication style with the ability to bridge conversations between laboratory technicians, domain scientists, and software teams.
- C2 CEFR level in English, written and spoken.
Nice to Have
- Experience with scientific/hierarchical file formats (e.g., HDF5, NetCDF, Parquet, Protobuf).
- Familiarity with industrial telemetry or time-series historian protocols (e.g., OPC UA, MQTT).
- Exposure to Azure analytics tooling (Azure Data Factory, Azure Event Hubs, Azure Data Explorer).
- Experience with vectorial databases.
Languages
- Fluent in Portuguese.
- B2 CEFR level in English, written and spoken.
- B2 CEFR level in German, written and spoken is a plus
What we offer you:
Challenges in an international environment | career opportunities in one of the best companies in the world | flexible hours | medical services | discounts for employees | sports and health related activities | canteen | good access to public transports | space for creativity.
Success stories don´t just happen. They are made...
Make it happen! We are looking forward to your application!