AI Quality Specialist
- Krakow| Hybrid
We are seeking a hands-on IT Operations Generalist to serve as the ultimate judge of quality for our AI-driven root-cause analysis engines. In this role, you will evaluate whether AI-generated diagnostics and troubleshooting guidance deliver genuine, actionable value to system administrators facing real-world network and infrastructure outages.
You will sit directly within the Product team, bridging the gap between deep IT operations reality and cutting-edge AI engineering. Rather than managing timelines, your primary focus will be practical validation: inspecting telemetry from real customer alarms, establishing ground-truth diagnoses, and building automated, break-fix lab environments to stress-test new AI models before they reach production.
If you have spent years managing mid-sized IT environments, know what a true root-cause diagnosis looks like under pressure, and enjoy breaking infrastructure on purpose, this role offers a unique opportunity to shape the future of AI-assisted IT monitoring.
YOUR TASKS
Daily AI Diagnostic Evaluation & Telemetry Audit
- Review AI-generated incident analyses daily, leveraging real product telemetry and raw customer alarm streams.
- Evaluate diagnostic accuracy: determine whether the identified root cause is correct, whether symptom alarms are grouped logically, and whether recommended remediation steps are practical and effective.
- Maintain rigorous standards for quality, identifying hallucinations, redundant alert advice, and unhelpful diagnostic loops.
Engineering Feedback Loop & Quality Metrics
- Identify recurring AI failure modes (e.g., repeating alarm text without diagnosis, missing dependency links,generic fixes) and translate them into actionable engineering tickets and prompt/model requirements.
- Develop, track, and report baseline quality metrics for AI accuracy, relevance, and actionability.
- Help design systems to capture and integrate real customer feedback and resolution outcomes as our AI beta program expands.
Lab Environment Construction & Fault Injection
- Design, deploy, and maintain realistic hybrid lab environments (virtualized and cloud-based) that simulate real-world enterprise IT infrastructures.
- Safely engineer and execute fault-injection scenarios—reproducing real outages such as storage failures, expired SSL certificates, saturated links, DNS misconfigurations, and cascading service dependencies.
- Ensure lab scenarios are fully repeatable and documented for reliable regression testing.
Test Scenario Library & Regression Testing
- Build and curate a comprehensive benchmark library of golden test cases with verified ground-truth answers.
- Execute automated and manual regression test suites against candidate AI models prior to every release, ensuring zero degradation in diagnostic precision.
YOUR PROFILE
Must-Have Technical Experience
- System Administration Foundation: 5+ years of hands-on experience as an IT Generalist, Systems Administrator, or Infrastructure Engineer in a mid-sized enterprise environment.
- OS & Infrastructure Mastery: Deep proficiency with Windows Server administration combined with practical working knowledge of Linux distributions.
- Networking Essentials: Strong understanding of core networking protocols and infrastructure services,including DNS, DHCP, VLANs, subnet routing, SNMP, and SSH.
- Cloud Platform Experience: Hands-on operational experience with at least one major cloud platform (AWS,Azure, or GCP).
Automation & Lab Building Skills
- Command-Line & Scripting: Strong command-line fluency in PowerShell and Bash, with sufficient scripting ability (Python strongly preferred) to automate lab deployment, state resetting, and fault injection.
- Virtualization & Lab Automation: Proven experience building test environments using hypervisors (Hyper-V,VMware ESXi, Proxmox) or Cloud VMs.
Troubleshooting Judgment & Soft Skills
- Root-Cause Intuition: Superior technical troubleshooting judgment—the ability to look at an alert storm and immediately separate primary root causes from secondary symptoms.
- Technical Communication: Exceptional written communication skills for authoring clear, concise bug reports,lab reproduction steps, and diagnostic reviews.
- Pragmatic Mindset: Driven by real-world utility rather than theoretical perfection; passionate about making tools genuinely helpful for sysadmins.
Nice-to-Have Qualifications
- Monitoring Tool Experience: Prior hands-on experience configuring, managing, or querying enterprise monitoring platforms—such as PRTG Network Monitor, SolarWinds, Datadog, LogicMonitor, or open-source solutions (Zabbix, Nagios/Icinga, Prometheus/Grafana, Checkmk).
- AI Tooling Awareness: Basic familiarity with prompt engineering concepts, LLM benchmarking, or evaluation frameworks.
FACTS FOR YOU
- Many possibilities to further develop your individual skillset
- Innovative and creative work environment with plenty of room for new ideas
- A structured onboarding phase with experienced colleagues
- Flexible working hours and a family-friendly environment
- Permanent employment contract
- A great working atmosphere in an enthusiastic international team with flat hierarchies
- Paessler is a fast-growing midsized company that has been in the market for 20 years. The company is focused on long-term goals and is currently comprised of 400+ employees
