Lovable6 дней назад
Data Scientist, Agent
Зарплата не указана
Полная занятостьОфис
Обязанности
- 01Define and own the metrics for agent quality: success, completion, error rates, and the behaviors that drive them
- 02Build the eval systems and experiment framework that decide whether an agent change ships, like an A/B-tested rollout that catches a change increasing errors before it reaches everyone
- 03Turn agent traces and telemetry into concrete fixes, working directly with the agent engineering team
- 04Build the tooling and agents that produce these evaluations continuously as the agent evolves
- 05Set the bar for how we judge agent behavior where there is no answer key to check against
Требования
- 01A data scientist who wants to make an AI agent measurably better, not just report on it
- 02Experience or strong interest in LLM evaluation and observability: building evals, scoring outputs, tracing agent behavior, and catching regressions
- 03Strong SQL and Python, applied statistics, and experimentation
- 04Comfortable designing A/B tests for agent changes where outcomes are noisy
- 05You build systems and agents that produce this insight continuously, rather than one-off analyses
- 06Instinct for what "good" looks like in agent behavior (success, error rates, task completion) and how to measure it when there is no clean answer key
- 07Entrepreneurial
- 08Thrives in ambiguity
- 09Works closely with the engineers building the agent