Elastic10 дней назад
AI QA & Evaluation Engineer
Зарплата не указана
Bangalore
Обязанности
- 01Be a primary contributor to our AI strategy, helping validate and test AI infrastructure, custom solutions and third-party SaaS offerings
- 02Design and implement comprehensive test strategies for AI/ML systems, including accuracy, bias, robustness, and regression testing
- 03Design and implement self-contained evaluation tasks, including prompts, supporting files, and detailed grading rubrics to assess AI performance on functional workflows
- 04Automate validation suites for agentic/multi-agent systems, integration testing, and CI/CD pipelines for ML models
- 05Validate that AI/ML models are consuming accurate, authorized, and properly structured data sources; ensuring data quality across training and inference
- 06Meticulously observe and document AI agent behaviors, producing crisp, precise summaries and reports on model performance and hallucinations
- 07Validate prompt engineering outputs from a data accuracy standpoint, ensuring responses are grounded in verified data sources
- 08Iterate and refine evaluation tasks and rubrics based on feedback and team collaboration to ensure robust benchmarking methodologies
- 09Ensure all AI data sources and structures meet governance, regulatory, and compliance standards, while implementing best practices for security and data privacy
- 10Collaborate with teams from different areas including IT Engineering, IT Operations, Data & Integrations, PMO, CRM, Risk & Compliance, and business technology
- 11Stay current on the latest work in AI and make technical recommendations to the organization
Требования
- 01Proficiency in Python, TypeScript, or other programming languages used in AI and test automation
- 02Proven skill in designing or applying rubric-based evaluation, grading against set criteria, or building structured scoring frameworks
- 03Direct experience with LLM evaluation frameworks and benchmarking tools such as LangSmith, Confident AI, etc.
- 04Knowledge of the GenAI stack and solutions including Retrieval Augmented Generation (RAG)
- 05High attention to detail and ability to notice subtle patterns or inconsistencies (such as data hallucinations or logic errors) that others might miss
- 06Advanced written communication skills, especially for documenting nuanced observations and feedback
- 07Experience with Cloud platforms (Azure, GCP, AWS)
- 08Thorough understanding of DevOps/automation/CI/CD tools: GitHub, Terraform
- 09Comprehension with AI ethics, risk management, and data governance
Условия
- 01Distributed company
- 02Flexible locations and schedules for many roles
- 03Generous number of vacation days each year
- 04We match up to $2000 (or local currency equivalent) for financial donations and service
- 05Up to 40 hours each year to use toward volunteer projects you love
- 06Minimum of 16 weeks of parental leave
- 07Health coverage for you and your family in many locations