Analyst, Applied AI

3 нед. назад

Sri LankaWorldwideSeniorHybrid

machine learningdata analysisstatistical analysis

Analyze and design multi-step tasks, grading rubrics, and ground-truth datasets to evaluate AI systems in real-world workflows at 's Applied AI team in Colombo.

Обязанности

We're hiring an Analyst to join our Applied AI team in Colombo. You'll design the tasks, rubrics, and datasets we use to evaluate AI systems on real-world workflows, and define the ground truth that decides whether a model's attempt counts as correct.
The quality of everything we build downstream depends on the quality of the tasks and ground truth you produce.
Design realistic, multi-step tasks that test AI systems on real-world workflows, and author the grading rubrics that score them
Build and curate ground-truth datasets from real operational data and domain research
Review model attempts against your tasks: analyze failure modes, catch scoring loopholes, and identify where models pass for the wrong reasons
Calibrate task difficulty so results separate models in a meaningful way
Analyze score distributions across models and turn results into clear, defensible findings
Contribute to research write-ups and published evaluations alongside our engineers and academic reviewers

Требования

Master's or PhD in AI, Machine Learning, Computer Science, Statistics, or a closely related quantitative field
Strong understanding of how large language models are trained and evaluated
Hands-on experience assessing LLM outputs: evals, annotation, red-teaming, or benchmark work
Comfortable with Python and data tooling for building datasets and analyzing results
Exceptional attention to detail and intellectual honesty; you flag a broken task even when the scores look good
Clear written communication; you can explain a subtle failure mode to both engineers and non-technical readers

Будет плюсом

Familiarity with reinforcement learning concepts (reward design, verifiable rewards)
Experience contributing to published research or open evaluation suites
Domain knowledge in recruiting, staffing, or other operations-heavy industries

Условия

Ground-floor role on a new applied AI team with direct ownership of what we measure and publish
Work reviewed by and published alongside experienced researchers
High visibility, fast-paced, execution-driven environment
Competitive pay and benefits

Другое

is building AI interviewers that automate and improve hiring through conversational AI. We work closely with companies to bring AI into real hiring workflows. We also run an applied AI team that studies where today's models succeed and fail across verticals.