What the work looks like day to day
An eval is a test for an AI system. You give it an input, run grading logic on the output, and record whether it succeeded. The evaluation engineer writes those tasks, decides what counts as success, and builds the harness that runs hundreds of them in parallel, records every step, grades the results and rolls them up into scores. For agents, one task can span many turns of tool calls, so the harness also has to set up and reset the environment the agent changes.
Grading comes in three kinds. Code-based graders check strings, unit tests, tool calls and final state. They are fast, cheap and repeatable, but they miss valid answers phrased differently. Model-based graders, often called LLM-as-a-judge, score against a rubric and handle open-ended output, but they vary between runs and need calibrating against people. Human review by subject experts is the slowest and most expensive, and it is the reference the other two are tuned against. Most of the craft is choosing the right mix for each task.
Then there is the release loop. Before a change ships, you run it against a saved dataset and compare scores with the last version. If something it used to handle now fails, the release waits. After launch you score real production traces, where there is usually no reference answer, and turn recurring failures into new test cases. Anthropic also says to read transcripts regularly, because a failed task can mean a real agent mistake or a grader that rejected a good answer.
Why so many AI job ads now ask for evals
Language model output breaks the assumptions of ordinary software testing. The same question can get two different answers that are both fine. Many tasks have no single correct answer. And the worst failures, like a confident false statement, an ignored policy or tools called in the wrong order, raise no error at all. Code can pass every functional test and still give a wrong or unsafe answer. Exact-match testing cannot catch that, so teams need people who can define good behavior and measure it.
Anthropic describes the cost of skipping this step. Without evals, debugging only starts after users complain, and teams cannot tell a real regression from noise. Its own coding agent started with feedback from staff and users and then added evals, first for narrow behaviors and then for broader ones. Employers now write the skill straight into ads. An OpenAI posting for a backend engineer on evals asks for experience building production evals and for familiarity with LLM evaluation methods, and the role covers regression monitoring, drift monitoring and golden datasets.
Which teams own the work
Few companies have a team called evaluation. The work usually sits in two places. Anthropic says its most effective setup was a dedicated evals team that owns the core infrastructure, while domain experts and product teams write most of the tasks and run the evaluations themselves. So an evaluation engineer may be an infrastructure engineer building the harness, or an applied engineer on a product team who owns that product's quality bar.
You will also see the work inside other job titles. The OpenAI example sits on a support automation team and pairs engineers with data science and research partners. On smaller teams, the machine learning engineer, the AI engineer or the data scientist building the feature is often the one writing the evals. Searching ads for the word evals finds more of these roles than searching the job title alone.
How to show this skill to an employer
Build a public eval suite for an LLM app you made. Anthropic says 20 to 50 simple tasks drawn from real failures is a good start, so you do not need a huge dataset to show judgment. Include at least one code-based grader and one model-based grader. Label a sample by hand, then report how closely the model grader agrees with your labels. Hugging Face's LLM-as-a-judge tutorial uses agreement between two human raters as the baseline for this comparison.
Then show the loop running. Wire the suite into CI so a prompt or model change that lowers scores blocks the merge. Run several trials per task and report pass rates, because output changes from run to run. Write up a case where reading transcripts showed the grader was wrong. Hosted and open-source tools exist for this, including Braintrust, LangSmith, Langfuse, Arize Phoenix and Harbor, and naming the one you used is fine. Interviewers care more about how you defined success and caught bad graders than about which tool you picked.
Pay
USD a year, before tax. US figures. BLS has no series for LLM evaluation, so the BLS rows use software developers from OEWS May 2025, moved to 2026 with the Employment Cost Index. That series covers the whole occupation, not eval specialists. The job ad rows are midpoints of base pay ranges in 139 live machine learning engineer ads from 56 employers on 5 November 2026. They lean toward large AI and technology employers and leave out stock and bonus. No source gives a figure for eval work alone. Refer to the Machine Learning Engineers salary guide for figures from live adverts.
What you need
- Python and backend engineeringEval harnesses are production software. The OpenAI evals posting asks for 4+ years of backend experience and lists Python, FastAPI and Postgres in its stack.
- Grader designYou need to know when to use code checks, rubric-based model judges, pairwise comparison or expert review, and what each one misses.
- Statistics for noisy outputResults vary between runs, so you run several trials per task and report metrics such as pass@k. You also need to tell a real regression from noise.
- Judge calibrationModel-based graders need checking against human labels before anyone trusts them. Measuring agreement between raters is a basic skill for this.
- Product judgmentMost of the hard work is writing down what success means for a specific task, policy and user. That means working closely with domain experts.
- Production monitoringAfter launch you score live traces with no reference answer and turn repeated failures into new test cases.
How to get there
- Years 1-3Build software or ML experienceMost eval roles grow out of backend, ML or data science work. Learn to ship and test a real service first.
- Months 1-3 of focusShip an LLM feature and test itBuild a small app or agent and write 20 to 50 tasks from the failures you actually see.
- Months 3-6Add graders and calibrate themCombine code checks with a model judge, hand-label a sample, and measure how often the judge agrees with you.
- Months 6-9Automate regression gatingRun the suite in CI with several trials per task and block changes that lower scores.
- Year 1 in roleTake on production qualityAt work, own monitoring for one LLM feature, turn trace failures into tests, and publish quality reports.
- Year 2+Move to evals infrastructure or leadBuild shared harnesses and tooling that product teams and domain experts use to write and run their own evals.
Open now on AutomationRoles.ai
Questions people ask
Is LLM evaluation engineer a separate job title?
Sometimes, but the work more often appears inside machine learning engineer, AI engineer, backend engineer and data scientist ads. OpenAI, for example, posted a backend software engineer role focused on evals for its support automation team. Search for the word evals as well as the title.
How is an eval different from a benchmark?
A public benchmark measures a model's raw ability on a fixed, shared test set, and it helps you choose a base model. An eval scores your whole system, with its prompts, tools, retrieval and policies, on the behavior you care about. You run evals again every time the system changes.
Can I trust an LLM to grade another LLM?
Only after you check it. Model-based graders are flexible and scale well, but they vary from run to run and need calibrating against human graders. Hand-label a sample, measure how often the judge agrees with you, and recheck it when you change the rubric or the judge model.
How much does LLM evaluation work pay?
No source gives pay for eval work alone. The nearest BLS series, software developers, runs about $108,500 to $177,400 at the 25th to 75th percentile in the US for 2026, and covers the whole occupation. One OpenAI evals posting listed base pay of $266K to $445K plus equity, which is a single employer and not a market rate.
Is there a certification for LLM evaluation?
None of the sources read for this guide describe a certification for evaluation work. A public eval suite with calibrated graders and CI gating shows the skill more directly than a credential would.
How many test cases do I need to start?
Fewer than most teams think. Anthropic suggests 20 to 50 simple tasks drawn from real failures, because early changes have large, visible effects. Mature products need larger and harder suites to detect small changes.
