What the work looks like day to day
Most of the job is adversarial testing. NIST AI 600-1, the generative AI profile of the AI Risk Management Framework, describes AI red teaming as structured testing that probes a system for flaws such as inaccurate, harmful, or discriminatory outputs. In practice you write attack prompts, run them at scale, score the results, and hand reproducible failures to the team that owns the model or product.
Prompt injection gets the most attention. The OWASP Top 10 for LLM Applications 2025 lists it first, as LLM01. NIST separates direct injection, where an attacker types malicious instructions into the system, from indirect injection, where instructions hide in data the system retrieves later, such as an email or web page. Agents that call tools raise the stakes, because a hijacked agent can take actions in other systems.
Microsoft's AI Red Teaming 101 series shows the range: single-turn tricks like persona hacking and encoding to evade filters, multi-turn attacks that steer a model gradually, and automation with PyRIT, its open-source Python Risk Identification Tool.
Building the defenses and the incident process
Testing is half the job. The other half is building controls and proving they hold. AWS's mapping of its agentic AI guidance to the OWASP list shows typical layers: threat modeling at design time, treating prompts as versioned code, input filters, output filters that block sensitive data, prompt logging, and monitoring for injection attempts. Limiting what an agent is allowed to do counts as a control too.
A current OpenAI posting for agent safety work spells out the trade-off you manage. The role red teams agentic systems for data exfiltration and unsafe tool use, then tries to cut missed harmful actions, unnecessary blocks, approval burden, and latency together. Controls that block everything fail too.
Incident work borrows from security operations. NIST AI 600-1 suggests policies for after-action reviews of AI incidents and for updating incident response and disclosure processes. Safety engineers often write the runbook: how a report comes in, who triages it, how to reproduce it, and when a fix ships.
How it differs from AI governance
Governance and safety engineering use the same frameworks from different ends. The NIST AI Risk Management Framework is voluntary and organizes the work into four functions: Govern, Map, Measure, and Manage. Governance specialists mostly own Govern: policies, roles, risk tolerance, documentation, and reporting to leadership and legal. Safety engineers mostly own Measure and Manage: the tests, the metrics, and the code that enforces a policy at runtime.
A simple test: if your main output is a policy document or an audit, you are doing governance. If it is a test harness, a filter, a permission boundary, or a bug report with a reproduction, you are doing safety engineering. NIST also advises that red teaming results get further analysis before they feed governance decisions, which is where the two roles meet.
Who hires, and how people move in
The hiring we can point to clusters in three places. AI labs have dedicated safety teams; OpenAI's Safety Systems group is one example. Cloud providers run AI red teams and publish guidance, as Microsoft and AWS do. Companies deploying their own assistants and agents need people who can test and harden them, often inside a security or platform team.
Microsoft pitches its red teaming course at security professionals learning AI attack vectors and at ML engineers adding security testing to their workflow. Those are the two common routes in. Security people already know threat modeling, isolation, and incident response, and need to learn how models fail. ML engineers know evaluation and training, and need an attacker's mindset. The OpenAI posting says a background in security or AI control is welcome but not required, and asks for strong systems or security instincts.
Pay
USD a year, before tax. No official series or source we read gives pay for AI safety engineers. These are US figures for the nearest category: BLS OEWS May 2025 software developers, moved to 2026 with the Employment Cost Index, covering all software developers, not AI or safety specialists alone. The job ad rows are midpoints of pay ranges in live AutomationRoles.ai AI developer ads on 5 November 2026; they lean toward large AI and technology employers and show base pay only, leaving out stock and bonus. Refer to the AI Developers salary guide for figures from live adverts.
What you need
- Threat modelingYou need to map attack surfaces, permissions, and isolation boundaries in a system that includes a model, its tools, and its data sources. Both AWS and OpenAI material put threat modeling first.
- Adversarial testing skillYou should know direct and indirect prompt injection, single-turn and multi-turn attacks, and filter evasion, and be able to turn a one-off jailbreak into a repeatable test case.
- Evaluation and measurementWrite evaluations that tell a strong mitigation from a brittle one, and track missed harms against false blocks. This is where ML experience pays off.
- Programming and automationRed teaming at scale means writing code to generate attacks, send them, and score the responses. Microsoft's material uses Python and its PyRIT tool.
- Systems and security basicsSandboxing, process isolation, least-privilege permissions, and logging come up constantly once agents can call tools.
- Incident handlingTriage reports, reproduce failures, run after-action reviews, and update the process when gaps show up.
How to get there
- Starting pointStart from security or MLMost people arrive with depth in one of the two. Security people bring threat modeling and incident response. ML people bring evaluation and model knowledge.
- First monthsLearn how generative AI failsStudy the OWASP Top 10 for LLM Applications 2025 and the risk sections of NIST AI 600-1. Work through a free red teaming course, such as Microsoft's AI Red Teaming 101.
- Portfolio stageBuild an attack and defense projectRed team a small app or agent you built yourself. Automate the attacks, add input and output filters, and measure how far they cut successful attacks and how often they block safe requests.
- In roleTake safety work in your current jobVolunteer to threat model or test your team's AI features, or join internal red team exercises. A written report with reproducible findings counts as evidence.
- Later careerOwn controls and incident processSenior safety engineers design system-level controls, set the safety metrics, and own the incident runbook across products.
Open now on AutomationRoles.ai
Questions people ask
Is AI safety engineering the same as AI red teaming?
Red teaming is one part of it. Safety engineers also build the guardrails that fix what red teaming finds, measure whether those controls work, and run the incident process. Some large organizations have dedicated red teamers who mostly attack.
Do I need a license or certification to do this work?
No. No license exists for AI safety work, and no regulator oversees the job itself. Certifications and courses can help you structure your learning, but evidence of real testing and defensive work matters more to hiring teams.
Is prompt injection really the main threat?
The OWASP Top 10 for LLM Applications 2025 ranks it first, as LLM01. Indirect injection worries teams most when an agent reads outside content and can act on it, because hidden instructions in an email or web page can redirect the agent's tool use.
How is this different from an AI governance role?
Governance owns policy, roles, risk tolerance, and documentation, mostly the Govern function of the NIST AI Risk Management Framework. Safety engineers build the tests, metrics, and runtime controls that measure and enforce those policies. The two roles work together closely.
How much do AI safety engineers earn?
No source we read reports pay for this specialty. The nearest US benchmark is the BLS software developers series, with a 2026-adjusted median around $140,200 across all developers. Live AI developer ads on this board had a median midpoint of $203,000 in base pay, skewed toward large employers.
