What an Inference Engineer Does (and What It Pays)

Training a frontier model gets the headlines. Running it is where the money goes. Every time someone asks an assistant a question, an agent calls a tool, or a coding assistant suggests a line, a model somewhere has to produce an answer, quickly and cheaply, millions of times a day. The person who makes that happen is the inference engineer.

This is one of the jobs I'm most excited about in AI right now. It sits right where the colossal data centres being built around the world meet the products people actually use, and it pays like it. In this edition of our What Each Role Does series, I'll explain what inference engineers do, the skills you need, how people get in, and what the published pay data really says.

What an inference engineer actually does

Prof Rod's role guide puts it neatly: inference engineers make models fast and cheap to serve, working on batching, caching, quantisation, kernels and the serving systems behind every API call.

A real job ad shows what that means day to day. Adaption's Inference Engineer posting in San Francisco says the engineer will "own the cost and performance of our inference stack." The responsibilities include improving throughput, cost and tail latency through KV-cache management, continuous batching, speculative decoding and quantisation; optimising long-context prefill and decode workloads based on real production traffic; tuning routing between in-house infrastructure and external providers based on cost, capacity and performance; working inside serving engines such as vLLM, SGLang and TensorRT-LLM, "going below the framework when needed"; and building profiling systems that show where time, memory and compute are going.

In plain English, a typical week might include:

• Making responses faster. Cutting the time to first token and the tail latency users feel when a system is busy.
• Making each answer cheaper. Squeezing more requests out of every GPU through smarter batching, caching and memory management.
• Shrinking models without breaking them. Quantising weights to lower precision while proving quality hasn't dropped.
• Choosing where requests run. Routing traffic between your own hardware and external providers depending on cost and capacity.
• Profiling everything. Finding the bottleneck, whether it's memory bandwidth, a slow kernel or a badly tuned scheduler.
• Keeping it reliable. All of the above, without the service falling over on launch day.

It's different from MLOps, which I covered in What an MLOps Engineer Does. MLOps engineers own the whole model lifecycle: pipelines, deployment, monitoring and retraining. Inference engineers go deep on one part of it, the serving layer, and obsess over speed and cost per request. In smaller companies one person might do both. At the frontier labs they're separate teams.

Why this job is booming

The economics explain it. Level's October analysis of inference engineer jobs cites Vercel AI gateway data putting cost per token at 43.3% of its April 2026 level, while total spend keeps rising. Prices per token keep falling, usage keeps exploding, and somebody has to make the maths work. Level is careful to say its data doesn't prove price cuts create jobs, and I agree that's the right caution. But every company selling AI at a falling price has a very strong reason to hire people who can serve it more efficiently.

The demand shows up across the market. Data Pay Index counted 513 open data and AI postings at 303 companies mentioning vLLM or inference serving as of 31 October, 3.9% of all the data and AI postings it tracks. Most of those (271) sit inside machine learning engineer roles, followed by AI engineer roles (161). So, like MLOps, plenty of inference jobs aren't titled "inference" at all.

The skills and tools that matter

Prof Rod analysed 55 inference engineer postings at 18 labs and found these skills asked for most often: inference and serving (100%), how language models work (67%), production systems (67%), Python (64%), GPU kernels (56%), and C++, Rust or Go (56%).

Level's tool count across 328 postings tells a similar story: PyTorch appeared in 44%, vLLM in 33%, Kubernetes in 16% and CUDA in 13%. Among postings with "inference" in the title, vLLM rose to 44%.

So the core toolkit looks like this:

• Python, plus a systems language. Adaption asks for strong Python and proficiency in C++, Rust or another systems language.
• Serving engines. vLLM, SGLang and TensorRT-LLM come up again and again.
• GPU fundamentals. CUDA, mixed precision, memory layout, kernels and quantisation. Anthropic's Staff+ Inference Velocity posting, quoted by Prof Rod, asks for "real depth in at least one accelerator ecosystem (CUDA/GPU, TPU, or Trainium/AWS Neuron) and genuine appetite to learn the others."
• How transformers work. You don't need to invent new architectures, but you must understand prefill, decode, attention and the KV cache well enough to optimise them.
• Production engineering. Kubernetes, observability, load testing and on-call.

AI assistants are useful here too. Tools such as Claude, OpenAI's models or Grok Bot are all reasonable starting points for drafting benchmarking scripts, reading unfamiliar kernel code or explaining a profiler trace. Just verify everything. In this job, a confident wrong answer about memory layout costs real money.

What inference engineers earn

As always, salary sources measure different things, so here are several real published figures side by side. All are US dollars, US-focused, and exclude equity unless stated.

• Level (26 October 2026): of 328 postings matching "inference engineer," 178 listed USD pay. The median listed minimum was $189,000 and the median listed maximum $277,000. For the 142 postings with "inference" in the title, the median range was $190,000 to $287,000.
• Data Pay Index (US postings seen since 18 October): a $225K median across 221 postings with pay. By level, senior $211K and staff/lead $245K.
• Metix AI Platform (21 September 2026): of 386 US postings with inference or model serving in the title, only 61 (16%) stated a salary. Half of those started between $150K and $200K, and 10 started at $250K or more.
• Prof Rod, frontier labs only: a median base of $325K across 33 US postings with a pay band, with the middle half running from $270K to $403K. Individual bands it lists include Cohere's Staff Software Engineer, Inference Infrastructure at $190K to $325K, Cartesia's Inference Engineer at $180K to $250K, and Anthropic's Staff + Senior Software Engineer, Inference Deployment at $320K to $485K.

My read: across the broader market, inference roles cluster around $190K to $280K advertised base. At frontier labs, they go well above that. Either way, it's among the best-paid engineering work in AI, and a notable premium on MLOps, which I found clustering around $180K to $205K in live postings last week.

One caution: most postings don't publish pay at all (only 16% in Metix's sample), and companies that publish may differ from those that don't. Treat these as signals, not offers.

Who's hiring

Prof Rod's frontier lab list is led by Together AI (12 postings), Anthropic (11) and OpenAI (8), followed by Cohere, Google DeepMind, Mistral AI, Thinking Machines Lab and others. On Level's broader count, NVIDIA had the most matches (43), then Amazon (36) and Capital One (24), with Anthropic, Together AI and Cerebras close behind. Capital One in that list is a great reminder of what I wrote in Banks and Hospitals Are Outbidding Big Tech for AI Engineers: serious AI infrastructure jobs are spreading well beyond the labs.

Remote is possible but not the norm. Level found 32% of postings allowed remote work, and 67% were in the United States.

How to get in

The honest truth: this is not an entry-level role. Level tagged 61% of postings as senior and another 17% as lead, with internships and junior roles only 5%. Prof Rod found a median of five years' experience among postings that stated a minimum. Adaption asks for five-plus years with "measurable improvements in cost or latency."

The good news is the doors are wider than you might think. Prof Rod found 80% of postings stated no degree requirement, and none required a PhD outright. Most people arrive one of three ways:

1. Backend or systems engineering. You already know performance, concurrency and production. Add GPU fundamentals and a serving engine.
2. ML engineering or MLOps. You know models and deployment. Go deeper on kernels, quantisation and profiling.
3. HPC or graphics programming. If you've written CUDA for simulations or games, you have the rarest part of the skill set already.

Then build proof. Take an open-weight model, serve it with vLLM or SGLang on a rented GPU, and benchmark it properly: time to first token, throughput, tail latency, cost per million tokens. Then improve it with quantisation, batching settings or speculative decoding, and write up exactly what moved and why. A clear before-and-after with numbers is precisely what hiring managers want to see. If you can contribute a fix to an open-source serving engine, even better.

Certifications worth considering

Certifications won't replace measurable results, but one maps neatly to this work. The NVIDIA-Certified Professional: Generative AI LLMs exam is 120 minutes, 60 to 70 questions, costs $200 and is valid for two years. NVIDIA recommends two to three years of practical experience with large language models, and lists performance profiling and efficient coding in Python, plus C++ for optimisation, among the expected skills. Cloud ML certifications from AWS or Google Cloud, which I covered in the MLOps explainer, are useful complements if you'll be serving models on those platforms.

My take

I'm very bullish on inference engineering. The world is building AI data centres on a scale that's hard to picture, and every one of those GPUs needs to earn its keep. The difference between a well-tuned and a badly tuned serving stack can be a multiple of the hardware bill. That makes the inference engineer one of the highest-leverage people in any AI company.

It's also a role that gets more important as AI gets cheaper, not less. Falling prices per token mean more usage, more models, more hardware types and more pressure on cost. If you're a systems engineer who loves making things fast, this is your moment. Browse the latest AI infrastructure and automation jobs and see what's open today.

Sources

• Prof Rod: Inference Engineer jobs (October 2026)
• Adaption: Inference Engineer job posting
• Level: Inference engineer jobs as LLM prices fall (26 Oct 2026)
• Data Pay Index: vLLM / inference serving jobs, demand and pay (October 2026)
• Metix AI Platform: US inference pay (21 Sep 2026)
• NVIDIA-Certified Professional: Generative AI LLMs