What does a RAG engineer do?

What the job involves day to day

Retrieval augmented generation (RAG) means grounding a model's answers in your own content. Microsoft's Azure AI Search docs list the hard parts: vague questions, content spread over many systems, limited token budgets, users who expect answers in seconds, and access control so people only see what they are allowed to see. A RAG engineer owns all five. In practice that means writing ingestion pipelines, designing the search index, tuning queries, and wiring the results into a prompt.

Databricks breaks the pipeline into stages: understanding the query, retrieving chunks, adding them to the prompt, choosing the model, then post-processing and guardrails. Most of a RAG engineer's time goes into the first two stages and into testing. When an answer is wrong, you trace it back. Was the right passage never indexed? Indexed but never retrieved? Retrieved but ranked too low? Or retrieved and then ignored by the model?

The core techniques employers expect you to know

Chunking comes first. A Databricks guide covers fixed-size chunking with overlap, semantic chunking that splits at sentence or section boundaries, recursive chunking and adaptive chunking. Each chunk is then embedded. Microsoft warns that you must embed queries with the same model and the same cleaning steps you used on the chunks. Index settings also matter. Microsoft describes exhaustive KNN, HNSW approximate search, and similarity metrics such as cosine and dot product.

Hybrid search runs keyword and vector queries side by side and merges the two lists, often with Reciprocal Rank Fusion. Reranking comes next. Microsoft's guidance is to pull a wide candidate set, around 20 to 50, merge it, rescore it with a cross-encoder, semantic ranker or language model, then send only the top 5 or 10 chunks to the model. Reranking makes answers more precise but adds latency, so you benchmark it against your test queries.

How it differs from general AI engineering

A general AI engineer might build agents, fine-tune models, write prompts and run deployment work all in the same month. A RAG engineer goes deep on information retrieval. The skills come from search engineering: relevance tuning, query rewriting and decomposition, metadata filters, field weighting and permission trimming. Model choice matters less than whether the right passage reaches the prompt.

Evaluation is the other big difference. Retrieval is scored with precision at K, recall at K and mean reciprocal rank, run against a set of test queries whose answers you know. Generation is scored separately. NVIDIA's docs on RAGAS metrics define context recall, context precision, faithfulness to the retrieved text, and response groundedness, each scored 0 to 1. Employers want engineers who track these numbers. A demo that merely looks good is not enough.

How to build a portfolio project that shows the skill

Pick a messy corpus you are allowed to use, such as public manuals, court opinions or a project's documentation, with PDFs and tables in it. Write 50 to 100 test questions and mark which passages answer each one. That labeled set is what separates your project from a tutorial clone.

Build a baseline with fixed-size chunks and vector search only. Measure recall at K and MRR. Then change one thing at a time: chunking strategy, hybrid search, a cross-encoder reranker, query rewriting. Record each result in a table, along with latency. Last, score answers for faithfulness and add a fallback for when retrieval comes back weak. Put the experiment log in your README. It shows how you think more clearly than the demo does.

Pay

USD a year, before tax. US figures. BLS OEWS May 2025, moved to 2026 with the Employment Cost Index. Software developers is the nearest series and covers the whole occupation, not RAG or AI specialists alone; no official series exists for RAG work. Job ad rows are midpoints of US pay ranges in 21 live AutomationRoles.ai AI developer ads from 11 employers on 5 November 2026. Those ads lean toward large AI and technology employers and show base pay only, without stock or bonus. Refer to the AI Developers salary guide for figures from live adverts.

What you need

• Python and APIsMost RAG code is Python that calls search, embedding and model APIs. You need to write clean pipeline code and test it.
• Information retrievalKnow keyword search, vector search, hybrid queries, rank fusion and reranking, and know when each one helps.
• Document processingParse PDFs, tables and images into text, then chunk it with sensible boundaries and metadata so it can be filtered.
• Evaluation disciplineBuild labeled test sets and track precision at K, recall at K, MRR, faithfulness and groundedness as you change the system.
• Production GenAI experienceAds for experienced engineers, such as a Databricks forward deployed AI engineer posting that is not open to entry-level applicants, ask for experience designing, building and deploying production GenAI and LLM applications.
• Security awarenessEnterprise RAG must enforce document permissions at query time so users only retrieve content they are cleared to see.

How to get there

1. Months 0-6Learn the software basicsGet comfortable with Python, REST APIs, SQL and Git. Build and deploy a small web service.
2. Months 4-8Learn search fundamentalsStudy keyword ranking, embeddings, approximate nearest neighbor indexes and similarity metrics. Index a small corpus both ways and compare the results.
3. Months 6-10Build a measured RAG projectWrite a labeled question set, build a baseline, then test chunking, hybrid search and reranking one change at a time.
4. Months 8-12Add grounding and guardrailsScore answers for faithfulness and groundedness, show citations, and add fallbacks for weak retrieval.
5. Year 1-3Ship RAG at workTake on retrieval work inside a software or data role. Production experience counts for more than side projects in senior ads.

Sources

1. Microsoft Learn: Retrieval-augmented generation (RAG) in Azure AI Search
2. Microsoft Azure Architecture Center: RAG information-retrieval phase
3. Databricks Community: Chunking strategies for RAG applications
4. Databricks docs: Improve RAG chain quality
5. NVIDIA docs: RAG evaluation metrics
6. Databricks careers: AI Engineer, Forward Deployed Engineering
7. U.S. Bureau of Labor Statistics: Software developers, Occupational Outlook Handbook