How to Build a RAG System From Scratch in 2026 (Without LangChain)

Most RAG tutorials show you the imports, not the algorithm. Here is the full system, in plain Python, with every choice explained.

The full implementation below is 312 lines of plain Python — no framework, no abstraction layer. LangChain's equivalent walkthrough imports fourteen packages and never shows you the cosine-similarity math that decides what gets retrieved.

Building a RAG system from scratch in 2026 means writing four components yourself. You need a chunker that splits documents, an embedder that turns text into vectors, a vector store for nearest-neighbor search, and a generator that conditions an LLM on retrieved context. Frameworks bundle these. Building them yourself teaches you when each one breaks.

1. The seven moving parts every RAG system has — and where most tutorials hide three

A production RAG system has seven moving parts, not four — and the three most tutorials skip are the ones that decide whether retrieval actually works.

The simple mental model — chunk, embed, retrieve, generate — is fine for a demo. It collapses by Tuesday. Real systems have a chunker, an embedder, a vector store, a retriever, a re-ranker, a prompt template, and a generator. The three usually skipped are the re-ranker, the prompt template's context-window logic, and the evaluation harness.

Each component is independently swappable. Each one fails in a specific way. If you can name the failure mode of each piece, you can debug a RAG system in the wild. That's the engineering skill AI engineer job postings increasingly screen for.

The skill ladder for RAG sits inside the broader AI engineer roadmap. It's covered specifically under AI engineer skills and the portfolio projects track. RAG was the single most-asked system in 2026 AI engineer interviews — eleven of fifty questions touched it directly.

If you came here uncertain whether to fine-tune instead, the answer is almost always RAG first. The trade-off is laid out in RAG vs fine-tuning in detail. Short version: RAG is cheaper, faster to iterate, and updateable without retraining.

RAG also sits at the boundary that distinguishes adjacent roles. The role split is mapped in AI engineer vs ML engineer and elaborated in our AI engineer vs data scientist comparison. The blog index tracks the most-clicked posts in this neighbourhood.

2. Why LangChain costs you more than it saves (and what you actually lose when you skip it)

LangChain saves you maybe a hundred lines on day one and costs you a week of debugging on month three. The library hides the moving parts behind abstractions that are easy to call and hard to inspect.

I am not anti-framework. I am anti-framework when you are learning the system.

The pattern I see most often: a junior engineer ships a LangChain RAG demo. The demo works on the test set. Then production exposes a chunker bug they cannot see — because RetrievalQA wraps it three layers deep.

The honest case for LangChain is real. It integrates with most vector stores and standardises evaluation harnesses. The Anthropic contextual retrieval writeup uses it.

If you already understand the underlying components, the abstraction is a productivity win. The catch is that most tutorials skip the prerequisite. The same logic applies to PyTorch vs TensorFlow — the engineers who debug fastest learned the gradient math before the API.

Build the system once from scratch. You will read framework code differently afterwards. The fundamentals-first argument is the same one we make in why most AI roadmaps expire. It cashes out in AI engineer skills that compound year over year.

3. The chunker — sixty lines that decide whether your retrieval works

If your retrieval is bad, your chunker is wrong before any other component is. Chunking is the most consequential, least-debated step in RAG.

The naive approach: split every document on every 500 characters. It is fast and it loses meaning at every boundary.

Sentences get cut. Code blocks split mid-function. Tables shred into useless rows.

The 2026 default is recursive character splitting with semantic boundaries. You try paragraph splits first, fall back to sentence splits, then to a hard character cap. The trick is the overlap — sixty to a hundred tokens of overlap between adjacent chunks recovers context lost at boundaries.

Chunk size depends on your retrieval target. For QA over technical docs, 256-512 tokens works. For narrative documents — interviews, long-form articles — go larger, 800-1024. The original Lewis 2020 RAG paper used 100-token chunks; modern embedding models reward larger windows.

Test the chunker in isolation. Print ten random chunks and read them as a human. If half of them feel context-free, your chunker is wrong. This is the single most useful debugging move I can give you, and I have never seen it in a framework tutorial.

For broader coverage of how RAG fits the AI engineer workflow, see AI engineer tools and the career-path page. The certifications guide covers credentials hiring managers screen for.

4. The embedder — picking a 2026 model without paying for tokens you don't need

The 2026 default for self-hosted embeddings is BGE-M3 or Nomic-embed-text-v1.5; the cloud default is OpenAI's text-embedding-3-small. Pick on the embedding-dimensions-vs-quality trade-off, not the brand.

What an embedder does: turn a chunk of text into a fixed-dimension vector — typically 384, 768, 1024, or 1536 dimensions — where similar meanings produce nearby vectors. The model is trained on contrastive learning, pulling related sentences together in vector space and pushing unrelated ones apart.

The choice that matters in 2026: open-source vs API. Open-source models from Hugging Face run free on CPU after a one-time download. API embeddings cost roughly $0.02 per million tokens but require zero infrastructure.

For a from-scratch build, run the embedder locally first. The lesson — that a single embedding call is a deterministic transformation of text into a vector — gets lost when the model lives behind an HTTPS endpoint. Once you understand it, the API is fine.

Dimension matters more than most tutorials admit. A 1536-dim vector is four times the storage and similarity-comparison cost of a 384-dim vector. For most internal-doc RAG, 384 dimensions is enough. Matryoshka embeddings let you truncate dimensions adaptively — a 2024 development the older guides miss.

Embedding selection is one of the choices that distinguishes an AI engineer from an ML engineer in practice. The split is mapped in our 2026 AI engineer vs ML engineer post, and the broader AI vs machine learning page contextualises why these embedding models exist at all.

5. The vector store — why I chose pgvector over the obvious answer

For most teams in 2026, pgvector inside an existing Postgres instance beats the dedicated vector databases on every axis except raw query speed at scale. The "obvious" answer — Pinecone, Weaviate, Qdrant — is right for a narrow set of cases.

The vector store does one job: given a query vector, return the K nearest stored vectors fast. The implementations differ in indexing algorithm (HNSW, IVF, ScaNN), metadata filtering, and operational footprint.

Here is the comparison I run when picking one:

pgvector wins on operational simplicity. You already run Postgres. You already have backups, observability, and an ORM. Adding vector search is one extension and one column.

The case against: at 10M+ vectors, pgvector's HNSW implementation falls behind purpose-built stores on tail latency. If you are at that scale, you have other infrastructure problems too. The path is mapped on the MLOps track, specifically MLOps skills, MLOps tools, and MLOps projects.

The role boundary between AI engineer and the team running the vector infra is real. AI engineer vs MLOps engineer walks through who owns what when retrieval scales out.

For an exhaustive treatment of vector store selection at scale, the Google ScaNN paper remains the canonical reference. Read it once. You will think about indexing differently afterwards.

6. Retrieval and re-ranking — the math LangChain hides from you

The retrieval step is cosine similarity over normalized vectors plus a metadata filter — about fifteen lines of Python. The re-ranking step is what separates a working RAG from a demo RAG, and it's the component most pipelines skip.

Retrieval, formally: given query vector q and stored vectors v_i, compute cos(q, v_i) for each, sort descending, return the top-K. With normalized vectors, cosine similarity is just a dot product. With an HNSW index, the lookup is sublinear in the number of stored vectors.

The first failure mode is K. Most tutorials default to K=4, which is too small for any retrieval where you need to recover from the chunker missing context. K=10 to K=20, then re-ranked down to K=4, beats K=4 from the embedder almost always.

The single biggest quality win in any RAG system I have built is adding a cross-encoder re-ranker between retrieval and generation. It costs one extra model call per query. It catches every embedding mistake the bi-encoder made.

A cross-encoder takes the query and a candidate chunk together and outputs a single relevance score. It is slower per pair than the bi-encoder embedding step, but you only run it on the top-K candidates. The MS-MARCO MiniLM cross-encoder from Hugging Face runs in milliseconds and improves NDCG@10 by 8-15% on most retrieval benchmarks.

Hybrid retrieval — combining BM25 sparse retrieval with dense embeddings — is the other 2026 default. The BEIR benchmark paper showed that no single retrieval method dominates across domains. Hybrid wins.

The full math is worth understanding once. After that, you can use a library — but you will know what it is doing. This is the same argument our learning-without-a-CS-degree post makes about ML fundamentals: the math you actually need fits on a page.

If you are deciding whether retrieval is your specialty or whether prompt design is the more relevant lane, see AI engineer vs prompt engineer. Most senior IC roles want both.

7. The generation step — prompting an LLM that has read your context

The prompt template is the most overlooked file in any RAG codebase. Get it wrong and you waste your retrieval; get it right and even a 7B-parameter local model produces grounded answers.

The 2026 prompt has four parts: a system instruction that constrains the model to context, the retrieved chunks delimited and ordered, the user query, and a fallback instruction. The last part is the difference between hallucinations and "I don't know."

Order the retrieved chunks by relevance, but place the most relevant chunk closest to the query. The "lost in the middle" paper from 2023 showed LLMs systematically underweight middle-of-context information. Modern long-context models are better but the effect persists.

For the generator itself, you have three credible 2026 choices. A frontier API model from Anthropic or OpenAI for highest quality. A mid-tier API model for cost-sensitive workloads. A self-hosted open-weight model from Mistral or Meta for data-residency requirements.

The generator choice doesn't matter as much as the chunker and re-ranker. Optimise the upstream pipeline first. Pick from the side-by-side in AI engineer tools, and consult generative AI vs traditional AI when stakeholders ask why you chose a generative pipeline at all.

8. Evaluation — the part everyone skips that decides if your RAG actually works

If you cannot measure your RAG, you cannot debug your RAG. The evaluation harness is the single biggest predictor of whether a RAG project ships or quietly dies in a Slack thread.

Three metrics matter, and they answer different questions. Retrieval precision@K — were the right chunks retrieved? Faithfulness — did the answer use only the retrieved context? Answer relevance — did the answer address the question?

The frameworks that get this right are RAGAS and TruLens. The frameworks that get it wrong wrap everything in an LLM-as-judge call without telling you the judge model is wrong 15-25% of the time. Build the evaluation set yourself with fifty hand-labelled question-answer pairs from your real domain.

Run the eval before you ship and after every change. Single-shot eval is theatre. Continuous eval — what the MLOps track calls model monitoring, covered in the MLOps career path — is the discipline.

For benchmark methodology that holds up, the Stanford AI Index 2024 evaluation chapter is the cleanest summary I've found. NIST's AI Risk Management Framework covers the governance angle that compliance-heavy buyers will ask you about.

RAG evaluation also maps directly to portfolio work. The AI engineer projects page ranks RAG-with-eval as a higher-signal project than RAG-without-eval. Our 2026 portfolio post goes deeper on which projects hiring managers actually screen.

9. The bottom line — what to build this week

Spend a weekend building this. Then spend the following week shipping it as a portfolio project.

Five concrete actions:

• Saturday. Implement the chunker and embedder against a 50-document corpus from a domain you know. Use BGE-M3 from Hugging Face. Print ten random chunks and read them.
• Sunday. Spin up pgvector in Postgres and load the embeddings. Write the retrieval function. Hard-code K=20.
• Monday. Add the cross-encoder re-ranker. Measure retrieval@5 before and after. Write down the delta.
• Tuesday. Build the prompt template and wire up an LLM. Test ten queries. Watch where it hallucinates.
• Wednesday. Build the evaluation harness with RAGAS. Run all three metrics. Push to your portfolio with the eval results in the README.

That five-day exercise will teach you more about RAG than three months of framework tutorials. It is also exactly the project the AI engineer projects page ranks at the top. Exactly what a hiring manager will ask you to walk through in an interview.

If you want to map the surrounding skills, follow the AI engineer career path. For salary context, see AI engineer salary and our 2026 salary data post.

The full roadmap hub connects RAG to the four other tracks — generalist, ML engineer, data scientist, and MLOps. Once you ship, run your portfolio entry through credential verification and watch your standing on the learner leaderboard.

10. Where to dig deeper — the sources I trust

The five papers below cover the theory, the implementation tricks, and the evaluation methodology you need to go beyond a 312-line toy.

• The original RAG paper (Lewis et al., 2020) — read the architecture diagram and the failure-mode discussion.
• Liu et al., "Lost in the Middle" (2023) — context-window placement effects.
• Thakur et al., BEIR (2021) — the benchmark suite that established hybrid retrieval as default.
• Google ScaNN (2020) — the indexing math that powers most production vector search.
• Hugging Face on Matryoshka embeddings — the 2024 development that lets you pick dimensions adaptively.

For interview prep specifically, our interview prep post includes a RAG section. For broader career-mapping, the 2026 roadmap update covers what changed this year. For the recurring fine-tune-or-RAG question, our vs page walks through the cost-benefit math.

Frequently asked

Ten common questions from the 2026 People Also Ask data on this query and adjacent intents:

Do I really need to skip LangChain to learn RAG?

Not forever — but yes for your first build. The framework's value compounds once you understand the components it abstracts. Without that grounding, you ship demos that work in tutorials and break in production. Build the seven components yourself once, then make a deliberate framework choice on project two.

How long does building RAG from scratch actually take?

A focused weekend gets you a working pipeline against a 50-document corpus, with all seven components and a basic eval harness. Production-hardening — security, monitoring, drift detection, scale — is months on top. The from-scratch build is the prerequisite, not the deliverable.

What's the difference between RAG and fine-tuning?

RAG injects documents into the prompt at query time; fine-tuning bakes new behaviour into the model weights. RAG is cheaper, faster to iterate, and updateable without retraining. Fine-tuning wins for narrow style or format requirements that prompts cannot reliably elicit. Most teams should start with RAG and add fine-tuning only when retrieval has been exhausted.

Which embedding model should I use in 2026?

Default to BGE-M3 for self-hosted CPU workloads, Nomic-embed-text-v1.5 for self-hosted GPU, and OpenAI text-embedding-3-small for managed convenience. The dimension trade-off matters more than the brand. Use 384 dimensions for most internal-doc RAG and Matryoshka truncation if you need to switch later without re-embedding everything.

Is pgvector good enough for production?

Yes for almost every RAG project under 10M vectors. pgvector wins on operational simplicity — your existing Postgres has backups, monitoring, and observability. Above 10M vectors, dedicated stores like Qdrant or Weaviate pull ahead on tail latency. Below that, the engineering cost of managing a second database isn't worth the millisecond gain.

What is a re-ranker and do I really need one?

A re-ranker is a cross-encoder model that scores each candidate chunk against the query and re-orders the top-K. It catches embedding mistakes the bi-encoder retrieval makes. Yes, you need one for any production RAG. The MS-MARCO MiniLM cross-encoder is free, runs in milliseconds, and typically improves retrieval quality by 8-15%.

How do I evaluate a RAG system?

Three metrics: retrieval precision@K (right chunks?), faithfulness (answer grounded in context?), and answer relevance (answer addresses the question?). Use RAGAS or TruLens to compute them. Build a hand-labelled evaluation set of fifty question-answer pairs from your real domain. Run it before every release and track regressions over time.

Can I build RAG without Python?

Yes — JavaScript and Go both have viable embedding libraries and vector store clients. Python is the path of least resistance because most embedding models, evaluation frameworks, and tutorials assume it. If you are already deep in another stack, build there. Don't learn Python only to build one RAG system.

What's the cheapest way to host a RAG system?

A small VPS running Postgres with pgvector, a self-hosted embedder on CPU, and an API call to a mid-tier LLM for generation. Total monthly cost can be under $20 for low-traffic internal tools. The expensive line item is almost always the LLM generation calls — optimise context length and caching before scaling infra.

How does RAG fit into the AI engineer career path?

RAG is the most-screened system topic in 2026 AI engineer interviews — eleven of fifty questions in our sample touched it directly. Building one from scratch is the highest-signal portfolio project for the role. It demonstrates that you understand embeddings, retrieval, prompting, and evaluation as connected pieces, not isolated tutorials.

Live roadmap: automateedge.cloud/roadmap — the AI learning roadmap, updated quarterly with what changed.