Karpathy's Alternative to RAG: Let LLMs Write and Maintain Their Own Knowledge Base

Andrej Karpathy has shared an architectural pattern for AI knowledge management that has reignited debate about whether vector database RAG — the dominant approach for giving AI systems access to external knowledge — is the right abstraction for most use cases.

Karpathy's proposal is conceptually simple but architecturally significant: instead of chunking documents into vector embeddings and retrieving them at query time, use an LLM to continuously maintain a structured markdown knowledge base that it actively writes, updates, and cross-references. At query time, the LLM reads from this structured wiki rather than retrieving from a vector index.

The Problem With RAG

RAG's limitations are well-documented in production deployments. Retrieval quality depends on embedding similarity, which does not always map to semantic relevance. Chunking decisions — how to split documents before embedding — are brittle and heavily influence answer quality in ways that are difficult to debug. Retrieved chunks often lack the context they need to be fully interpretable. And the entire system creates a pipeline with many failure modes that are opaque to the end user.

Mintlify, the developer documentation platform, recently published a detailed post-mortem on why they abandoned RAG in favor of a virtual filesystem approach. Karpathy's proposal is complementary but distinct: where Mintlify replaced retrieval with structured navigation, Karpathy replaces the static document store with a dynamically maintained knowledge graph expressed in markdown.

The Architecture

In Karpathy's model, the LLM takes on an active role that most current architectures reserve for humans: when new information arrives, the LLM decides where it belongs in the existing knowledge structure, writes it in a consistent format, cross-references it with related entries, and flags entries that need updating. The knowledge base is essentially an LLM-authored wiki that evolves continuously.

Key properties of the approach:

• Human-readable: The entire knowledge base can be read, audited, and corrected by humans — unlike a vector index, which is opaque.
• Self-healing: The LLM can identify contradictions and stale information during the maintenance pass and flag or correct them.
• Linting-compatible: Structured markdown can be linted for consistency, completeness, and format compliance — applying software engineering quality tools to knowledge management.
• Context-preserving: Entries include provenance, related concepts, and narrative context that chunked embeddings strip out.

The Trade-offs

The approach is not without costs. Maintaining the knowledge base requires continuous LLM inference, which adds latency and cost to the ingestion pipeline. The quality of the knowledge base depends heavily on the LLM's organizational judgment, which may not generalize well across all knowledge domains. And for very large knowledge bases, the markdown wiki model may scale less gracefully than vector indexes.

Karpathy's proposal is likely most compelling for knowledge bases that are medium-sized, frequently updated, and where answer quality matters more than retrieval throughput — precisely the characteristics of many enterprise internal knowledge applications.