Deterministic AI: Why I Replaced Vector DBs with Git & Markdown

Over the past eighteen months, the consensus orthodoxy for constructing persistent memory for artificial intelligence agents has been uniform and unyielding: “Chunk your enterprise documents, compute high-dimensional vector embeddings, and retrieve context via cosine similarity inside a vector database.” In theory, Retrieval-Augmented Generation (RAG) promises magical, unbounded semantic recall. In production across our financial technology platforms at Homeville Group, vector search broke down in frustrating, predictable, and costly ways.

Earlier, I documented our initial journey in The Git-Backed Brain and How I Built a Shared Brain for My AI Agents. Today, after processing millions of production tokens across hundreds of autonomous agent sessions, I want to explain why we completely tore out our vector database infrastructure in favor of something aggressively simple: deterministic Markdown files version-controlled inside a Git repository.

The Structural Flaws of Vector Search in Agentic Workflows

Vector databases excel at fuzzy semantic matching—identifying similar customer support inquiries or searching through vast, uncataloged libraries of scanned PDF whitepapers. But autonomous coding and architectural agents do not need “fuzzy similarity.” They require exact, deterministic context.

Here is why vector search continually failed our autonomous workflows:

1. Chunking Destroys Document Hierarchy

To feed text into an embedding model, you must slice documents into arbitrary chunks (typically 256 to 1024 tokens). When an agent requests context on a co-lending escrow audit pipeline, vector search might retrieve Chunk 4 of an Architecture Decision Record (ADR). But Chunk 4 is meaningless without the architectural invariants established in Chunk 1. Stripped of structural hierarchy, the agent acts on incomplete premises.¹

2. Cosine Similarity is Inherently Non-Deterministic

Similarity scores fluctuate wildly based on minor prompt phrasing. Prompting an agent with “How do we handle RBI co-lending reports?” might retrieve ADR-012.md with an 0.82 score today, but surface an obsolete meeting transcript with an 0.84 score tomorrow because of a single synonym. In mission-critical software architecture, retrieval unpredictability directly induces hallucinated code patterns.

3. You Cannot git bisect a Vector Index

When an agent produces a non-compliant database query because it ingested outdated policy guidance, how do you debug the vector database? You cannot run git log on a high-dimensional vector space. You cannot inspect diffs between embedding updates, nor can you easily roll back an index to a known good state at 14:32 UTC last Thursday.

The Deterministic Alternative: File Systems + Git

Instead of converting enterprise documentation into opaque mathematical vectors, we organized our organizational intelligence into an explicit Markdown taxonomy inside a private Git repository:

manthan/
├── context/
│   ├── working-memory-2026-08.md    # Active monthly focus & sprint objectives
│   └── active-sprints.md           # Current sprint state & operational blockers
├── decisions/
│   ├── ADR-001-colending-escrow.md  # Immutable architecture decision records
│   └── ADR-002-rbi-compliance.md
├── systems/
│   ├── banking-integrations.md     # Component specs & data schemas
│   └── audit-telemetry.md
└── logs/
    └── sessions/                   # Automated JSON telemetry per agent session

How Agents Access Context Deterministically

Rather than executing probabilistic RAG queries over a vector index, our agents leverage native file access and Model Context Protocol (MCP) servers to navigate explicitly:

• Bootstrapping via Index Files: Every agent session starts by inspecting context/working-memory-YYYY-MM.md, which links explicitly to active technical specifications.
• Deterministic URI Referencing: When working on the co-lending engine, the agent system prompt instructs it to directly open decisions/ADR-001-colending-escrow.md.
• Zero Latency & Zero Inference Cost: There is no external embedding API call, no vector calculation, and zero mathematical possibility of pulling irrelevant documents.

Side-by-Side Architectural Comparison

When Does Vector Search Still Make Sense?

Vector databases are not without purpose. If you are indexing 50,000 unstructured customer support emails, querying thousands of disparate legal contracts with arbitrary naming conventions, or running fuzzy semantic search across public e-commerce catalogs, RAG remains indispensable.

But for agentic software development, system design, and regulated engineering workflows, you do not have millions of unstructured documents. You have a few dozen critical specifications, architecture decision records, and active codebases. Inserting a fragile, probabilistic vector pipeline between your autonomous agent and your technical documentation introduces failure points where you desperately need determinism.

Conclusion: Less Magic, Better Software

In the current artificial intelligence cycle, it is remarkably easy to mistake infrastructure complexity for sophistication. Replacing vector databases with Git and Markdown eliminated operational overhead, lowered agent turnaround latency, and excised an entire taxonomy of silent retrieval bugs from our development cycle.

Before reaching for a vector database for your agent’s memory layer, ask yourself: Could this simply be a clean directory tree in Git? More often than not, the simplest tool is the most durable.

---