Over the past year, the standard advice for building long-term memory for AI agents has been unanimous: "Chunk your documents, embed them into a vector database, and perform cosine similarity searches using RAG (Retrieval-Augmented Generation)."
In theory, vector databases promise magical, infinite semantic memory. In practice, when running multi-agent workflows across production engineering systems at Homeville Group, vector search continuously broke down in frustrating and predictable ways.
Earlier, I wrote about The Git-Backed Brain and How I Built a Shared Brain for My AI Agents. Today, after millions of tokens and hundreds of subagent sessions, I want to explain why we completely threw out our vector DB infrastructure in favor of something aggressively simple: deterministic Markdown files stored inside a Git repository.
The Flaws of Vector Search in Agentic Workflows
Vector databases excel at fuzzy semantic matching—like finding similar customer support tickets or querying unstructured PDF libraries. But autonomous coding and architectural agents don't need "fuzzy similarity." They need exact, deterministic context.
Here is why vector search failed our agent workflows:
1. Vector Chunking Destroys Context & Hierarchy
To feed text into an embedding model, you must split files into chunks (typically 256 to 1024 tokens). When an agent asks for context on a co-lending escrow audit pipeline, vector search might retrieve Chunk 4 of an Architecture Decision Record (ADR). However, Chunk 4 might depend entirely on a constraint defined in Chunk 1. Without the parent document structure, the agent acts on incomplete assumptions.
2. Cosine Similarity is Non-Deterministic
Similarity scores fluctuate based on slight variations in prompt phrasing. Prompting an agent with "How do we handle RBI co-lending reports?" might pull `ADR-012.md` with a 0.82 score today, but pull an outdated meeting summary with a 0.84 score tomorrow because of a minor wording tweak. In software architecture, retrieval unpredictability leads to hallucinated code patterns.
3. You Can't `git bisect` an Embedding Index
When an agent produces a bug because it used obsolete API guidelines, how do you debug the vector database? You can't run `git log` on a high-dimensional vector space. You can't inspect diffs between embedding updates, nor can you roll back a vector index to exact timestamps easily.
The Alternative: Deterministic File Systems + Git
Instead of converting text into mathematical vectors, we structured our agent memory as a simple Markdown taxonomy inside a Git repository:
manthan/
├── context/
│ ├── working-memory-2026-08.md # Active monthly focus & goals
│ └── active-sprints.md # Current sprint state & blockers
├── decisions/
│ ├── ADR-001-colending-escrow.md # Immutable architecture records
│ └── ADR-002-rbi-compliance.md
├── systems/
│ ├── banking-integrations.md # Component specs & data schemas
│ └── audit-telemetry.md
└── logs/
└── sessions/ # Automated JSON output per agent runHow Agents Access Context Deterministically
Rather than performing RAG queries over a vector store, agents use standardized tools (like native file readers or custom MCP servers) to fetch explicit files based on explicit indices:
- Index File Bootstrapping: Every agent session starts by reading a small, fixed index file (
context/working-memory-2026-08.md) that links to active specifications. - Explicit Document Paths: When working on the co-lending engine, the agent system prompt instructs it to directly read
decisions/ADR-001-colending-escrow.md. - Zero Inference Overhead: There is no embedding API call, no vector calculation, and zero chance of pulling unrelated snippets.
Comparing the Two Approaches
| Metric / Feature | Vector DB / RAG | Git + Structured Markdown |
|---|---|---|
| Retrieval Accuracy | Probabilistic (85-90%) | Deterministic (100%) |
| Infrastructure Cost | $50–$300/mo (Vector DB + Embeddings) | $0 (Existing Git repo) |
| Auditability & Diffs | Opaque index updates | Native `git log`, `git diff`, `git blame` |
| Multi-Agent Sync | Complex index synchronization | Standard `git pull` / `git push` |
When Does Vector Search Still Make Sense?
Vector databases aren't useless. If you are indexing 50,000 unstructured customer support emails, 100,000 legal contracts with arbitrary naming, or running semantic search over public product catalogs, RAG is indispensable.
But for **agentic software development, system design, and internal engineering workflows**, you don't have millions of unstructured documents. You have a few dozen critical specs, architecture decisions, and codebases. Placing a complex vector pipeline between your agent and your documentation introduces fragility where you need determinism.
Conclusion: Less Magic, Better Software
In AI engineering, it is easy to mistake infrastructure complexity for sophistication. Replacing vector databases with Git and Markdown stripped away operational overhead, reduced latency, and eliminated an entire class of retrieval bugs in our agents.
Before reaching for a vector database for your agent's memory, ask yourself: Could this just be a clean file tree in Git? More often than not, the simplest tool is still the best one.