centralmcp — RAG Architecture & Decision (2026-06-03)

Repo: https://github.com/secure-ssid/centralmcp

TL;DR decision

Current default backend = embedded, no Docker, no background services:

Redis Stack remains a documented, supported server option for anyone who wants it — but it is not the default for the cloned-and-run experience.

Why this and not Redis (reconciling the audit)

The audit recommended Redis Stack — correctly, for its scope: “two backends are running and the git history is mid-flip; converge on one with the least code change.” Redis is already wired in the working tree and holds both the docs and tool indexes.

But the project’s primary goal is “anyone can download the repo and run it,” with the portal as a consumer of the MCP. Against that goal, a Redis/Docker service is the exact friction we want to remove. LanceDB delivers the same capabilities the audit credited to Redis (hybrid BM25+vector, one store for docs+tools, metadata filtering) without a server, and uniquely allows shipping a prebuilt index file. fastembed removes the last service (Ollama).

Axis Redis Stack (audit pick) LanceDB + fastembed (current default)
Docker / services Redis container (or local install) + Ollama none (in-process)
“clone → run” UX install Docker, start Redis, ingest 40k docs uv sync → run (ship prebuilt index)
Hybrid (BM25+vector) yes (RediSearch) yes (native)
Reranking build it built-in (RRF default)
One store for docs + tools yes yes
Embeddings Ollama service fastembed in-process (same nomic model)
Migration effort none (already wired) one-time storage-layer rewrite + re-ingest

Net: Redis wins only on “no work today.” For a distributable tool, the one-time migration buys a dramatically simpler install for every future user.

Why retrieval quality goes up, not down

Deployment (embedded vs server) does not affect retrieval quality — the design does. The target design is strictly better than today’s vector-only Redis path:

  1. API/field/enum/endpoint questions → exact SQLite lookup, not vectors. A large slice of the corpus is OpenAPI specs (structured JSON). Embedding them is lossy; vector search returns fuzzy-similar prose instead of the authoritative enum/field list. A lookup_api(endpoint|schema|field) tool over the parsed specs is exact and lossless — and doubles as the API-correctness checker the audit needed.
  2. Prose questions → hybrid (BM25 + vector) + rerank. Today’s path is vector-only and misses exact identifiers (WPA3_SAE, endpoint paths, error codes). BM25 catches those; a cross-encoder rerank promotes the truly relevant chunk. (~+15–30% precision in practice; Anthropic measured up to 67% retrieval-failure reduction with contextual + hybrid + rerank.)
  3. Same embeddings, fixed prefixes. fastembed can run nomic-embed-text-v1.5 in-process — identical semantics to today — while fixing the missing search_query:/search_document: prefixes (see fix R3).
  4. Agentic safety net. search_docs/ask_docs are called by an LLM that can re-query when results are thin.

Backend-agnostic RAG fixes (from the audit — apply regardless of backend)

These are correctness/quality fixes; most are inherited or simplified by the LanceDB move.

Target module layout

pipeline/clients/
  lance_client.py     # open table, hybrid search(query, k, source_filter) -> hits for the default embedded path
  embed_client.py     # fastembed wrapper: embed_document(list) / embed_query(str); model nomic-embed-text-v1.5
  specs_index.py      # build + query SQLite over OpenAPI specs: get_endpoint / get_schema / get_field / get_enum
mcp_servers/
  rag.py              # search_docs (hybrid) + ask_docs (cited) + lookup_api (exact)  — all READ_ONLY
ingestion/
  ingest_docs.py      # chunk prose -> embed_document -> LanceDB ; parse specs -> SQLite ; emit prebuilt artifacts
data/                 # prebuilt, shippable: docs.lance/  +  specs.sqlite   (attach to GitHub Release)

No docker-compose.yml requirement for the default path. redis-stack stays available as an optional localhost-only “Server backend” for power users, with Docker named volumes so Redis/Ollama state does not clutter the repository. Compose is allowed to generate project-scoped container names, which avoids container-name collisions when multiple local checkouts are tested side by side.

Implemented migration sequence

This sequence is complete for the default local path. Redis remains optional; Qdrant is historical context only.

  1. Add deps: lancedb, fastembed; keep redis only for the optional server backend.
  2. embed_client.py (fastembed, nomic-embed-text-v1.5, embed_document/embed_query with prefixes — R3).
  3. lance_client.py: create a hybrid table (vector + FTS on text), search() with source filter + reranker (R5).
  4. specs_index.py: parse ingestion/sources/openapi_specs/*.json → SQLite (endpoints, schemas, fields tables) with FTS; query helpers (R2 resolved).
  5. Rewrite ingest_docs.py: prose → LanceDB (header-aware chunking R6, batched embeds R4); specs → SQLite. Emit data/docs.lance + data/specs.sqlite.
  6. rag.py: search_docs (hybrid), lookup_api (exact), ask_docs (cited R7). Point tool_router’s aruba_tools index at LanceDB too.
  7. Re-ingest once; run the eval harness (below) to confirm quality ≥ current.
  8. Keep generated data/* out of git; use release assets or local ingest for prebuilt indexes. Redis remains an optional backend.

Eval harness (measure “is it selecting correct info” — before/after)

A small, labeled question set + runner so the backend swap is proven, not asserted. Lives at tests/eval/.

Metrics: recall@5 (did an expected source appear in top-5), mrr (rank of first correct), api_exact (did lookup_api return the exact enum/field). Target: api-lookup api_exact = 100% (it’s structured), howto recall@5 ≥ today’s baseline.

Baseline measured 2026-06-03 (current Redis, vector-only, no prefixes, specs missing from index), and re-measured the same day after wiring lookup_api (SQLite specs index, H13 cosine fix + boost recalibration, query/document prefixes):

Metric Baseline (Redis, vector-only) After lookup_api (2026-06-03) Final: embedded LanceDB hybrid (2026-06-03) Target
howto_recall@5 (prose) 0.80 0.80 0.90 ≥ 0.80 ✅
api_exact (API lookups) 0.50 0.90 1.00 1.00 ✅
source_hit@5 (overall) 0.50 0.80 0.90 ≥ 0.75 ✅
mrr 0.339 0.679 0.90 ≥ 0.50 ✅
keyword_hit — 0.80 1.00 —

Final evaluated corpus (2026-06-03 full rebuild): 53,052 chunks / 7 ingested sources (Redis index had 40,900 and was missing aos_techdocs, openapi_specs, and most of techdocs_html) + 213-spec SQLite index + router tool index (currently 213 core tools / 312 read-only optional starters / 346 read-write optional starters). 18/20 eval questions hit at rank 1. Shippable artifacts: data/docs.lance (190 MB), data/specs.sqlite (18 MB), data/tools.lance (0.6 MB).

Tracked RAG refresh targets live in ingestion/source_manifest.json. The current manifest covers 9 rebuild sources: the evaluated default source set plus DevHub (https://devhub.arubanetworks.com) and Switching Feature Navigator (https://feature-navigator.arubanetworking.hpe.com/wired?mode=explore). Keep those inputs represented in local ingestion/sources/ before packaging public RAG indexes.

The API-lookup rows almost all missed the spec sources at baseline — direct empirical evidence of R2 (OpenAPI specs absent from the active index). howto retrieval is already decent, confirming the redesign’s value is concentrated in (a) structured API lookup and (b) hybrid+rerank for exact identifiers, not in replacing vector search wholesale. Re-run uv run --with pyyaml python tests/eval/run_eval.py after each change.

The remaining api_exact miss was mac-reg-update-url: the CNAC MAC-registration API was not in the 212 ingested config specs. Closed 2026-06-03: the Central NAC Service spec (25 paths, 60 schemas — cnac-mac-reg/visitor/named-mpsk/dpp/certificates/jobs) is not served by the internal-ui cnxconfig docs host, but the readme.io reference pages embed the full OAS document; ingestion/scrape_cnac_spec.py extracts it to cnac-client-registration.json (213 specs total). With it indexed, api_exact = 1.00 — all 10 api-lookup questions resolve through lookup_api with the correct spec at rank 1, no prose fallback needed.

Original open questions and current defaults

These questions were captured during the migration decision. The current repository defaults are embedded LanceDB + SQLite, fastembed, release/ignored data/* indexes, and Redis as an optional server backend.

  1. Embedding model: keep nomic-embed-text-v1.5 (via fastembed) for identical semantics, or move to bge-base-en-v1.5? (Both good; nomic = no quality change, just drops Ollama.)
  2. Ship prebuilt index in-repo or as a Release asset? Release asset keeps the repo small; in-repo is zero-step but bloats clones.
  3. Keep a Redis “server option” appendix, or go all-in embedded and remove Redis entirely? Current default: embedded LanceDB + SQLite, with Redis still available as an optional backend.