Enterprise Knowledge Base Architecture: Best Practices

Most enterprise knowledge base failures are not model failures. They are chunking failures, retrieval failures, or access-control failures that make the model appear to underperform.

Chunking Strategy Is Everything

The default approach — split by 512 tokens with 50-token overlap — performs poorly on structured enterprise documents. Better approaches:

Semantic chunking: split at paragraph and section boundaries, not fixed token counts. A 200-token section that covers one complete concept retrieves better than a 512-token window that spans three concepts.

Hierarchical chunking: index both summaries and full sections. Retrieve summaries for initial scoring, full sections for the final context window.

Embedding Model Selection

OpenAI's text-embedding-3-large outperforms smaller models for technical English content. For multilingual corpora, multilingual-e5-large is worth the extra latency. Never mix embedding models without re-indexing everything.

Access Control at Retrieval Time

For enterprise deployments, document-level access control must happen at the vector database level, not in post-processing. Filtering after retrieval can surface documents the model should not see even if the final response is withheld.

The correct pattern: store the document's access group as metadata, include it in the vector query filter. The document never enters the context window if the user lacks permission.

Hybrid Search Is Not Optional

Pure vector search misses exact-match queries. A user searching for "invoice #INV-2024-04892" needs BM25, not cosine similarity. Production knowledge bases need hybrid retrieval with configurable weighting.