The private docs weren't in the prompt. Retrieval was the work.

GPT-4 could reason, explain, and synthesize. It still couldn't answer a specific Archibus workflow question without the private product context.

Archibus has configuration guides, help content, schema references, and workflow documentation from many releases. GPT-4 had no access to those materials during the conversation. It needed the right source material with each question. In 2023, making that reliable still took a lot of work.

Trying to pass the full corpus into one chat completion call hit the token limit fast. Even the wider context options at the time needed carefully selected text. I couldn't brute-force the whole corpus into one prompt. Before choosing an embedding store, I had to decide what I was storing and what shape it should have.

The rewrite pass

Before choosing a vector store, I reshaped the source material for retrieval.

Archibus help documents are written for someone looking at the product and stuck on a step. Useful there. Not shaped for cosine similarity. A structure that helps someone move through a UI screen may not help a retrieval system find the right chunk when the question uses different words than the docs.

My first experiment rewrote each section. Count the tokens, ask the model to make the section easier to search, then store the result. I wanted the stored text to sound more like the questions people would ask than the original document structure did.

encoding = tiktoken.encoding_for_model("gpt-3.5-turbo-16k")
tokens = len(encoding.encode(source_text))

response = openai.ChatCompletion.create(
    model="gpt-3.5-turbo-16k",
    messages=[
        {"role": "system", "content": "Rewrite these docs so they are easier to search."},
        {"role": "user", "content": source_text},
    ],
)

The FAQ-style rewrite wasn't always right. It was one way to make the text easier to retrieve, but it forced a better question: what did "retrievable" mean for this material?

The retrieval apparatus

In 2023, LangChain was the obvious place to start. It was everywhere. Its document loaders, text splitters, and vector store wrappers could turn a raw file into stored, embedded chunks in a handful of lines.

splitter = RecursiveCharacterTextSplitter(chunk_size=1000, chunk_overlap=200)
docs = splitter.split_documents(loader.load())

embeddings = SentenceTransformerEmbeddings(model_name="all-MiniLM-L6-v2")
store = SupabaseVectorStore.from_documents(
    client=supabase,
    documents=docs,
    embedding=embeddings,
    table_name="documents",
)

It worked, then raised harder questions. How large should a chunk be? What happens when the document structure doesn't match the words people use? When does the embedding model matter, and when is it noise?

LangChain helped at first, then its moving abstractions got in the way. Code that worked at tutorial scale broke with real material. I eventually used the embedding APIs and vector store clients directly. Less magic meant fewer surprises once the scripts stopped being examples.

The vector store parade

Over the next stretch, the bench tried a lot of stores: Supabase/pgvector, FAISS, in-memory DocArray, ChromaDB, Weaviate hybrid search, Milvus, Vertex AI RAG Engine, Haystack with its own pipeline abstractions, and LlamaIndex. Later, I added sparse and hybrid retrieval through a BGE-M3 model backed by Ollama.

Each store exposed a different tradeoff. Weaviate hybrid search showed that BM25 and vector similarity often disagree in useful ways. Milvus raised different deployment questions than Chroma. Vertex AI RAG forced the question of how much the cloud should own. I also moved from text-embedding-ada-002 to newer OpenAI generations, Google's text-embedding-004 when Gemini entered the stack, and local all-MiniLM-L6-v2 when remote API calls were too slow. None of those switches mattered as much as source prep.

Source prep is most of the work

The source material kept making the bigger difference.

The Archibus knowledge base spans help files, schema references, workflow documentation, and implementation guides from many release cycles. It wasn't structured for ingestion, deduplicated, or consistently formatted. Most of the scripts I kept from that period weren't about embeddings. They flattened directory trees into one corpus, removed noisy or irrelevant files, deduplicated chunks, normalized sources, merged documents, and used tree-sitter to split code differently from prose.

Changing the embedding model helped less than fixing the source. Clean, well-cut chunks reliably beat a better model fed bad input.

The model was not the bottleneck. The corpus was.

A chat interface is a product question

Once I trusted retrieval enough, I put a chat interface on it. The command-line test had become something a person could actually use.

if prompt := st.chat_input("What would you like to know about Archibus?"):
    st.session_state.messages.append({"role": "user", "content": prompt})
    response = query_engine.query(prompt)
    st.session_state.messages.append({"role": "assistant", "content": str(response)})

Retrieval alone couldn't answer what the user should see when a search failed. It couldn't decide how chat history should change the question, or whether to search all documentation or a narrower context. The script worked. Those choices still needed a product decision.

When retrieval needs memory and tools

Next I gave the retrieval system memory and tools.

worker = Worker(
    role="Customer Service Agent",
    instructions="Use the knowledge base and tools before answering.",
    memory=Memory(storage=chroma_storage),
    actions=[DuckDuckGoSearch, WebBaseContextTool, ReadFileAction, WriteFileAction],
)

admin.assign_workers([worker])
admin.run("Answer the user's request with the available tools.")

The system could now reason, remember, and act instead of only looking something up. That was much closer to an agent, with the same design questions as chat at a larger scope.

What the bench produced

Those scripts were bench work for learning. They were never meant to ship.

What came out of the bench wasn't a favorite vector store. On this bench, cleaning the corpus and fixing chunk boundaries changed the answers more than switching embedding models did. Chat and retrieval were separate design problems that shared a pipeline. Agent loops extended retrieval instead of replacing it.

ArchiBot's later knowledge systems became an MCP server and shaped what went into the product. They weren't built from scratch. They grew from the bench work: cleaning, reshaping, chunking, embedding, storing, retrieving, and asking a better question.

Reliable retrieval started with cleaning and reshaping the material.