This is a case study of an internal AI product that earned adoption instead of being mandated, written as a postmortem: what was instrumented, what the data killed, and how a side project became infrastructure. I built it solo, zero to production.
The problem
Product knowledge at a fast-moving company lives in too many places at once: Linear, Notion, GitBook, the source code, Slack threads, and the heads of whoever shipped the feature. New hires took weeks to get productive, and PMs and support burned hours re-answering the same deep product questions. I built a full internal company-brain agent, a Glean for Stellic, to close that gap. Largely built in Claude Code, the first thing I open every morning.
The adoption curve
Internal tools are where AI adoption claims go to die, so the honest version is the curve, not the peak. Launch week: 16 daily users, mostly people who sat near me. There was no mandate and no rollout program at any point. Usage grew one proof at a time, typically when someone watched a colleague get a correct, sourced answer to a question that would otherwise have interrupted an engineer.
Today it is used by 60% of the company daily, measured in PostHog, not by survey, and it has produced 20,000+ agent responses in its first six months. The adoption metric I trust most is behavioral: new hires now answer expert-level product questions in their first week, because the cost of asking dropped to zero.
How it's built
The unglamorous parts are what made it fast, cheap, and trustworthy enough for daily use. Most of the design applies techniques from the current agent-engineering literature.
- A thin connector layer. Each source (Linear, Notion, GitBook, source code, Slack, the warehouses) is a small connector behind a shared interface, so adding a new source is a day of work, not a project. The connectors double as the permission layer: retrieval runs against what the asking user is already allowed to see, mirroring each source system's own access controls, so the agent can never become a side door around permissions. The tool surface follows the design principles in Anthropic's writing effective tools for agents: few tools, sharply defined, returning high-signal context instead of raw dumps.
- Context management. The agent keeps lightweight identifiers (ticket IDs, file paths, doc slugs) and loads content just-in-time through tool calls instead of pre-stuffing the window, compacts long sessions, and takes structured notes across turns. This is the playbook Anthropic describes in effective context engineering for AI agents, and it is why answer quality holds up on long debugging threads instead of rotting as context fills.
- Cache discipline, to a 90% hit rate. Prompts are assembled stable-prefix-first: system prompt, tool definitions, and connector schemas up front, volatile retrieval last. That ordering keeps prompt caching hitting on roughly 90% of requests, which Anthropic prices at up to a 90% input-cost reduction and better than 2x latency. Retrieval itself runs on the contextual embeddings plus contextual BM25 recipe from Anthropic's contextual retrieval work, with chunk contexts generated once per document against the cached corpus. Speed and accuracy stopped trading off against each other.
- Two-tier memory. The agent maintains institutional memory (facts learned from one person that are valuable to everyone, like why a feature behaves differently for one partner configuration) and personal memory (each user's team, role, and recurring questions). The extract-consolidate-retrieve loop follows the Mem0 architecture (ECAI 2025), and the two-tier split extends the saved-memories versus history-reference design OpenAI ships in ChatGPT with an org-level tier a consumer assistant does not need. I shipped the same memory system in both this agent and the partner-facing AI Analytics product.
What I instrumented
From day one, every query was logged with its retrieval trace: what was asked, what context was fetched, what was answered, and whether the user came back. Adoption and retention live in PostHog. The logs are the roadmap. When a category of question kept failing, that pointed at a missing connector. When a feature I liked showed no usage, that was its obituary.
What the data killed
- Docs-first retrieval. The first version leaned on documentation as the corpus of record. The query logs showed the hardest, most valuable questions ("why does this sync fail for this configuration") were only answerable from source code and Slack history, so the connector layer expanded to cover them and the docs became one source among several.
- My original roadmap. I had planned a polished standalone interface. Usage data showed people asked questions where they already worked, so the investment went into the Slack surface and into workflows nobody requested but the logs suggested: triaging incoming Linear tickets for PMs, and translating raw technical output (a data-sync crash dumping stack traces) into plain-English root cause and next steps by researching the codebase and querying our observability layer.
What it turned into
The retrieval layer outgrew the product. The Elasticsearch MCP server I wrote for it (FastMCP, Python) is now the standard grounding layer for every agent surface at the company, including the partner-facing AI Analytics product. That is the pattern I keep seeing in AI product work: the demo is the agent, the durable asset is the retrieval, memory, and evaluation infrastructure underneath it.
Why it matters
A side project became infrastructure the company relies on, and it is the clearest example of how I work as a PM. I did not write a spec and wait. I built the thing, put it in people's hands, instrumented everything, and let real usage decide what it became.