This is a case study of shipping an AI product to enterprise customers who have every reason to distrust it: universities, answering questions about student data, in reports that go to accreditors and administrators. It is written as a postmortem, including the parts where the first version was wrong. I built it solo, from zero to production.
The problem
University staff at Stellic's partner schools assembled student-success and compliance reports by hand. Pulling the underlying data meant knowing which exports to run and how to join them, so a single report could take days and lived with whoever knew the spreadsheet. The product question was whether provosts, registrars, and directors of advising could simply ask, in Slack, in natural language ("which juniors are off-track for spring graduation and why") and get an answer grounded in their institution's actual data.
The prototype
I built the first version myself, PM as the engineer of record, because the fastest way to find out whether the idea survived contact with real data was to put a working agent in front of real partner questions. The prototype wired a text-to-SQL agent to institutional data and answered in natural language. In a demo it looked like magic. That version did not survive, and the reasons it died shaped everything after.
The paid alpha
We ran a paid alpha with five universities. Paid was the point: a free pilot tells you people will accept a gift, a paid alpha tells you the problem is worth budget, and it earns you partners who complain precisely when something is wrong.
Because the data is FERPA-adjacent student data, the alpha was compliance-gated before the first query ran: data-rights tiers controlling what each institution allows the model to see, and zero-data-retention configuration on the model side. Enterprise AI work is mostly this. The agent is the easy half.
What partners pushed back on
The feedback that mattered came from staff running real reports in weekly escalations and office hours, and it clustered on two things:
- Consistency over brilliance. The disqualifying failure was not a wrong answer, it was the same question returning different numbers on different runs. A report that goes up the chain of a university administration cannot be probabilistic. One partner put it plainly: they would rather have a number the agent refuses to produce than a number that changes.
- Provenance of the data. Institutions wanted to control exactly which data the agent could reach and to understand what grounded each answer, per the data-rights tiers they had negotiated. "The model figured it out" is not an acceptable lineage for a compliance report.
How it's built
The architecture that replaced the prototype is a two-path retrieval system with aggressive caching, and most of it applies techniques from the current agent-engineering literature rather than inventing new ones.
- Hybrid retrieval on Elasticsearch. Institutional data is indexed for both semantic and lexical search, following the contextual embeddings plus contextual BM25 recipe from Anthropic's contextual retrieval work, where chunk-level context is prepended before embedding and indexing. Anthropic measured that approach cutting failed retrievals by 49%, and 67% with reranking, and the lexical half is what makes queries full of course codes and term identifiers land.
- Constrained Django ORM lookups for everything else. Live records, counts, and structured facts that do not belong in a search index are answered through typed, whitelisted Django ORM query builders instead of free-form text-to-SQL. The model composes from a fixed vocabulary of lookups, so every query is permission-checked, reproducible, and auditable. The tool surface is designed the way Anthropic's writing effective tools for agents prescribes: few, unambiguous, high-signal tools rather than a raw database handle.
- Context management. The agent loads context just-in-time through tool calls rather than front-loading everything retrievable, and compacts long report-building sessions, per the playbook in Anthropic's effective context engineering for AI agents.
- Cache discipline. Prompts are assembled stable-prefix-first (system prompt, tool definitions, institutional schema, then volatile context last) so prompt caching hits on roughly 90% of requests, which is the difference between a Slack-speed answer and a coffee-break answer, and most of the inference bill.
- Two-tier memory. The agent maintains institutional memory (facts learned from one user that are valuable to every user at that institution, like how a school's terms or colleges are actually structured) and personal memory (each user's role, portfolio, and recurring questions). The extract-consolidate-retrieve loop follows the Mem0 architecture (ECAI 2025), and the two-tier split mirrors the saved-memories versus history-reference design OpenAI ships in ChatGPT, extended with an org-level tier that consumer assistants do not need.
The eval harness and the consistency requirement
The pushback became the spec. I built an eval harness that gates every release: a suite of real partner-shaped questions with known-correct answers, run against each candidate build, with consistency as a first-class metric alongside accuracy. Same question, repeated runs, the answers have to agree. A release that improves average quality but regresses consistency does not ship. This is the discipline I now think every serious AI product needs: the eval harness is not QA at the end, it is the definition of the product's promise.
What changed between alpha and production
- Text-to-SQL was replaced. Generated SQL was the main source of both wrong and unstable answers, because small phrasing changes produced different queries. The hybrid Elasticsearch retrieval and the constrained ORM layer made answers reproducible and auditable.
- Releases became eval-gated. In the alpha, judgment shipped changes. In production, the harness does.
- Data access became contractual, not configurable. The data-rights tiers moved from an alpha-era setting into the structure of the product, so what the agent can see at each institution is a term of the partnership.
The product is now in production, generating on demand the reports staff once assembled by hand over days.