By skyforbes A cautionary tale about confusing magic with reliability. Based on real experiences building AI features in B2B SaaS — names and details changed to protect the innocent (and guilty).
Monday morning: Marketing popped champagne bottles. Friday afternoon: Support threatened to quit en masse.
The crime? I shipped a “conversational AI search” powered by RAG that transformed our reliable keyword filter into an eloquent liar. NPS nosedived. ARR wobbled. The CEO asked if I’d “considered pivoting my career toward fiction writing.”
This is the autopsy of how I turned a boring-but-profitable B2B SaaS into a trust-destroying slot machine. If you’ve stared at ChatGPT and thought “we should build that for our users,” grab popcorn. This is your warning shot wrapped in dark comedy.
The Scene: A Search Box Nobody Hated
The product: DocuVault — document management for legal teams. 10K users. $2M ARR. Boring as oatmeal, profitable as a parking lot.
The old search:
- Elasticsearch with faceted filters
- 200ms p95, deterministic, predictable
- Query: “contract amendment 2023” → Result: exactly that