The JustinBot: Devouring Our Support Inbox With AI
After a Support Week, I'd come off angry, peeved, and exhausted. Productivity tanked for days. Burnout clouded decision-making. During team catch-ups, "Are you just saying that because you recently finished support?", was a common refrain
More often than not, the answer was a reluctant "yes".
Support Weeks draw us away from the team for an entire cycle. The context switching is brutal, making it impossible to create room for deep work. Delving into the inbox is useful when first closing an acquisition. But there are diminishing returns. At this point, it was a "code red" problem.
We'd already tried the obvious moves. Justin helped us implement better systems and tools for support. We shut our inbox down on weekends. Expanded our documentation. Tweaked the application to address common issues. These changes were all positive. But, the requests became incrementally more difficult (good! We solved a lot of low-hanging fruit.) and increased as a function of the underlying growth our companies were experiencing (it could be worse!). Powering through support still cost our Partners multiple hours every day. Even trivial requests were taking longer than expected. Not to mention the implicit costs of having open loops and mental fatigue caused by the ordeal. Not good!
§Why off-the-shelf bots failed
We looked into AI support bots like Finn from Intercom and HelpScout's bots. But out-of-the-box solutions have three fatal flaws:
- They only read documentation. They can reset a password, but they fail at complex queries. If it wasn't in the documentation, good luck! The bots would make up an answer, confusing both the user and human reviewer who inevitably picked up the pieces.
- Users hated them. Even I have trauma from other site's chatbots, running around in circles and never giving truthful answers. Trying to straighten out a billing error becomes a feat parallel to hacking your way to ticket escalation so a human would review it.
- They lacked context. They didn't know who they were talking to, what the user was doing, where they were in the process. Bots would start from 0 every time. Sometimes it could guesstimate at the user given the initial query. But this wasn't consistent. And the user's information was rarely used to help solve the underlying problem.
Having an obstacle and getting trapped in an auto-responder turns support into a wrestling match. If an artist has a billing issue, the bot spits out a generic, irrelevant article. It sucked! Users stomped away. Tickets which required my attention tanked summarily with customer satisfaction scores. This was a complete and utter dead-end.
§What we built instead
We knew AI could work if we removed those false constraints. So, we built our own: JustinBot (named after a famous film, book, and software blogger, you should read his stuff).
JustinBot evaluates a request in two steps before responding. First, it reads the message. Next, it chooses between two paths:
Path 1: The Documentation Route If the bot is over 90% confident that an article answers the query, it sends the link. This handles about 15% of cases. Most commercial bots stop here, leaving the other 85% of users frustrated. This capability was on par with the off-the-shelf solutions we were familiar with. But this was not the goal. If all which was achieved was being "on par", this would be a colossal waste of time. Which brings us to Path 2, where the real benefits were.
Path 2: The Context Route For the remaining issues, JustinBot builds a user profile. It reviews everything we know about the user to provide a hyper-specific, contextual answer. Say the artist has a payment issue. The bot will do the following:
- Check the artist's recent applications
- View the current payment configuration
- Run through a variety of common scenarios
- Identify the one which matches closest to the original message
- Configure a response
The same logic covers most other tickets — account setup, settings, our custom modules. I've tried to give it as much context as possible: the account's history, error logs, and the language we use in support responses. The goal is to replicate what a real support agent would have on hand before they reply. And because we are using AI, we are able to collect and process this information at 100x the speed while 5% of the cost.
Responses would include as much information as the user could possibly want. It would incorporate what we knew about them, to let the user know we weren't guessing. If possible, a helpdoc was linked to. With this, users were now getting contextual, helpful, and speedy responses. Our support team was able to step back big time.
§A human reads every reply
Steve Jobs used to run Wednesday Marketing and Advertising meetings, where their top agencies and marketing directors would present what was slated to be released, and when. For a period of time, every billboard, commercial, and piece of collateral was approved by Steve himself. Apple's image was so important, it could not be delegated.
Our customer support is equally important, for us. No matter how good JustinBot gets, I don't want to give it full autonomy to respond to our users. Instead, it sets up drafts for our support team to review and send. This lets us catch obvious errors (which then get fixed), tweak language, and double-check that everything is running smoothly.
The idea of artificial intelligence working for us while I sit back and relax in the Florida Keys is not reality. I want to be supervising. Making sure we're heading in the right direction, course-correcting, building incremental improvements into the existing system. And I can't delegate reviewing others' work too much.
So: a human stays in the loop. Able to press the red button if necessary. Able to suggest improvements. Able to adjust the system where necessary. It means we move a tad slower, but with confidence. We catch problems earlier, and spot what's working sooner. The trade-off is worth making, especially when our reputation is on the line.
§The payoff
JustinBot now handles ~80% of Tier 1 requests. It answers simple questions instantly, saving us from the context switching, investigation, writing responses. And on the requests it can't answer, the reasoning it produces points us at the right thread to pull. If the bot has already checked the settings and the payment is still failing, we know to look elsewhere.
By feeding JustinBot the full context of a user's account, we get something that acts like a human agent, without the human cost.