Kinney Drugs pulls back AI phone assistant after hundreds of customer complaints
Kinney Drugs is scaling back its AI assistant, Burt, after customers reported incoherent calls, wrong dosages, and missed prescription notifications. The bot, introduced in May, will be replaced by the classic touch-tone phone system for incoming calls. President John Marraffa acknowledged the experience was not right, but emphasized privacy and security were not compromised. Burt will still be used for outbound communications like refill texts, but patients must opt in.
But one thing I want to point out is getting privacy and security right does not mean we got the experience right. We did not, and we own that.
- dkarl
Cases like this raise a question. As an engineer, the errors that coding AIs make are an annoyance, and reassuring in terms of my job prospect.
As a consumer, the errors that AIs make are more than an annoyance. I have yet to meet an AI phone assistant that can do anything more than an explicitly programmed phone tree, and several, including the one for the pharmacy I use, that do far less. They are undoubtedly much more expensive to create and test.
Clearly there's a bubble going on, and unprincipled people chasing funding and promotions as usual, but why are companies like CVS paying more to get less?
I think it is literally to waste our time. It's one more layer of defense to stop you from talking to a person.
There are two big wins in health care. One is to increase the scope of health care that gets paid for. More and more expensive treatments, to bring the money in. The second big win is to stop people from getting treatment. Deny coverage, confuse them and wear them out with bureaucracy, and in this case stop them from talking to an expensively educated human pharmacist. That's what an AI does better than a phone tree, because it more effectively creates a sense of helplessness and valuelessness in the customer.
Expanding the scope of care you're entitled to and then fighting against your ability to get it isn't a contradiction any more than it's a contradiction when a person breathes in and breathes out. It's two synergistic parts of a system whose goal ultimately has […]
- dj_axl
FWIW I know of a pharmacy network that's implementing AI. Not only does the AI make egregious errors like speaking in Spanish when the conversation is in English. It also has a limited context window for ground rules of 5000 tokens or less. So correcting all the various errors goes over the instructions limit. As a result if the implementation goes through it will be of poor quality. All I can think is that people with no technical expertise are making these decisions, it is easy for any techie to predict that AI of this level won't be capable.
- jonathan-adly
So - I run a company that does AI agents for pharmacy. We are at Series B now and growing well - series C in 9-12 months given growth. We just acquired a competitor of that vendor so very much an insider.
First, The technology works, and it scales, but the whole bottleneck is domain expertise and implementation. These are expensive, and hard to scale. We hire pharmacists as project managers, that's how important domain expertise and implementations are.
Second, the amount of noise of "Voice AI for <industry>" is incredible. Most of them are completely clueless about the industry & basically "YC-striver" type who can only sell to other yc companies. They fail hard the moment they touch critical functions of the real world.
We have hundreds of pharmacies with us now and they mostly successful and happy. Our biggest friction is customers being burned by vendors like this and writing off AI altogether as "not ready".
- cmiles8
Companies will pawn off thier customers to AI at their peril.
In many ways this is a repeat of the India call center train wrecks of the 00s. On paper, letting someone in Bangalore vs onshore handle incoming customer service calls looked like a path to amazing savings. In practice the customer experience was horrendous and companies CTRL-Zed these decisions and rapidly brought customer service back onshore again. AI is just that story of shortsighted decisions by weak leadership playing out all over again.
- SalaevAl
Pharmacy phone lines are a terrible first use case for voice AI. Older callers, drug names, insurance mess, zero patience. Automating "store hours" is fine. Automating refill exceptions is how you get this outcome.
- criddell
This is my favorite AI customer service story:
https://arstechnica.com/tech-policy/2024/02/air-canada-must-...
> Air Canada essentially argued that “the chatbot is a separate legal entity that is responsible for its own actions”
- Glyptodon
Not Pharmacy, but I literally had Shopify checkout button clickjack me and then AI agent and human agents (chat, email, and phone) refuse to cancel the resulting order today. Despite contacting within 5 minutes. Given how hostile so many companies are, I question why they even bother appearing to have support.
But this doesn't seem likely to translate well to pharmacy, where there will be consequences for treating customers like vermin, as opposed to actual incentives to do it.
I wouldn't be shocked if the tech needed in places like pharmacy or other sectors where there's some level forced responsibility leads to the version that's actually palatable elsewhere.
- drewg123
Actual personal service from a human being is the main reason that I'd choose a small regional pharmacy over a local CVS or mail-order. It seems to me like personal service is the ONLY advantage a regional chain like this has. If I were them, I'd lean into it rather than burning customer's goodwill with AI.