AI scribe's false drug claim leaves patient devastated after surgery
Error by AI scribe during medical appointment leaves patient devastated
A new report warns that AI medical scribes can 'hallucinate' false symptoms, with serious consequences for patients. Rebecca Green's urologist's AI scribe falsely recorded that she micro-dosed psychedelic mushrooms, which appeared in a post-operative letter to her GP. Green, who was receiving workers' compensation, feared the error could impact her case. The report found other errors, including wrong body parts and misdiagnoses, and highlighted inconsistent consent practices. Regulators are reviewing the tools, but none are currently approved.
I was gobsmacked and in tears … it made up that I take [psychedelic] mushrooms. I have never done mushrooms in my life.
- kioleanu
I am shocked by the comments in this thread ranging from “yeah but humans also make mistakes” to “yeah but how many mistakes does an AI do compared to a human”. Neither of those is the point here. We have doctors that have employed AI as a software to help them and that software is flawed. I am a software developer and if I write some piece of medical software with bugs that are so blatant in it, I'd see hell up to potentially being sued into oblivion
- pizzly
The other side is that I have witnessed doctors making mistakes personally on medical files. They are overworked and only see you for 15 minutes and in a rush to kick you out and so of course they make errors. It would be interesting to compare whether transcription software or humans make the most mistakes. Guessing the answer will be very nuanced such as transcription software failing for accents not trained on, how over worked the clinic is etc.
- Arn_Thor
You can optimize work. But you can't optimize AWAY work. If you depend on human attention, expertise and judgement, as we do in medicine, you need that trained doctor to have the time to look at the transcript and think carefully about the contents. If we use AI as a way to help doctors spend even less time on each patient's case, we're going at it the wrong way. It shouldn't help cut costs, it should help improve the quality of care.
- newsomix9xl
Not only did the AI apparently hallucinate psilocybin consumption, but the doctor then hallucinated internal bleeding as a risk, which is not a known risk of such mushrooms.
But this seems a bit click baity.
You can challenge your medical record, and presumably an AI transcription service would be in there. Unless .au is special in that way.
- arjie
The classic whisper model will give you text in silent moments too. A lot of these guys just wrapped that. You can’t just do the minimal thing. Well I guess you can and you’ll get sales but it won’t correctly solve the problem.
My cousin used one of these apps with patient consent and said she then has to listen to the audio and rewrite it all. Stopped for that reason. Wasn’t even a time saver.
Strangely, I think Robin really just mistimed this. They stopped just as the state of the art came out and with their human in the loop transcription they might have been quite useful.
- tanelih
> There's evidence that for the first few weeks of using AI, [people] check things pretty carefully and then at a point they stop checking because they assume it will be right.
I think this is an issue people often overlook / does not get enough weight in the discussion.
I do wonder that, when such profound technologies (such as AI, social media) are rolled out, should they be subjected to studies from the "human perspective" for a longer period of time.
However, this might be impossible in the current system.
- eelioss
I am surprised the doctors do not have to review a bit the transcription. I do have to review my code before deployment
- red75prime
It's easy to bring down the hammer of scope insensitivity by vividly describing a single person's struggle. The question is how AI affects error rates.