LLMs mistake deadly webcaps for chanterelles in mushroom ID test
Mushroom hunting with LLMs: what can go wrong?
A developer benchmarked top vision LLMs against 1,040 mushroom photos from the FungiTastic dataset, covering 55 species from Polish foraging lists. Gemini 3.6 Flash led with 64% top-1 accuracy, while Claude and GPT models lagged. The most alarming errors: deadly webcaps and death caps were frequently misidentified as edible chanterelles and grisettes, highlighting the life-or-death risks of relying on AI for foraging.
A deadly webcap is called a chanterelle - the same mistake that kills foragers.
- Insanity
Trusting an LLM with a decision that can potentially be fatal is a good Darwin test I suppose.
Sadly I don’t think the general populace understands that LLMs are unreliable. (And even people in tech can vastly overestimate the capabilities.. at least judging by the insane spending on them).
- julienchastang
Ha! I just had a very interesting conversation with ChatGPT about a mushroom that I found in my front lawn. I am 99% sure it was an edible oyster mushroom which included analysis the of the size, growing substrate (dead ash tree root which turns out to be a useful identifying characteristic -- I learned a lot about mycology in a short time). Plus I don't use fertilizers, herbicide, pesticides, fungicides (obviously) on my lawn so I think it would have been quite tasty. Did I eat it? Hell, no. But then I got to thinking, would I trust a human to correctly identify it? I am actually not sure I would either. Penalty is just too great for misidentification LLM or not.
- WarmWash
I've used Gemini quiet a bit for foraging. It's definitely a "trust, but verify" situation. It's really good at at least getting you in the right lane to verify, rather than having to comb hundreds of pictures.
So even if it's wrong, a short verification usually reveals why it was wrong. I haven't had a situation yet where it was catastrophically wrong (i.e. the mushroom it called out looks nothing like the mushroom imaged). It also will generally warn of lookalikes.
I also use it a lot for general plant ID, and it's really impressive there too, with much lower stakes (I'm not eating those).
- zikduruqe
There are old mushroom hunters, and there are bold mushroom hunters.
There are no old bold mushroom hunters.
- Legend2440
Is it possible to reliably (>99%) identify mushrooms from a single image, with any method?
All image datasets have an intrinsic minimum error that comes from the limit of information in the data, regardless of what model or method you use to analyze it.
You might not be able to tell apart two similar-looking mushroom species just by looking.