How DoorDash Uses LLM Juries to Build Accurate Food Metadata at Scale
Building Food Metadata with LLM Juries
We tackled the challenge of standardizing millions of unique food items by building an AI-led metadata platform. Our system uses LLM juries for high-quality evaluation, context-optimization agents to refine prompts tenfold, and distributed computing to slash processing time. This approach replaced slow human labeling, boosting accuracy by 20% while cutting costs, ensuring a superior search experience for our users.
We found that the consensus LLM tags were about 20% more accurate than typical human-annotated labels.
- sigmar
>Evaluators validate each tag individually — for example, protein, preparation, or health, individually rather than judging the item as a whole.
Am I reading this right that the jury is multiple LLMs each iterating through each tag and voting on each? Why wouldn't you tune one LLM to be really competent at a single tag? Like a single "spicy evaluator LLM" or "protein evaluator LLM"?
- vector_spaces
I am sorry to be harsh but I find it amateurish that they would use an AI generated hero image for this and presumably fabricated LLM output -- fabricated by an AI image generator no less
Whenever I create an image like this for the purpose of a demo, I make certain that it demonstrates either real input/output or at least is exemplary of real input/output because the whole point is to instill confidence in the tool. Sure, if the raw outputs aren't clean/comprehensible enough for presenting to stakeholders or others, fine, clean them up to make them comprehensible or add explainers, but there shouldn't be any need to fabricate the inputs.
I feel obligated to respond to the hypothetical "But they don't want to tie it to a particular restaurant or brand" -- you don't have to! Doordash has taken generic food photos for this exact purpose.
- TeeWEE
Basically it’s AI on top of AI for metadata extraction.
There are a lot of claims in the article but not a lot of hard data. In the end they still don’t know if the data is correct.
Good luck with your glutes allergy.
The weird thing for me is the prompt optimization loop? Why not fine tune the model instead of AI generating the prompt?