LLM Classification Is Just Feature Engineering
LLM Classification Is Feature Engineering
Treating an LLM as a classifier is painful: you get hard labels, no calibration, and no principled way to trade precision for recall. But wrap the LLM's verdict in a logistic regression and it becomes just another feature. The author shows how this restores calibration, lets you add structured covariates, and turns prompt-tweaking into standard ML feature engineering. A test case on 4,618 irony-labeled tweets demonstrates the approach.
These failures are not the fault of the LLM: it's not designed as a classifier and indeed has no mechanism for plausibly doing some of these things. But only because we're thinking of things incorrectly…
- vova_hn2
I think that this can be automated by using two LLMs: a stronger/more expensive for generating prompts and a weaker for actual classification. Approximate algorithm:
1. Give "strong" LLM the task formulation and some labeled examples. Ask it to generate a prompt for the "weak" LLM.
2. Run "weak" LLM on the training set with generated prompt from 1, use replies as features for a smaller ML model (logreg, decision tree etc).
3. Pick examples from the training set that your small model is most wrong about and ask "strong" LLM to generate one more prompt (like in 1), except this time you are using the misclassified examples instead of random.
4. Run "weak" LLM on generated prompt from 3, add results as one more feature for your model.
5. Repeat 2 - 4 until your token budget for this task is exhausted or required score on cross validation set is reached.
I was thinking about creating an open source library that implements this, but I'm not sure if anyone really needs it. I suspect that people who need something like this already made their own implementation.
- softwaredoug
In my work on LLM as a judge, I prefer to use LLM decisions as features in a downstream classic ML model for the final decision. It works really well
https://softwaredoug.com/blog/2025/01/21/llm-judge-decision-...
- iforgotmypasswo
Shouldn’t this article be about the disadvantages of using TypeSafe’s Jev as a classifier?
This is a bit of an outdated take as of two days ago. Dear lord things move fast these last few years. Some of this is still relevant. Fine tuning Jev once available could address certain concerns.
(Very excited as I got an invite email for TypeSafe today! I don’t have time for all the little experiments I want to run with Jev and Astra combined!)