Ten AI models gave the same wrong answer, and it's worse than hallucination

When AI Knows More Than It Can Offer

A literary translator asked ten leading AI systems, ChatGPT included, how to reach publishers and editorial offices. All ten proposed the same blogger-style playbook: publish translation tips, build a personal brand, grow followers. That solves a different problem entirely. The author calls it objective substitution — the model swaps the user's actual task for a statistically familiar one, and RLHF's preference for structured, friendly, complete-looking answers may be reinforcing it. The output is polished and plausible, which is exactly why the error slips through.

The model begins to reproduce not necessarily the most effective solutions but solutions that bear the marks of previously approved answers.
  1. windyVector

    This is first time i've really felt like commenting here, but super interesting article.

    Having tested both restricted and unrestricted models, certain queries which I know to have a solution can feel 'gated' by the restricted models. Rather than simply saying "I can't answer that", they choose to answer a similar, but slightly different question, very much along the lines of what the article discusses. To me, this is a fault of RL avoiding these "I can't do that" type responses while still remaining aligned.

    When you do push them on the solution, the models will often start to get more defensive, but reveal their awareness of it (along the lines of "Yes I know, but I can't tell you/do it"). It is very interesting to imagine the amount of solutions a model might have to a given query, that end up (intentionally or unintentionally) gated because of this process.

    I'd be interested to see if this behaviour of "almost-answering" the question is just emergent from these alignment policies.

    try an unrestricted model afterwards, id also be interested to see how that changes creativity.

  2. saghm

    I ran into restrictions for the first time at my job this week. I've been trying to figure out a bug that's occurred a few times (but never in a way we've been able to reproduce directly) over the past couple of months in our WASM code running in the browser. Somehow the combination of me sharing my Chrome console logs from a localhost session and it looking at the local build I made (in release mode but with debug info) triggered the security blocks and it no longer would respond to anything in the conversation other than with the blanket security warning. Frustrated, I tried in a separate conversation to just have it add diagnostics in the code so that the next time the bug occurs we'd at least have more info, but I guess my local state was tainted or something (maybe it was reading saved context from the first conversation) because even that the same blockage.

    I tried contacting OpenAI support, hoping that the fact that my company is literally paying for this product and there's absolutely no reason that debugging my own local code and reading my own local logs could possibly be unsafe for anybody, but it's been a couple days and the only thing I've heard back is a useless email purportedly from someone named "Justin" who says things that sure sound like they're not a real human:

    > You attached the Chrome log so Codex could find the relevant errors in your company’s WebAssembly build. My earlier suggestion to pull out an excerpt asked you to do the work you were trying to […]

  3. evolarjun

    Like many of the commenters it appears, my reaction to the incorrect solution to the problem was basically "duh, that's what they do". However, in reality I wouldn't necessarily have realized that until I read further and new more about what the prompt was trying to do.

    However, I think this is a reasonable explanation of a pattern I hadn't really articulated, but that I've seen in my work as a biologist. I think most of us have learned to prompt around the inappropriate or generic answers to get an answer we're looking for, but you do have to know to do that. And sometimes I just give up because the LLM fails to listen and I get frustrated enough to just do things myself (though less so with the latest round of flagship models).

    The issue the author describes is maybe not unexpected for people who work with LLMs in their field. But the issue is worth keeping in mind when prompting for something in a domain you're not familiar with, and it's not necessarily appreciated by people who are only casual users. Maybe there are solutions, but real humans don't understand a question and answer a different one all the time, so I'm hopeful there's a solution.

  4. slfnflctd

    I'm not convinced. The solution to this problem in most cases seems to me to still boil down to just more detailed prompting.

    As I see it, there are far greater concerns about guardrails being baked in - possibly at the request of poorly informed government officials - cordoning off multiple broad categories of knowledge needlessly. Which is what I thought this post was going to be about.

  5. garrickvanburen

    A fantastic articulation of one of my concerns in the space: yes, there are times we want the average of the internet. But those times are scarce.

More from this day

2026-11-18