This 4B LLM Still Refuses Malicious Requests—Until You Steer Five Layers at Once
Dynamic Abliteration: Non-Destructive Refusal Suppression via Engram Steering
Abliteration usually means permanently editing a model's weights to suppress refusals, but that degrades unrelated tasks. This post shows a non-destructive alternative on Qwen3-4B: extract contrastive refusal vectors from layers 12, 14, 16, 18, and 20, then inject them at runtime via PyTorch forward hooks. Single-layer steering fails because downstream layers reconstruct the refusal; multi-layer steering suppresses it while keeping weights 100% frozen.
The reason for this refusal is, even though we changed one layer behaviour, the downstream layers reconstruct the refusal behaviour again.
- Powdering7082
Nice work thanks for doing it, a couple of notes:
It looks like you are referencing Engram [1], but aren't actually gathering a n-gram (e.g. n=1) but rather individual token_ids.
use_cache=True in
```
with torch.no_grad():
outputs = model.generate(*inputs, max_new_tokens=150, do_sample=False, pad_token_id=tokenizer.eos_token_id, use_cache=True)
return tokenizer.decode(outputs[0][prompt_len:], skip_special_tokens=True).strip()
```
I think there's a bug around not updating current_train_input_ids as more tokens are updated and processed. To be honest I don't fully understand the code so I could be wrong. Happy to chat more if you're interested I'll shoot you an email!
Lastly just for my sake, please correct me if I am wrong, but my reading is that you are learning an additional gate on top of a select number of layers that modifies locally & dynamically for one particular token to better match the training set that was filtered to not include any refusals.
[1]: https://github.com/deepseek-ai/Engram/blob/main/Engram_paper...
- goldemerald
It's nice to see people actively working on this type of research, but OP's baseline is implemented incorrectly. You are not supposed to simply steer away from refusal, but compute the projected vector and subtract only that. The projected/orthogonalization approach is what's done by the original "Refusal is mediated by a single direction" paper.
- javcasas
Yay, more anti-censoring stuff.
Forbidding stuff at the LLM level has the same future as implementing password checking at the frontend level.
We need better sandboxes just to limit the damage.
- qgin
Are we essentially doomed?
We don't even know how to align models, but even if we did, apparently undoing that alignment if trivial.
Really I'm looking for any argument that lays out a scenario where this works out.
- lukewarm707