KillSwitch: a programming language built so LLMs can't use it

Here's FizzBuzz:

https://github.com/dom96/KillSwitch/blob/main/tests/fizzbuzz...

Somehow I think this is more adversarial towards humans than LLMs!

This is not "a programming language that LLMs can't use", it's "a programming language that's adversarial to LLMs that haven't been trained how to use it".

I was hoping for something like a language with properties that change/morph as it functions, that make an LLM unable to use it by not having a means to track and continuously reference the changing cipher/state or something.

Security by obscurity is not security at all.

You're attempting to use a few little gotchas against LLMs that only make sense if you're looking at it one dimensionally as a standard execution flow. AI is not limited by that. Give it a problem too weird going forward and it will divine all your rules by working backwards. Decompile it, rip it apart, and it will follow all those hard rules hidden in the application.

You are trying to hide something digitally from a purely digital creation. This is not a recipe for success.

"Writing code is probably something LLMs have excelled at the most. But how much of their ability to write code is due to their understanding of it vs. just them recalling code snippets which were present in their training dataset or their context window?"

This makes me think mostly about how humans work.

Programming is to humans like speaking a language. You have ideas for what to achieve and express, and if you are a proficient programmer then outputting those thoughts in a programming language is second nature.

Programming (like most things humans learn) takes practice. It is NOT enough to simply read a specification and then you know how to code (except for transferring skills between similar languages). You repeat, make mistakes, and learn from them.

I would be tempted to say that humans definitely "repeat snippets from their training dataset". We mainly recall and express from our training/experiences. Creativity is mostly recombination of things already seen.

I wonder if another approach would be to map the language syntax onto some banned/censored LLM topic(s) such that programs end up as "readable" instructions for topics the LLM refuses to read and/or write, like producing napalm, nukes, bioweapons, etc. Doesn't have to be factually correct instructions, the terminology itself is enough to shortcircuit most LLMs.

(Inspired by recent malware including bioweapon instructions in the source code as a way to shut down LLM-based analysis.)

If found that stack languages are a massive weak spot for LLMs. It requires accurate tracking of what is on the stack and for some reason that doesn't really work. So even the best models end up making the same basic mistakes over and over again.

I feel like most things at the Esolang Wiki fit the bill. Chef, Piet, and Befunge are two favorites that spring to mind. This reminds me of Chef in particular.

https://esolangs.org/wiki/Chef

https://esolangs.org/wiki/Befunge

https://esolangs.org/wiki/Piet

Edit: Hmm... I guess probably less usable by humans.