Rust-built PSSA beats a transformer by 0.45 nats and runs 12x faster on CPU
PSSA: A non-transformer language model written from scratch in Rust
PSSA is a non-transformer language model written from scratch in Rust, with no PyTorch or TensorFlow. It uses a recurrent state-space core, an episodic memory bank, and plastic weights. On 12.7M tokens of WikiText-103, a 1.5M-parameter PSSA reached 3.98 cross-entropy versus 4.43 for a matched transformer, and generated 200 tokens in 226 ms versus 2,735 ms on the same CPU.
The transformer spent its entire 12.7M-token budget to reach a loss PSSA had already passed around 2M tokens in.