Deterministic Simulation Testing Isn't Just for Distributed Systems
Deterministic Simulation Testing Is for Everyone
Deterministic simulation testing is usually associated with wizard-grade systems like FoundationDB and TigerBeetle, but the author argues every application has an environment and a machine. Make the machine deterministic, simulate the environment, and you can compress a year of runtime into an hour, surfacing bugs in development before users ever find them. Web apps and embedded systems get the same benefits as distributed databases.
A good simulation is an exercise in amplifying the deviousness of those parts of the environment we cannot control.
- csbartus
Today this is the second post I see where the 'traditional' testing methods (unit, e2e, integration, perf, pen, etc) are re-categorized, are put in new perspectives.
The other one is https://alperenkeles.com/posts/vocab-for-testing/
I truly understand our (only) job is to become the oracles of the machine: https://www.osequi.com/manifesto
- drooby
This is the future, I'm convinced.
I’ve been building a framework of my own and it’s already found dozens of important and critical bugs on my team at work.
The next problem to solve after this, though, is formalized intent.
A bug is often a bug if the app is not behaving as intended. What we intend is not always so clear, and it will remain an area of the engineer's and product designer's responsibility, because software is complex, and what we want cannot be computed so simply. Defining and understanding intent will remain a "human bottleneck" for many companies.
- TacticalCoder
Deterministic testing is one thing and it's badly needed.
But I'm totally convinced we're moving to deterministic execution: n various implementations, in different languages, running on different stacks, and a minimal (maybe even hardware) voter taking care of the n-modular redundancy to pick the correct "answer".
0-day Postgres CVE? No problem: one of the implementation gives bogus answers and is detected as broken.
0-day Java? Same thing.
This shall make 0-day in anything but the voters a thing of the past (and the voters shall be minimal and have a tiny attack surface).
The answer can be an API query but we can go much lower: network packets, deterministically created, by n systems and all going to the voter(s). Whatever the "answer" is, it's going to be deterministic and verified from the lowest possible level.
The implementations shall be written from specs, formally verified, etc. But the one thing that'll guarantee any fuck up is caught is going to be n independant implementations of the same specs.
And as soon as one of the implementation/stack return even just one network packet that doesn't match what the other systems do produce: something is fucked up and has to be fixed.
This shall require throwing a huge lot of the non-deterministic bullshit away but we'll see systems doing just that: n-modular redundancy and voters, at the lowest level possible.
I'm 100% sure we're going there (and AI is going to write most of that for us).