Are you good at AI, or just using it?
We're working on a ladder for individual AI proficiency and would love feedback on both the levels and the definitions. We've proposed levels from L0 (New) to L5 (Loop), and we're seeking input on whether these are the right levels, if any names are unclear or overlapping, what observable behaviors distinguish each level, whether 'loop' makes sense as an individual proficiency, and if there should be an L6. We're defining these because in customer conversations, we've found that people are not very good at self-evaluating their own AI proficiency, and frequent use often gets mistaken for proficiency.
They are approximately the right levels, although: - Orchestrate is situationally useful. Quite often you're better off using agents in series rather than in parallel. Most people bragging about parallel agents or orchestrated agent workflows are just bragging. - Automate is hard to pin down. Not every process requires automation. But anything that is guarded by mechanical (deterministic) process is VERY helpful for keeping agents in line. - 'loop' is not quite that. Loop as it's currently being described in the field is about automating the process so that it remains entirely within the LLMs (which is a pipe dream, btw). They're actually saying that you don't have to write prompts anymore! That's a dead end. However, 'loop' in the sense you're describing (feeding results back into the documentation) is valid.
I don't care what my level is. I fire it up, try to prompt what I need to and get things done. If it doesn't do what I want, I update my prompt accordingly.
It doesn’t make sense to me to see that as a ladder. You’re not improving by going up. It’s different techniques and ways to implement, integrate LLMs. They each have their use cases and pros-cons, similar to any other topic in software and system design. The role of an engineer is to understand and manage trade-offs
'Automate' and 'Loop' are definitely poor names, given that all AI automates and the cheap subscriptions do loop to get things done even with what you call L2. I wouldn't call the steps on this ladder 'proficiency', to my mind they're too close to each other. Someone can go 0-2 in a few minutes if they're comfortable and enthused about reading the basic chat UIs. 3/4/5 are like, if this was a tech tree in a video game, all three would be unlocked at the same time by researching 2.
Simply knowing how to use AI is already quite impressive; you don't need the ability to develop AI. I believe you should quickly apply AI to areas you are familiar with. Soon you'll discover what AI can and cannot do for you. After that, you can find the areas you need to think about and explore in depth, while those should be left to AI.
L9: solving actual problems with A"I", not just increasing the speed of a crappy loop.