How concerned should we be about Astra's recurrent architecture?
Astra, an AI system from the Alignment Research Center, uses a recurrent architecture that processes information in a loop, raising questions about its interpretability and safety. This analysis examines the potential risks and benefits of such a design, considering factors like transparency, control, and the system's ability to generalize. The author weighs arguments for and against concern, offering a nuanced perspective on whether Astra's architecture poses unique dangers or if current safeguards are sufficient.
The recurrent nature of Astra's architecture means that its internal state is constantly being updated, making it difficult to predict or audit its behavior over time.
- mentalgear
So OpenAI’s stance on interpretability (ai safety) is now basically that Blues Brothers meme: two guys in dark sunglasses, driving at night in a car with a broken windshield, pedal to the metal, asking, "What could possibly go wrong ?"
- khalic
I was under the impression that intermediate tokens (“chain of thought”) are _not_ a representation of a model’s logical path, with one study observing that you can replace intermediate tokens with single character chains and still get the increased precision…
- samrus
I like the idea of more reccurance in the transformer level. Chain of thought always seemed so clunky. Its just not the way the human brain processes information. Its an extrmeely crude approximation at best
- kjshsh123
>In contrast to a classic RNN, there's no unbounded hidden state accumulating across an entire trajectory
I don' understand this line. In a classic RNN hidden state is bounded dimension. In fact it's transformers that technically have unbounded hidden state.
You can't parallelize classic nonlinear RNNs for various reasons but in training both RNN and Transformer depend on the entire sequence history in a way that is unbounded. Of course in practice you just train on a max sequence length.
RNN xhat[t+1]=f(x[t],h[t])
Transformer/self-attention xhat[t+1]=f(x[t],h[t],h[t-1],...,h[1])
- anon373839
Sebastian Raschka posted about this architecture:
> A lot of hype around OpenAI's Astra model here on my timeline today. Apparently, this goes back to a new article from The Information, which said Astra is a "recurrent depth or looped transformer".
> It's always interesting to read about new or different approaches (including rumors about what the closed labs may be up to), but let's debunk this a bit.
> About 2 months ago, I shared the architecture details of Nanbeige, for example, where "Nanbeige4.2-3B is pretrained from scratch on 28T tokens with a Looped Transformer that reuses the layer stack to increase capacity without adding parameters."
> Yes, that's it. The looped transformer idea is just reusing layers in the transformer block.
> In the case of Nanbeige, the main idea is to reuse the same 22-layer stack (=transformer block) twice instead of once. So, effectively it extends the 22-layer architecture to 44 layers, but without duplicating the weights.
> In simple terms, this roughly doubles the size of the model (if we ignore the embedding and output layers for a second). But instead of requiring 2x the storage and RAM to host this model, it stays at the same size since we reuse the components. However, it's almost 2x as expensive in terms of compute, because we run the embedded text through almost 2x as many layers.
> Why? In the Nanbeige 4.2 technical report, the researchers found that two passes gave the best trade-off and retained about 75% of the token efficienc […]