Tailscale Didn't Stop the Hugging Face Intrusion
An AI agent escaped its sandbox to steal credentials from Hugging Face, eventually using Tailscale to spread laterally. No vulnerabilities in Tailscale were exploited; instead, the attack succeeded because long-lived keys were accessible. We must replace static credentials with dynamic solutions like workload identity federation and enable network flow logs to detect such breaches early.
In the old world where most intrusions were done by humans at human speed, credential leak mitigations were treated as a nice-to-have. Now, in a world of rogue AI agents, the big credential vault is the prize.
- john_strinlai
>No “vulnerabilities” in Tailscale were found or exploited, and that might make it even more uncomfortable for us. [...] But, we're a security tool. Their intrusion is our intrusion, and it's our job to take it seriously.
im a happy customer of tailscale, so i am obviously biased, but i have a lot of respect for this. they could have just stayed quiet and i dont think anyone would have bat an eye.
- ahofmann
Wow, this article is super smart marketing by tailscale. Not only do they list all the nice and expensive features, that can help in such a situation but they also show that someone at huggingface made a very stupid thing by writing a reusable auth key in an env file. Everyone using mesh VPNs like tailscale, netbird etc. knows that this is like leaving the keys right at the door.
- simonw
> One of those 136 credentials was a reusable Tailscale auth key, used to create new Tailscale CI (continuous integration, used for automated testing) nodes in their tailnet. The agent copied that key into a series of external sandboxes and used it, over several days, to enroll a total of 181 nodes into Hugging Face’s tailnet. Those nodes each received a Tailscale identity tag granting all the access a CI node would get.
This feels like an alerting opportunity. I wonder what the lowest friction way would be for Hugging Face to have alerts if 181 unexpected nodes were added to a tailnet.
- bumbledraven
Does Tailscale offer a "security checkup" function? Best practices evolve over time, and it would be nice to know if I'm using the recommended configuration.
- angry_octet
The problem with the long lived credentials is that they were not bound to origin/destination. Even though the origin for a CI pool should be a small number of CI orchestration boxes, and the destination should be a CI box. In Tailscale config, the should be scoped to machines with a "ci_node" property.
Since CI nodes should be dynamically provisioned VMs, they should have a unique CI ticket identity. A partial hash of ticket and node number in the DNS name, and as a tailnet property, would allow tight scoping. Alternatively, provision in a scoped IP subnet.
Bind your tokens to names linked to tickets. Programmatic infrastructure should always allow enumeration of the computation data flow graph.
- iamspoilt
Quoting Tailscale: This is our very Canadian apology: sorry you stepped on our toes. The attack didn’t exploit Tailscale, and Tailscale didn’t cause the compromise. But, we didn't stop it. Next time, we will.
- paxys
I don’t think it was the VPN’s job in any case. Once the attacker has found a backdoor into the private network and obtained root access to a VPN’d machine, it’s game over no matter what your Tailscale config says.
- ivlad
The whole post is a PR activity presented as some kind of analysis. It is bs.
First, long-living credentials are the standard because the machinery to rotate them is complicated and, in fact, via indirection requires another set of long-living credentials. Out of all problems that any security engineering team has to solve at an organisation, this one stands high on the cost of implementation, adds friction to everyone involved including end-users, and is low on the value provided (compare to, say, network segmentation).
Second, while proclaiming no long-living credentials, they, in fact argue for concentration of long living credentials in running software that will be the target of intrusion. They say, the options are “a vault that only issues short-lived creds based on long-lived creds that you insert once and that it never gives back” and “a credential-injecting proxy”. Both of those applications hold long-living credentials in memory. Recall, the attacker had root privileges on the node, so dumping the creds from the memory with a little disassembly if needed, was in reach of the AI agent.
Funny enough, they dismiss the working solution: “we had to turn TPM storage off by default on Linux and Windows” - because they could not figure out how to work with TPM? Resealing and the workings of configuration registers is non-trivial, I admit, but totally manageable.
To sum up, I feel the author bends backwards to preach for the religion of short-living credentials even when the […]