GitHub's first fix for Actions and Issues outage fails
Incident with Actions, Pull Requests and Issues
GitHub is investigating degraded performance affecting Actions, Issues, and Pull Requests. Users may see errors loading the Actions tab, using the Checks API, queueing new runs, creating or updating issues, and requesting PR reviews via REST and GraphQL. An initial mitigation did not fully succeed, and notifications are delayed. Pull Requests is now operating normally, but the root cause is still under investigation.
Our initial mitigation was not fully successful, and users may still experience delays and errors with Actions, issues and projects, and notifications are delayed.
- bearjaws
People want to dunk on GitHub, but honestly going from 50M of <thing> per day to 2B <thing> per day will break pretty much any system anyone has ever built here. Even if it was over the span of ~8 months, many things will fall over and then as you patch those the next domino will fall.
I feel bad for the team, but will be great growth opportunities for all.
- orsenthil
Yeah, only yesterday I let claude build and deploy me a end to end project that will flicker my lifx light-bulb when commits to any open source project lands. In the morning I saw it stopped around 11 pm PST yesterday. I was wondering why and I was wondering, how easy it is do a complete working project like this in minutes and what is stopping people to do 100s of such project per day. Github wasn't built for this scenario.
- mglvsky
Why they didn't prompt something like "please make whole system more reliable, don't make mistakes"?
- kqgnkqgn
Given that Github seems to persistently have these scaling challenges, IMO either one of these two things must be true:
1) either they are less competent at running large scale infra vs other large tech companies
2) or git itself adds unique challenges, that make this more difficult to scale
I think 2) is much more likely than 1).
So why stick with git? Is it feasible to make a new VCS that scales better if you start from scratch?
Obviously getting adoption of something else would be a challenge. But Github has enough market-share that they could plausibly pull this off. Particularly using costs or reliability to incentivize migration.
(Not saying that this would necessarily be good for the community in general but I think good for Github.)
- throwitaway222
Perhaps they need to make actions free and actions plus. Actions free does everything the current product does, but has deferred updates in the UI. And possibly have actions free/plus push updates to your own agentic infrastructure instead. Just configure a URL and all your actions updates go there
- zanderwohl
I have not understood, and still do not understand, why everyone locks themselves into github's CI/CD pipeline. It's not hard to set up checks that run on a generic docker setup. I have a home server that I use for all my builds and checks, with a github actions config to run manually if I can't VPN into my home network.
Besides being more reliable, it's just faster to run on a machine you control.
- igsomething
How long until they increase their plans' pricing and reduce the free Actions minutes to drive bad users away? I would rather have 500 minutes/month of reliable GHA than this.
- loveparade
I think it would be more efficient to post on HN when there is a day without GitHub issues.