Disruption with some GitHub services

Resolved

Between 01:36 and 08:11 UTC on Thursday March 12, GitHub.com experienced elevated error rates across Git operations, web requests, and related services. During a planned infrastructure upgrade, a configuration issue caused newly provisioned Kubernetes nodes to run an incompatible version of etcd, which disrupted cluster consensus across several production clusters. This led to intermittent 5XX errors on git push, git clone, and page loads. Deployments were paused for the duration of the incident.Once the incompatible nodes were identified, they were removed and cluster consensus was restored. A validation deploy confirmed all systems were healthy before normal operations resumed.To prevent recurrence, we are adding programmatic enforcement of version compatibility during node replacements, implementing monitoring to detect split-brain conditions earlier, and updating our recovery tooling to reduce restoration time.

Resolved

This incident has been resolved. Thank you for your patience and understanding as we addressed this issue. A detailed root cause analysis will be shared as soon as it is available.

Update

We've identified the root cause and are working on resolving the underlying issue. Some users may have encountered intermittent failures and errors. We're continuing to see reduced error rates.

Update

We are investigating elevated error rates. Error rates are now decreasing and we're continuing to monitor the situation.

Update

We are investigating elevated error rates. Error rates are now decreasing and we're continuing to monitor the situation.

Investigating

We are investigating reports of impacted performance for some GitHub services.

Investigating

We are investigating reports of impacted performance for some GitHub services.

Investigating

We are investigating reports of impacted performance for some GitHub services.