Downtime Postmortem - Blog2020-03-05
On the 2nd of March, 2020 we had a period of downtime.
The downtime started at 4:29PM AEST and ended at 7:27PM AEST.
The cause is simple enough: Our data host ran into some problems after they upgraded a particular part of their stack, causing all attempts to access data to result in either an Unauthorized errror, or a Server Fault error.
We contacted them rather quickly, but they were already in the process of attempting to fix it.
Unfortunately, implementing that fix took some time.
Avoiding this in future is not currently possible, as we rely on the data host to provide our data. Multiple host fallback is an option in the future - but we're not currently at the point where we can financially afford to store terabytes of data across multiple hosts.
We don't expect another failure any time soon, and the host does have a good reputation for stability. This has been the worst outage in several years.
- 2021-03-10 Sherlock in Color
- 2021-01-08 Playlists
- 2020-10-24 Return of Service
- 2020-10-16 Enhanced Audio and Upload Delays
- 2020-09-10 Experimental Interface
- 2020-06-26 Gunsmith Hits HD
- 2020-06-11 Creating Something From Nothing
- 2020-03-18 Filmscope Progress
- 2020-03-10 2019 Releases
- 2020-03-05 Downtime Postmortem
- 2020-02-12 Temporary Signup Problems
- 2020-02-12 Statistics
- 2020-02-01 Search Regression
- 2020-02-01 High Load DDoS Attack
- 2020-01-30 The Phantom reCreeps
- 2020-01-25 Simple is Best
- 2020-01-06 New Features
- 2020-01-04 Displaying Credit
- 2019-12-29 Performance Enhancements
- 2019-12-18 Experimental Rendering
- 2019-12-10 MPV Support
- 2019-12-03 Introducing Filmscope
- 2019-11-21 DDoS Attack
- 2019-11-20 Security Report
- 2019-11-20 Report
- 2019-10-21 The World of Preservation
- 2019-10-15 Endless Battle For Quality
- 2019-10-06 Giving Back
- 2019-10-02 What's in a Cookie?
- 2019-10-01 PGP
- 2019-09-28 SIXTEENmm