Friday, February 3, 2012

Data loss of public pads

On Feb 2 2012 around 17:30 UTC an incident caused the complete loss of about 5200 public pads (http://titanpad.com/<padname> URL). Of those 5200 pads about 870 were accessed in the last 9 days.

The data loss was introduced while trying to resynchronize an outdated database slave server which was used for backup purposes.


UPDATE:

A backup with the state as of Jan 30 2012 00:17 UTC exists and is available at http://devpad.amd.co.at/.

This address will be reachable until Feb 14 2012 00:00, please copy the content of the pads you lost over to http://titanpad.com/, if necessary under a new name. The broken Pads on titanpad.com will be removed eventually.

Unfortunately it's not possible to recover the changes between Jan 30 and Feb 2 for the broken/lost pads, we're very sorry about this.

Offering a stable and reliable hosted EtherPad site was the main reason why we started TitanPad in the first place and compromising our vision like this doesn't sit lightly with us. We'll improve our monitoring, backup processes and operations documentation to prevent fiascos like this in the future.