In addition to storing backups in HDFS, we also write them to an off-site location. All of our database servers also use global transaction IDs (GTIDs), which gives us another layer of control when replaying transactions from binlog backups. This then enables us to perform point-in-time restores by replaying transactions from binlogs on top of a restored dump to bring the database’s state up to a certain point in time. Since dumps are only taken once a day, having binlog backups means we get full coverage of every transaction executed against the database being backed up. I’ll refer to diff and full backups as dumps for the remainder of this post. Binary log (binlog) backups are constantly streamed to HDFS from the database master.īoth full and diff backups pass the -single-transaction option to mysqldump so we get a consistent snapshot of the database and can both be taken from secondary as well as primary instances.Metadata records which full backup the diff was created against. We do these by taking another full dump and only storing the difference between it and the last full backup that was taken. Differential (diff) backups are taken on days we don’t take full backups.Full logical backups are taken every few days using mysqldump.Currently, we take three kinds of backups, all of which are stored in HDFS: Backups overviewīefore building a continuous restore pipeline, we first need to understand the nature of backups available to us. This post will focus on ORC’s internals - specifically the internal peon state machine and some of the challenges we had to overcome when coming up with the restore process for a single database. The data CRT collects about each restore job’s progress helps us understand our resource requirements for database restores, and ORC helps us verify backup integrity. Peons host a local MySQL instance where they perform the actual restore.

The warchief receives new restore jobs from CRT and assigns them to peons. ORC Restore Coordinator (ORC) – Comprises the restore workers (peons) and a load balancer (warchief).It looks for databases with new backups and creates restore jobs for them, monitors the restore’s progress, and ensures each backup is successfully restored. Continuous Restore Tier (CRT) – Responsible for all scheduling and monitoring around restores.Our restore system consists of two main components: To that end, we’ve built a system that continuously tests our ability to restore our databases from backups.

In such a disaster event, not only do we have to recover the service as quickly and reliably as possible, but we also need to ensure we don’t lose data in the process. Facebook’s MySQL databases are spread across our global data centers, and we need to be able to recover from an outage in any of these locations, at any given point in time.

Mysql restore backup