This document briefly outlines strategies for backing up a Rhino cluster install. This includes backing up the Rhino SLEE itself as well as any PostgreSQL instances.
There exists a number of options depending on what type of backup/restore functionality is required.
- If you are using something like LVM at the OS level, you can create a snapshot volume of the filesystems holding the PostgreSQL data files and the Rhino installations and back up from the snapshot. PostgreSQL may need to perform recovery on restore. This backup preserves everything exactly, in a machine-readable form only.
- You can back up the contents of the PostgreSQL management database using the standard PostgreSQL tools (e.g. pg_dump). To restore from this after a complete cluster failure, you will need to reinstall Rhino by hand, then restore the database contents. This backup preserves all management state in a machine-readable form only. On restore, you must reinstall exactly the same version of Rhino that the backup was made from.
- You can export the entire current state of the cluster using the 'rhino-export' tool. To restore this, you would reinstall Rhino and import the saved export. This backup preserves all management state in a human-readable form that is fairly easy to modify or examine if needed - it is essentially an 'ant' build script that performs management commands to restore the entire cluster state. You can also import this type of backup into a more recent version of Rhino for upgrade purposes.
Additionally you will have to handle backups of any other modifications or generated output that are not known to the Rhino management system, e.g. modifications to files in the Rhino install. This is especially true of (2) or (3).
(1) can be done on a per-node, per-database basis. (2) and (3) are essentially preserving cluster-wide state, so you can't use them to quickly recover a single node - if a single node failed, you would simply reinstall Rhino on new hardware and allow it to join the existing cluster and it would recover state from there.
The actual strategies adopted do depends a lot on your backup requirements and the layout of your Rhino nodes vs. PostgreSQL instance.
A common approach is to use both (1) and (3) - (1) to deal with recovering from individual node failure, and (3) to deal with recovering from catastrophic cluster failure, data corruption, rolling back to an earlier point in time, and for upgrades.