OpenSolaris Backup Solution with Rsync and ZFS Snapshots

Jon Smith — Wed, 02 Jun 2010 10:37:32 +0000

I have recently been forced to learn how to admin an OpenSolaris machine with ZFS filesystems. After doing some reading on ZFS, I realised how using snapshots with an rsync server would make an excellent backup solution.

Using several scripts to control rsync and the snapshots has allowed me to setup a system that has data retention periods applied and only uses the space on disk required to keep the changed files. Cloning a ZFS snapshot from a week ago (or even six months ago) allows you to create a synthetic full backup from the point in time the backup was taken.

To create my backup solution I had a spare Sun Thor x4540 storage system (the next model in line released after the Thumper storage systems). This system has 48 * 500gb SATA harddrives (with a fairly conservative disk setup gives about 16TB of usable disk space), the following solution could easily be implemented with any system that will support OpenSolaris as well as have enough disk space to fullfil the requirements.

I have created a Backup ZFS pool using all of the Thor’s 48 disks. This pools root ZFS system is mounted at /backup on the filesystem. Within this ZFS filesystem I created another filesystem /backup/rsync this will be where we will create all of the individual filesystems as our backup targets. You can do this with any pool, the important fact is having a ZFS filesystem called rsync (or whatever you wish really) and have ZFS filesystems under this point for each backup target.

root@backup-01:/# zpool list
NAME     SIZE   USED  AVAIL    CAP  HEALTH  ALTROOT
backup  19.0T  6.81T  12.2T    35%  ONLINE  -
rpool   14.9G  5.49G  9.38G    36%  ONLINE  -
root@backup-01:/#
root@backup-01:/#
root@backup-01:/#
root@backup-01:/# zfs list
NAME                                 USED  AVAIL  REFER  MOUNTPOINT
backup                              5.65T  9.90T  36.5K  /backup
backup/rsync                        3.29T  9.90T  53.1K  /backup/rsync
backup/rsync/dvc                    2.23T  9.90T  1.97T  /backup/rsync/dvc
backup/rsync/library                3.04G  9.90T   392M  /backup/rsync/library
backup/rsync/grp-core-cifs           451G  9.90T   436G  /backup/rsync/grp-core-cifs
backup/rsync/grp-web-frontend       7.56G  9.90T  7.50G  /backup/rsync/grp-web-frontend
backup/rsync/grp-web-mysql           632M  9.90T   254M  /backup/rsync/grp-web-mysql
rpool                               9.83G  4.82G  81.5K  /rpool
rpool/ROOT                          3.06G  4.82G    19K  legacy
rpool/ROOT/opensolaris              8.79M  4.82G  2.87G  /
rpool/ROOT/opensolaris-20100520-01  3.05G  4.82G  2.89G  /
rpool/dump                          2.22G  4.82G  2.22G  -
rpool/export                         122M  4.82G    21K  /export
rpool/export/home                    122M  4.82G    21K  /export/home
rpool/export/home/smithj             122M  4.82G   122M  /export/home/smithj
rpool/swap                          4.43G  9.15G   101M  -
root@backup-01:/#

Next step is creating the rsyncd.conf file for rsync and import the manifest for running the rsync service under OpenSolaris.

The rsyncd.conf file needs to be placed in /etc for rsync to automatically pickup the settings. You may use the following as a sample for your own rsyncd.conf

read only = yes
list = yes
uid = root
gid = root

[host-one]
path = /backup/rsync/host-one
read only = no
hosts allow = <>
hosts deny = *

[host-two]
path = /backup/rsync/host-two
read only = no
hosts allow = <>
hosts deny = *

I found the original manifest for rsync at http://hub.opensolaris.org/bin/view/Community+Group+smf/manifests, then made minor modifications to work out of the box. Download the modified rsync.xml file from the end of this article and put it in the /var/svc/manifests/network folder on your OpenSolaris server. Once you have done this you need to run a service import.

root@backup-01:/# svccfg import /var/svc/manifest/network/rsync.xml
root@backup-01:/# svcadm enable svc:/network/rsync:default

Then run the following to ensure there are no issues with the service

root@backup-01:/# svcs -x

When you want to add a new target for a host to backup remember to create a ZFS filesystem for that host and also add an entry into the rsyncd.conf file. Once you have updated your rsyncd.conf file you need to restart the service

root@backup-01:/# svcadm restart svc:/network/rsync:default

You can now rsync anything to your backup targets, a basic backup script for linux could be as simple as

rsync -a -v -v --progress --stats --delete --exclude "/proc" --exclude "/dev" --exclude "/sys" / backup-01::host-one

The final step is to implement a rotation/retention scheme using ZFS snapshots. The original download of snapadm.pl script came from Sun. I found a bug in this version of the script, the fixed version can be downloaded from the bottom of this article. Copy the snapadm.pl file to /usr/sbin and make the file executable.

The retention policy I have put into place is keep:

14 Daily Backups
4 Weekly Backups
6 Monthly Backups

To do this add the following to your root crontab

# Daily Backup - Keep 14 Days
0 12 * * * /usr/sbin/snapadm.pl -ret=14:0:0 -filesystem=backup/rsync -rec -cl=daily

# Weekly Backup - Keep 4 Weeks
0 13 * * 0 /usr/sbin/snapadm.pl -ret=28:0:0 -filesystem=backup/rsync -rec -cl=weekly

# Monthly Backup - Keep 6 Months
0 14 1 * * /usr/sbin/snapadm.pl -ret=182:0:0 -filesystem=backup/rsync -rec -cl=monthly

Each line contains a different backup type. Daily with a retention of 14 days. Weekly each Sunday with a retention of 28 days (4 weeks). Monthly on the first of the month with a retention of 182 days (half a year).

The Options for snapadm.pl used for the script

-ret=Days:Hours:Minutes
-filesystem=ZFS File System Path
-rec – Recursively snapshot down the tree
-cl – Class, just a way to label the backup type

The end result is the following list of snapshots

root@backup-01:~# zfs list -t snapshot
NAME                                                        USED  AVAIL  REFER  MOUNTPOINT
backup/rsync@daily_Thu_20100520_1445                      26.6K      -  33.2K  -
backup/rsync@daily_Fri_20100521_1200                          0      -  41.5K  -
backup/rsync@daily_Sat_20100522_1200                          0      -  41.5K  -
backup/rsync@daily_Sun_20100523_1200                          0      -  41.5K  -
backup/rsync@weekly_Sun_20100523_1300                         0      -  41.5K  -
backup/rsync@daily_Mon_20100524_1200                      28.2K      -  46.5K  -
backup/rsync@daily_Tue_20100525_1200                      26.6K      -  48.1K  -
backup/rsync@daily_Wed_20100526_1200                      26.6K      -  48.1K  -
backup/rsync@daily_Thu_20100527_1200                          0      -  48.1K  -
backup/rsync@daily_Fri_20100528_1200                          0      -  48.1K  -
backup/rsync@daily_Sat_20100529_1200                          0      -  53.1K  -
backup/rsync@daily_Sun_20100530_1200                          0      -  53.1K  -
backup/rsync@weekly_Sun_20100530_1300                         0      -  53.1K  -
backup/rsync@daily_Mon_20100531_1200                          0      -  53.1K  -
backup/rsync@daily_Tue_20100601_1200                          0      -  53.1K  -
backup/rsync@monthly_Tue_20100601_1400                        0      -  53.1K  -
backup/rsync@daily_Wed_20100602_1200                          0      -  53.1K  -
backup/rsync/dvc@daily_Tue_20100525_1200                   287K      -  2.06T  -
backup/rsync/dvc@daily_Wed_20100526_1200                  5.75G      -  2.07T  -
backup/rsync/dvc@daily_Thu_20100527_1200                  26.5G      -  2.02T  -
backup/rsync/dvc@daily_Fri_20100528_1200                  17.2G      -  2.00T  -
backup/rsync/dvc@daily_Sat_20100529_1200                  3.72M      -  1.97T  -
backup/rsync/dvc@daily_Sun_20100530_1200                      0      -  2.00T  -
backup/rsync/dvc@weekly_Sun_20100530_1300                     0      -  2.00T  -
backup/rsync/dvc@daily_Mon_20100531_1200                  27.3G      -  2.03T  -
backup/rsync/dvc@daily_Tue_20100601_1200                      0      -  1.97T  -
backup/rsync/dvc@monthly_Tue_20100601_1400                    0      -  1.97T  -
backup/rsync/dvc@daily_Wed_20100602_1200                   184K      -  1.96T  -
backup/rsync/library@daily_Fri_20100521_1200                  0      -  31.5K  -
backup/rsync/library@daily_Sat_20100522_1200                  0      -  31.5K  -
backup/rsync/library@daily_Sun_20100523_1200                  0      -  31.5K  -
backup/rsync/library@weekly_Sun_20100523_1300                 0      -  31.5K  -
backup/rsync/library@daily_Mon_20100524_1200              24.9K      -  31.5K  -
backup/rsync/library@daily_Tue_20100525_1200               390M      -   390M  -
backup/rsync/library@daily_Wed_20100526_1200               389M      -   389M  -
backup/rsync/library@daily_Thu_20100527_1200               384M      -   384M  -
backup/rsync/library@daily_Fri_20100528_1200               388M      -   388M  -
backup/rsync/library@daily_Sat_20100529_1200               388M      -   388M  -
backup/rsync/library@daily_Sun_20100530_1200                  0      -   389M  -
backup/rsync/library@weekly_Sun_20100530_1300                 0      -   389M  -
backup/rsync/library@daily_Mon_20100531_1200               391M      -   391M  -
backup/rsync/library@daily_Tue_20100601_1200                  0      -   392M  -
backup/rsync/library@monthly_Tue_20100601_1400                0      -   392M  -
backup/rsync/library@daily_Wed_20100602_1200              24.9K      -   391M  -
backup/rsync/grp-core-cifs@daily_Fri_20100521_1200        13.7M      -   301G  -
backup/rsync/grp-core-cifs@daily_Sat_20100522_1200         278M      -   435G  -
backup/rsync/grp-core-cifs@daily_Sun_20100523_1200            0      -   436G  -
backup/rsync/grp-core-cifs@weekly_Sun_20100523_1300           0      -   436G  -
backup/rsync/grp-core-cifs@daily_Mon_20100524_1200         373M      -   435G  -
backup/rsync/grp-core-cifs@daily_Tue_20100525_1200         501M      -   434G  -
backup/rsync/grp-core-cifs@daily_Wed_20100526_1200         486M      -   434G  -
backup/rsync/grp-core-cifs@daily_Thu_20100527_1200         492M      -   434G  -
backup/rsync/grp-core-cifs@daily_Fri_20100528_1200         439M      -   436G  -
backup/rsync/grp-core-cifs@daily_Sat_20100529_1200         500M      -   436G  -
backup/rsync/grp-core-cifs@daily_Sun_20100530_1200            0      -   436G  -
backup/rsync/grp-core-cifs@weekly_Sun_20100530_1300           0      -   436G  -
backup/rsync/grp-core-cifs@daily_Mon_20100531_1200         381M      -   436G  -
backup/rsync/grp-core-cifs@daily_Tue_20100601_1200            0      -   436G  -
backup/rsync/grp-core-cifs@monthly_Tue_20100601_1400          0      -   436G  -
backup/rsync/grp-core-cifs@daily_Wed_20100602_1200        38.1M      -   437G  -
backup/rsync/grp-web-frontend@daily_Fri_20100521_1200     4.57M      -  7.49G  -
backup/rsync/grp-web-frontend@daily_Sat_20100522_1200     4.95M      -  7.49G  -
backup/rsync/grp-web-frontend@daily_Sun_20100523_1200         0      -  7.49G  -
backup/rsync/grp-web-frontend@weekly_Sun_20100523_1300        0      -  7.49G  -
backup/rsync/grp-web-frontend@daily_Mon_20100524_1200     5.44M      -  7.49G  -
backup/rsync/grp-web-frontend@daily_Tue_20100525_1200     5.59M      -  7.49G  -
backup/rsync/grp-web-frontend@daily_Wed_20100526_1200     5.45M      -  7.50G  -
backup/rsync/grp-web-frontend@daily_Thu_20100527_1200     5.43M      -  7.50G  -
backup/rsync/grp-web-frontend@daily_Fri_20100528_1200     5.57M      -  7.50G  -
backup/rsync/grp-web-frontend@daily_Sat_20100529_1200     5.56M      -  7.50G  -
backup/rsync/grp-web-frontend@daily_Sun_20100530_1200         0      -  7.50G  -
backup/rsync/grp-web-frontend@weekly_Sun_20100530_1300        0      -  7.50G  -
backup/rsync/grp-web-frontend@daily_Mon_20100531_1200     5.56M      -  7.50G  -
backup/rsync/grp-web-frontend@daily_Tue_20100601_1200         0      -  7.50G  -
backup/rsync/grp-web-frontend@monthly_Tue_20100601_1400       0      -  7.50G  -
backup/rsync/grp-web-frontend@daily_Wed_20100602_1200      179K      -  7.50G  -
backup/rsync/grp-web-mysql@daily_Thu_20100520_1445        35.7M      -   257M  -
backup/rsync/grp-web-mysql@daily_Fri_20100521_1200        29.4M      -   251M  -
backup/rsync/grp-web-mysql@daily_Sat_20100522_1200        29.6M      -   251M  -
backup/rsync/grp-web-mysql@daily_Sun_20100523_1200            0      -   251M  -
backup/rsync/grp-web-mysql@weekly_Sun_20100523_1300           0      -   251M  -
backup/rsync/grp-web-mysql@daily_Mon_20100524_1200        29.8M      -   251M  -
backup/rsync/grp-web-mysql@daily_Tue_20100525_1200        30.1M      -   251M  -
backup/rsync/grp-web-mysql@daily_Wed_20100526_1200        30.4M      -   252M  -
backup/rsync/grp-web-mysql@daily_Thu_20100527_1200        30.6M      -   252M  -
backup/rsync/grp-web-mysql@daily_Fri_20100528_1200        32.9M      -   254M  -
backup/rsync/grp-web-mysql@daily_Sat_20100529_1200        33.1M      -   254M  -
backup/rsync/grp-web-mysql@daily_Sun_20100530_1200            0      -   254M  -
backup/rsync/grp-web-mysql@weekly_Sun_20100530_1300           0      -   254M  -
backup/rsync/grp-web-mysql@daily_Mon_20100531_1200        33.3M      -   254M  -
backup/rsync/grp-web-mysql@daily_Tue_20100601_1200            0      -   254M  -
backup/rsync/grp-web-mysql@monthly_Tue_20100601_1400          0      -   254M  -
backup/rsync/grp-web-mysql@daily_Wed_20100602_1200        79.7K      -   254M  -
rpool/ROOT/opensolaris-20100520-01@install                  148M      -  2.82G  -
rpool/ROOT/opensolaris-20100520-01@2010-05-19-21:49:46     13.8M      -  2.87G  -
root@backup-01:~#

To do a file restore you need to make a ZFS Clone of the snapshot, this will allow you to copy the files required. The most important thing is that you MUST destroy the clone once you have finished the restore, failure to do so will result in the snapshot being unable to be deleted by the retention policy.

I certainly recommend reading the Solaris ZFS Administrators Guide and the ZFS Best Practices Guide. Both these guide certainly helped me understand ZFS and how to manage it.

Download:

Rsync Manifest – rsync.xml
Snapshot Admin Script – snapadm.pl

pg_standby triggering unexpectedly

Jon Smith — Fri, 21 Aug 2009 05:57:10 +0000

I have been having a problem with my redundant PostgreSQL setup over the last couple of weeks. I run two Postgres servers with a WAL Logging Warm Standby system using pg_standby from 8.3. When the master server is under heavy load and generating a lot of WAL traffic the secondary server has been tripping and going active.

After checking all the logs and putting pg_standby in debug mode there was still no clue as to why this was happening.

The archive_command on my master server was using

cp -i "%p" /var/lib/pgsql/slave/pg_logrestore/"%f"

I have read somewhere (I can’t remember where or when) that when you use cp to copy the WAL log the file appears in the destination without it being a complete copy so this got me to wondering if this was causing my problem. Knowing that RSync will use a temp file when copying and then rename the file once completed I thought I would give that a go.

My new archive_command is

rsync -q "%p" /var/lib/pgsql/slave/pg_logrestore/"%f"

Since making this change the standby server hasn’t triggered without reason and the problem appears to be fixed.

The Lazy Sys Admin» RSync

OpenSolaris Backup Solution with Rsync and ZFS Snapshots

pg_standby triggering unexpectedly