I have updated the snapadm.pl file contained in my post and have also provided a download link here.

Download:

• snapadm.pl

Read on for more details:

diff snapadm.pl_.txt snapadm.pl_1.txt
181c181
<               my $result = `/sbin/zfs snapshot -r $SnapFS\@_$class_name\_$ztimestamp`;
---
>               my $result = `/sbin/zfs snapshot -r $SnapFS\@$class_name\_$ztimestamp`;

The snapshot being taken for the recursive option was incorrectly adding an _ (underscore) to the classname. This meant that when the isExpired method performs a regex search on the snapshot name then no match returns and therefore no snapshot gets destroyed.

Using several scripts to control rsync and the snapshots has allowed me to setup a system that has data retention periods applied and only uses the space on disk required to keep the changed files. Cloning a ZFS snapshot from a week ago (or even six months ago) allows you to create a synthetic full backup from the point in time the backup was taken.

To create my backup solution I had a spare Sun Thor x4540 storage system (the next model in line released after the Thumper storage systems). This system has 48 * 500gb SATA harddrives (with a fairly conservative disk setup gives about 16TB of usable disk space), the following solution could easily be implemented with any system that will support OpenSolaris as well as have enough disk space to fullfil the requirements.

I have created a Backup ZFS pool using all of the Thor’s 48 disks. This pools root ZFS system is mounted at /backup on the filesystem. Within this ZFS filesystem I created another filesystem /backup/rsync this will be where we will create all of the individual filesystems as our backup targets. You can do this with any pool, the important fact is having a ZFS filesystem called rsync (or whatever you wish really) and have ZFS filesystems under this point for each backup target.

root@backup-01:/# zpool list
NAME     SIZE   USED  AVAIL    CAP  HEALTH  ALTROOT
backup  19.0T  6.81T  12.2T    35%  ONLINE  -
rpool   14.9G  5.49G  9.38G    36%  ONLINE  -
root@backup-01:/#
root@backup-01:/#
root@backup-01:/#
root@backup-01:/# zfs list
NAME                                 USED  AVAIL  REFER  MOUNTPOINT
backup                              5.65T  9.90T  36.5K  /backup
backup/rsync                        3.29T  9.90T  53.1K  /backup/rsync
backup/rsync/dvc                    2.23T  9.90T  1.97T  /backup/rsync/dvc
backup/rsync/library                3.04G  9.90T   392M  /backup/rsync/library
backup/rsync/grp-core-cifs           451G  9.90T   436G  /backup/rsync/grp-core-cifs
backup/rsync/grp-web-frontend       7.56G  9.90T  7.50G  /backup/rsync/grp-web-frontend
backup/rsync/grp-web-mysql           632M  9.90T   254M  /backup/rsync/grp-web-mysql
rpool                               9.83G  4.82G  81.5K  /rpool
rpool/ROOT                          3.06G  4.82G    19K  legacy
rpool/ROOT/opensolaris              8.79M  4.82G  2.87G  /
rpool/ROOT/opensolaris-20100520-01  3.05G  4.82G  2.89G  /
rpool/dump                          2.22G  4.82G  2.22G  -
rpool/export                         122M  4.82G    21K  /export
rpool/export/home                    122M  4.82G    21K  /export/home
rpool/export/home/smithj             122M  4.82G   122M  /export/home/smithj
rpool/swap                          4.43G  9.15G   101M  -
root@backup-01:/#

Next step is creating the rsyncd.conf file for rsync and import the manifest for running the rsync service under OpenSolaris.

The rsyncd.conf file needs to be placed in /etc for rsync to automatically pickup the settings. You may use the following as a sample for your own rsyncd.conf

read only = yes
list = yes
uid = root
gid = root

[host-one]
path = /backup/rsync/host-one
read only = no
hosts allow = <<host-one IP Address>>
hosts deny = *

[host-two]
path = /backup/rsync/host-two
read only = no
hosts allow = <<host-two IP Address>>
hosts deny = *

I found the original manifest for rsync at http://hub.opensolaris.org/bin/view/Community+Group+smf/manifests, then made minor modifications to work out of the box. Download the modified rsync.xml file from the end of this article and put it in the /var/svc/manifests/network folder on your OpenSolaris server. Once you have done this you need to run a service import.

root@backup-01:/# svccfg import /var/svc/manifest/network/rsync.xml
root@backup-01:/# svcadm enable svc:/network/rsync:default

Then run the following to ensure there are no issues with the service

root@backup-01:/# svcs -x

When you want to add a new target for a host to backup remember to create a ZFS filesystem for that host and also add an entry into the rsyncd.conf file. Once you have updated your rsyncd.conf file you need to restart the service

root@backup-01:/# svcadm restart svc:/network/rsync:default

You can now rsync anything to your backup targets, a basic backup script for linux could be as simple as

rsync -a -v -v --progress --stats --delete --exclude "/proc" --exclude "/dev" --exclude "/sys" / backup-01::host-one

The final step is to implement a rotation/retention scheme using ZFS snapshots. The original download of snapadm.pl script came from Sun. I found a bug in this version of the script, the fixed version can be downloaded from the bottom of this article. Copy the snapadm.pl file to /usr/sbin and make the file executable.

The retention policy I have put into place is keep:

• 14 Daily Backups
• 4 Weekly Backups
• 6 Monthly Backups

To do this add the following to your root crontab

# Daily Backup - Keep 14 Days
0 12 * * * /usr/sbin/snapadm.pl -ret=14:0:0 -filesystem=backup/rsync -rec -cl=daily

# Weekly Backup - Keep 4 Weeks
0 13 * * 0 /usr/sbin/snapadm.pl -ret=28:0:0 -filesystem=backup/rsync -rec -cl=weekly

# Monthly Backup - Keep 6 Months
0 14 1 * * /usr/sbin/snapadm.pl -ret=182:0:0 -filesystem=backup/rsync -rec -cl=monthly

Each line contains a different backup type. Daily with a retention of 14 days. Weekly each Sunday with a retention of 28 days (4 weeks). Monthly on the first of the month with a retention of 182 days (half a year).

The Options for snapadm.pl used for the script

• -ret=Days:Hours:Minutes
• -filesystem=ZFS File System Path
• -rec – Recursively snapshot down the tree
• -cl – Class, just a way to label the backup type

The end result is the following list of snapshots

root@backup-01:~# zfs list -t snapshot
NAME                                                        USED  AVAIL  REFER  MOUNTPOINT
backup/rsync@daily_Thu_20100520_1445                      26.6K      -  33.2K  -
backup/rsync@daily_Fri_20100521_1200                          0      -  41.5K  -
backup/rsync@daily_Sat_20100522_1200                          0      -  41.5K  -
backup/rsync@daily_Sun_20100523_1200                          0      -  41.5K  -
backup/rsync@weekly_Sun_20100523_1300                         0      -  41.5K  -
backup/rsync@daily_Mon_20100524_1200                      28.2K      -  46.5K  -
backup/rsync@daily_Tue_20100525_1200                      26.6K      -  48.1K  -
backup/rsync@daily_Wed_20100526_1200                      26.6K      -  48.1K  -
backup/rsync@daily_Thu_20100527_1200                          0      -  48.1K  -
backup/rsync@daily_Fri_20100528_1200                          0      -  48.1K  -
backup/rsync@daily_Sat_20100529_1200                          0      -  53.1K  -
backup/rsync@daily_Sun_20100530_1200                          0      -  53.1K  -
backup/rsync@weekly_Sun_20100530_1300                         0      -  53.1K  -
backup/rsync@daily_Mon_20100531_1200                          0      -  53.1K  -
backup/rsync@daily_Tue_20100601_1200                          0      -  53.1K  -
backup/rsync@monthly_Tue_20100601_1400                        0      -  53.1K  -
backup/rsync@daily_Wed_20100602_1200                          0      -  53.1K  -
backup/rsync/dvc@daily_Tue_20100525_1200                   287K      -  2.06T  -
backup/rsync/dvc@daily_Wed_20100526_1200                  5.75G      -  2.07T  -
backup/rsync/dvc@daily_Thu_20100527_1200                  26.5G      -  2.02T  -
backup/rsync/dvc@daily_Fri_20100528_1200                  17.2G      -  2.00T  -
backup/rsync/dvc@daily_Sat_20100529_1200                  3.72M      -  1.97T  -
backup/rsync/dvc@daily_Sun_20100530_1200                      0      -  2.00T  -
backup/rsync/dvc@weekly_Sun_20100530_1300                     0      -  2.00T  -
backup/rsync/dvc@daily_Mon_20100531_1200                  27.3G      -  2.03T  -
backup/rsync/dvc@daily_Tue_20100601_1200                      0      -  1.97T  -
backup/rsync/dvc@monthly_Tue_20100601_1400                    0      -  1.97T  -
backup/rsync/dvc@daily_Wed_20100602_1200                   184K      -  1.96T  -
backup/rsync/library@daily_Fri_20100521_1200                  0      -  31.5K  -
backup/rsync/library@daily_Sat_20100522_1200                  0      -  31.5K  -
backup/rsync/library@daily_Sun_20100523_1200                  0      -  31.5K  -
backup/rsync/library@weekly_Sun_20100523_1300                 0      -  31.5K  -
backup/rsync/library@daily_Mon_20100524_1200              24.9K      -  31.5K  -
backup/rsync/library@daily_Tue_20100525_1200               390M      -   390M  -
backup/rsync/library@daily_Wed_20100526_1200               389M      -   389M  -
backup/rsync/library@daily_Thu_20100527_1200               384M      -   384M  -
backup/rsync/library@daily_Fri_20100528_1200               388M      -   388M  -
backup/rsync/library@daily_Sat_20100529_1200               388M      -   388M  -
backup/rsync/library@daily_Sun_20100530_1200                  0      -   389M  -
backup/rsync/library@weekly_Sun_20100530_1300                 0      -   389M  -
backup/rsync/library@daily_Mon_20100531_1200               391M      -   391M  -
backup/rsync/library@daily_Tue_20100601_1200                  0      -   392M  -
backup/rsync/library@monthly_Tue_20100601_1400                0      -   392M  -
backup/rsync/library@daily_Wed_20100602_1200              24.9K      -   391M  -
backup/rsync/grp-core-cifs@daily_Fri_20100521_1200        13.7M      -   301G  -
backup/rsync/grp-core-cifs@daily_Sat_20100522_1200         278M      -   435G  -
backup/rsync/grp-core-cifs@daily_Sun_20100523_1200            0      -   436G  -
backup/rsync/grp-core-cifs@weekly_Sun_20100523_1300           0      -   436G  -
backup/rsync/grp-core-cifs@daily_Mon_20100524_1200         373M      -   435G  -
backup/rsync/grp-core-cifs@daily_Tue_20100525_1200         501M      -   434G  -
backup/rsync/grp-core-cifs@daily_Wed_20100526_1200         486M      -   434G  -
backup/rsync/grp-core-cifs@daily_Thu_20100527_1200         492M      -   434G  -
backup/rsync/grp-core-cifs@daily_Fri_20100528_1200         439M      -   436G  -
backup/rsync/grp-core-cifs@daily_Sat_20100529_1200         500M      -   436G  -
backup/rsync/grp-core-cifs@daily_Sun_20100530_1200            0      -   436G  -
backup/rsync/grp-core-cifs@weekly_Sun_20100530_1300           0      -   436G  -
backup/rsync/grp-core-cifs@daily_Mon_20100531_1200         381M      -   436G  -
backup/rsync/grp-core-cifs@daily_Tue_20100601_1200            0      -   436G  -
backup/rsync/grp-core-cifs@monthly_Tue_20100601_1400          0      -   436G  -
backup/rsync/grp-core-cifs@daily_Wed_20100602_1200        38.1M      -   437G  -
backup/rsync/grp-web-frontend@daily_Fri_20100521_1200     4.57M      -  7.49G  -
backup/rsync/grp-web-frontend@daily_Sat_20100522_1200     4.95M      -  7.49G  -
backup/rsync/grp-web-frontend@daily_Sun_20100523_1200         0      -  7.49G  -
backup/rsync/grp-web-frontend@weekly_Sun_20100523_1300        0      -  7.49G  -
backup/rsync/grp-web-frontend@daily_Mon_20100524_1200     5.44M      -  7.49G  -
backup/rsync/grp-web-frontend@daily_Tue_20100525_1200     5.59M      -  7.49G  -
backup/rsync/grp-web-frontend@daily_Wed_20100526_1200     5.45M      -  7.50G  -
backup/rsync/grp-web-frontend@daily_Thu_20100527_1200     5.43M      -  7.50G  -
backup/rsync/grp-web-frontend@daily_Fri_20100528_1200     5.57M      -  7.50G  -
backup/rsync/grp-web-frontend@daily_Sat_20100529_1200     5.56M      -  7.50G  -
backup/rsync/grp-web-frontend@daily_Sun_20100530_1200         0      -  7.50G  -
backup/rsync/grp-web-frontend@weekly_Sun_20100530_1300        0      -  7.50G  -
backup/rsync/grp-web-frontend@daily_Mon_20100531_1200     5.56M      -  7.50G  -
backup/rsync/grp-web-frontend@daily_Tue_20100601_1200         0      -  7.50G  -
backup/rsync/grp-web-frontend@monthly_Tue_20100601_1400       0      -  7.50G  -
backup/rsync/grp-web-frontend@daily_Wed_20100602_1200      179K      -  7.50G  -
backup/rsync/grp-web-mysql@daily_Thu_20100520_1445        35.7M      -   257M  -
backup/rsync/grp-web-mysql@daily_Fri_20100521_1200        29.4M      -   251M  -
backup/rsync/grp-web-mysql@daily_Sat_20100522_1200        29.6M      -   251M  -
backup/rsync/grp-web-mysql@daily_Sun_20100523_1200            0      -   251M  -
backup/rsync/grp-web-mysql@weekly_Sun_20100523_1300           0      -   251M  -
backup/rsync/grp-web-mysql@daily_Mon_20100524_1200        29.8M      -   251M  -
backup/rsync/grp-web-mysql@daily_Tue_20100525_1200        30.1M      -   251M  -
backup/rsync/grp-web-mysql@daily_Wed_20100526_1200        30.4M      -   252M  -
backup/rsync/grp-web-mysql@daily_Thu_20100527_1200        30.6M      -   252M  -
backup/rsync/grp-web-mysql@daily_Fri_20100528_1200        32.9M      -   254M  -
backup/rsync/grp-web-mysql@daily_Sat_20100529_1200        33.1M      -   254M  -
backup/rsync/grp-web-mysql@daily_Sun_20100530_1200            0      -   254M  -
backup/rsync/grp-web-mysql@weekly_Sun_20100530_1300           0      -   254M  -
backup/rsync/grp-web-mysql@daily_Mon_20100531_1200        33.3M      -   254M  -
backup/rsync/grp-web-mysql@daily_Tue_20100601_1200            0      -   254M  -
backup/rsync/grp-web-mysql@monthly_Tue_20100601_1400          0      -   254M  -
backup/rsync/grp-web-mysql@daily_Wed_20100602_1200        79.7K      -   254M  -
rpool/ROOT/opensolaris-20100520-01@install                  148M      -  2.82G  -
rpool/ROOT/opensolaris-20100520-01@2010-05-19-21:49:46     13.8M      -  2.87G  -
root@backup-01:~#

To do a file restore you need to make a ZFS Clone of the snapshot, this will allow you to copy the files required. The most important thing is that you MUST destroy the clone once you have finished the restore, failure to do so will result in the snapshot being unable to be deleted by the retention policy.

I certainly recommend reading the Solaris ZFS Administrators Guide and the ZFS Best Practices Guide. Both these guide certainly helped me understand ZFS and how to manage it.

Download:

• Rsync Manifest – rsync.xml
• Snapshot Admin Script – snapadm.pl

I’m a little bit sad that I saw this. Australia day is always about listening all day to find out what hits number one (then being outraged when it isn’t a song you like, having said that I like this song)

BTW i take no credit for the image or finding the leak, I am just sharing it, original at http://i49.tinypic.com/ab2zrk.jpg

I have been waiting for the ESM tools for Exchange 2010 to hopefully work with Exchange 2003 but that hasn’t happened either. Although I did find a really quick and easy solution this morning after reading the following posts on TechNet (Exchange System Manager for exchange 2003)

You will need to download the Exchange System Manager for Windows Vista then use something like WinRAR to extract the contents.

You will also need to make sure you already have the RSAT tools installed (Remote Server Admin Tools)

Finally open up a Command Prompt as Administrator. Browse to the directory that contains the ESMVISTA.MSI file. Run the following:

ESMVISTA.MSI /q

Once this is done you should be able to open the “Active Directory and Computers” and edit an Exchange User with all the required email tabs.

After searching through forums and mailing lists I was unable to find an answer so I started looking into mplayer itself. If i used mplayer on it’s own while Mythtv was running the same problem would occur. If I exited Mythtv then mplayer functioned perfectly. I found a possible idea from the Ubuntu forums whereby someone else was having slight playback issues (no mention of Mythtv though).

The problem appears to be with the audio driver mplayer tries to use. On my system it comes up with an error about pulse (that is something to fix some other time). To fix this I told mplayer to use ALSA instead, and it worked!!

So to fix with MythTV go into the Setup -> Media Settings -> Video Settings -> Player Settings then set Default Video Player to

mplayer -fs -zoom -ontop -ao alsa %s

Alternatively you could just add

-ao alsa

before the %s.

The upgrade the VMWare ESX 4.0 Update 1 ran smoothly and all hosts came back to life after the upgrade as they should. Installation of the client on Windows 7 was also painless, and best of all it actually worked.

To anyone from VMWare who may be watching, it would be nice if you had some form of announcements mailing list so we could be informed of these updates being made available.

At this time I wrote my own craptacular mashup called The Gob Shite, this site was a mashup of Google’s YouTube, Google’s News feeds and a new API of my own called the TrendEngine. TrendEngine was developed using Google Hot Trends information and provided some level of stability to the constantly changing trend list. It does this by detecting reoccuring trends in the list and applying a weighting formula to them.

TrendEngine currently operates from a private server written in PHP, I am currently converting this code to Java to run on Google’s App Engine. The new TrendEngine service will be publicly released shortly and will be listed up on ProgrammableWeb for all to use.

More information to come soon.

There are certain things that when it doesn’t work you wouldn’t be surprised. We all have that application that hasn’t been updated in years but we still need to use on a regular basis. I can somewhat understand why these things don’t work within a new operating system. But for something that gets updated on a regular basis to not work, that is a different story altogether and that really annoys me.

Given that some form of Windows 7 has been available to vendors for a long time it shows VMWare with a high level of contempt for it’s users. There is absolutely no excuse for there not to be an update to support Windows 7, especially considering there have been reports of it not working from early RC days.

The official response from VMWare (KB 1011329) is to just use a XP virtual machine. There is not even a mention on how long they will take to fix the issue.

Kiwi Si at Techhead has found a better workaround but still while this gets things working there should be a better option. We all pay a lot of money for VMWare and as such should deserve better treatment.

This release bring further customisation ability for those who need it. You can now take full control of how you want Custom Menu Links to operate, fully automatic integration or manual integration (modifying your theme for LJ Custom Menu Links).

Changes

• Fixed compatibility issues by adding the ability to disable the core wordpress override and manually add the LJ Custom Menu Links code to your theme

Wordpress Plugin Directory: LJ Custom Menu Links

I run 4gb of ram in my laptop so I though that I would go 64 bit (and get back that extra 600 or so mb of ram that gets stolen on 32bit machines). The initial install went well, including the install and subsequent windows update yielded a full set drivers for my machine (except the fingerprint sensor, for which Dell haven’t released a Windows 7 driver for yet either).

Everything started to fall apart when I needed to install the Cisco VPN Client. No deal, the Cisco VPN Client wont install on 64 bit windows installs. After some research I found that people were having success using the new XP Mode feature of Windows 7. XP Mode allows you to run an XP virtual machine on select hardware (needing CPU virtualization support). I quickly realised that this wouldn’t be very practical as you can only access the VPN resources from within the virtual machine (I also had limited success in actually getting the Cisco client installed in the XP Mode machine anyway).

So that was it. I need VPN access for my work, and 64 bit was getting in the way of that. Back to 32 bit I go. The installation ran just as smoothly as the 64 bit install, and once again with a full suite of drivers (still with the exception of the fingerprint reader, but that’s not a big loss).

It may be just me but the 32 bit installation even feels like it is performing better then the 64 bit installation, windows feel snappier to move around, open and close and the machine is quicker to boot as well as resume from hibernation. The question that I am left with is that we can run 32 bit applications on a 64 bit platform with wow (Windows on Windows) so why can’t there be a similar situation for drivers. I understand there would be a performance hit for doing this but with 64 bit being ultimately pushed as the way forward there needs to be a solution to allow legacy applications and devices to operate.