OK, whoops, this is when you realise that the machines has the wrong image id assigned to them. Easy you think, you created the task as a group you can just as easily delete the active task as a group. No such luck here, what FOG will do to a group of computers is simply setup an individual task for each and every machine in the group. So your only option now is to click delete on each active task, now multiply this out to 1000 machines you have an annoying problem.

There is a simple solution, although a word of warning here “PLEASE MAKE SURE YOU HAVE A BACKUP”. All the tasks are contained within the “tasks” table within MySQL. So if you feel comfortable crack open a console connection and open up a console to MySQL.

root@fog:/# mysql -u root
Welcome to the MySQL monitor.  Commands end with ; or g.
Server version: 5.0.67-0ubuntu6.1 (Ubuntu)

Type 'help;' or 'h' for help. Type 'c' to clear the buffer.

mysql> use fog;
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Database changed
mysql> delete from tasks where taskState = 0;
Query OK, 8 rows affected (0.23 sec)

mysql>

This will delete all the active tasks running on the fog server. However the command files will still exist in the TFTPROOT folder. Exit out from MySQL and change directory into your TFTPROOT folder. Once inside change into the pxelinux.cfg folder.

You will notice several files in this directory, these are the files that control what a machine will do once it is PXE booted. To delete the command files you ONLY have to delete the files starting with “01-”, this is import as the other files will tell the machines what to do when they don’t have any active tasks.

root@fog:/tftpboot/pxelinux.cfg# rm 01-*

This will have completely removed all the active tasks.

SUSE Studio allows you to build/create your own Linux distribution based on SUSE linux.

“Build an appliance — or your own custom Linux distro — with a few mouse clicks. Customize it to your heart’s content, and share it with the world!”

You can currently choose from one of many options available to create a OpenSUSE, SLES 10, SLED 10, SLES 11, or SLED 11 machine. You can create a custom boot menu with the option to upload your own image for bootsplash. Complete control over packages, ability to overlay files (this means you can inject scripts or your own program you are distributing) right into the machine.

Once you have completed creating your custom machine you can build it into a Live CD/DVD or a disk image (this can be DD’d onto a USB Stick or another for booting) or a VMWare Disk Image or a XEN Disk Image.

From here your SUSE Studio machine is just a quick download away, the machine I test built was just a 220MB download and on a 20mbit internet connection I was able to download my image at roughly 550kb/sec and took around 15 minutes.

This is a very well polished service and is currently still in beta. I can’t wait to see what these guys come up with next.

After checking all the logs and putting pg_standby in debug mode there was still no clue as to why this was happening.

The archive_command on my master server was using

cp -i "%p" /var/lib/pgsql/slave/pg_logrestore/"%f"

I have read somewhere (I can’t remember where or when) that when you use cp to copy the WAL log the file appears in the destination without it being a complete copy so this got me to wondering if this was causing my problem. Knowing that RSync will use a temp file when copying and then rename the file once completed I thought I would give that a go.

My new archive_command is

rsync -q "%p" /var/lib/pgsql/slave/pg_logrestore/"%f"

Since making this change the standby server hasn’t triggered without reason and the problem appears to be fixed.

I kept getting the following error in my logs:

Aug 18 07:53:53 marvin postfix/smtpd[23938]: warning: SASL authentication
  failure: cannot connect to Courier authdaemond: No such file or directory
Aug 18 07:53:53 marvin postfix/smtpd[23938]: warning: localhost[127.0.0.1]:
  SASL LOGIN authentication failed: generic failure

After consulting with Google for at least an hour i finally hit upon a page that helped me.

I found the answer in the following howto Postfix w/o Maildrop. The authdaemon socket setup described on this page was slightly different to the usual suggestions (most help I found online suggested to create a hardlink to the socket file inside the Postfix chroot, this doesn’t work for Ubuntu cause the /var/run is on a different filesystem and a symlink didn’t work for me either). The following commands were taken directly from the howto listed above (all credit should go to the writer of that page).

/etc/init.d/courier-authdaemon stop
rm -rf /var/run/courier/authdaemon/
 /var/spool/postfix/var/run/courier/authdaemon/
mkdir -p /var/spool/postfix/var/run/courier/authdaemon/
ln -s /var/spool/postfix/var/run/courier/authdaemon/
 /var/run/courier/authdaemon
/etc/init.d/courier-authdaemon start
postfix reload

The end result is that Postfix is not reading/accessing a symlinked version of the authdaemon socket and everything works as expected.

You will need your linode account number to use the graphs, you can find this number by looking at the graphs provided to you in your Linode account management interface. The title of the graph will be “http://www.linode.com/ – yourhostname (linode12345) – CPU – …….”. You will need the text from inside the ( ) for your username ie linode12345.

To install just copy the linode.pl file into the scripts directory of your Cacti installation. This script was taken from http://www.linode.com/forums/viewtopic.php?t=776 and modified for use with cacti.

Then just import the cacti host template file into Cacti. You should then be able to add the graphs to your relevant host.

Requirements:

• Cacti 0.8.7d
• Perl XML::LibXML
• Perl LWP::UserAgent
• Perl IO::File

Download

• Monitoring Linode Stats with Cacti

Ultimately if you are looking at having your own update repository then you will already have your own installation repository (if you don’t you can always go through the steps in Part 1 of my Autoyast how-to).

First of all create a directory inside your local repository called “updates” and then create a sub-directory for each of the versions you need (for me I only have opensuse11.1).

For you to update your individual servers after creating the repo you will need to have the “yast2-online-update-frontend” package installed.

Use the following command to initially create and update your repo.

rsync -a -v -v --delete-after --progress --stats
mirror.internode.on.net::opensuse/update/11.1/
/var/lib/autoinstall/repository/updates/opensuse11.1-current/

This command will copy about 8gb’s of data, but as it is using rsync you can cancel it and restart it as often as you need. Be sure to select a mirror near you from the following list OpenSUSE Mirror List.

Next log on to one of your clients and perform the following:

1. Go into YaST

2. Software -> Software Management

3. Click Add

4. Select HTTP

5. Enter the relevant details

6. You should end with with a screen somewhat like the following

7. Goto Software -> Online Update

8. Select the required updates

And that’s it.

On the cards were CentOS, Ubuntu LTS, and OpenSUSE. Our main requirements were the ability to access upgrades and the use of an auto build system for rapid deployment. CentOS was ruled out early in the piece (unfortunately I have to say only for petty reasons, none of which are worth listing here). So the competition was between Ubuntu and OpenSUSE, now my personal opinion here would be for Ubuntu to be the successor but unfortunately it had a few fatal flaws.

The main positive for Ubuntu was the 5 year life cycle for Ubuntu LTS releases, 8.04 was the last so that gave me 4 years of supported updates. Also the Debian pedegree tied with the ideology of ease of usability. The only real subtractor was the Ubuntu auto build facilities. They just plain failed to function, I followed the directions laid out somewhere in the Ubuntu help pages but the installation of the required components failed. It was just plain clunky and immature, hopefully in time it will get there.

OpenSUSE had the smallest amount of change required for implementation. Almost all of the existing system documentation was still relevant between SLES and OpenSUSE including all custom scripts and applications just slid into place and worked out of the box. Autoyast was also a big massive tick in its favour, we had already been using the autoinstall system for our SLES setup. The only downside to choosing OpenSUSE was the short life cycle. Each OpenSUSE release will only support updates for two years at which point the move to the latest version will be required (either that or go through the process of selecting a new distribution again).

A couple of application compatibility issues surrounding going through two major application changes, those being PHP 5.1 to 5.2 and PostgreSQL from 8.1 to 8.3 but all issues were relatively simple to fix as our applications are developed in house. In summary, the migration has been a complete success and has been relatively painless, there were no real gotchas.

I have read a lot of blog posts and articles today and will list the ones I found most helpful below.

There is a lot of information out there that is now mostly out of date. PostgreSQL 8.3 changed the game when it came  performance tuning and a lot of the hard and fast rules that had been true for many years are now no longer true. So my one piece of advise when it comes to looking at the information out there, make sure it is relevant to the version of Postgres you are using.

The other thing to keep in mind are there are certain performance parameters that can improve performance, yet reduce reliability or time to restore objectives for your database and also the inverse of this decreased performance and increased reliability. You need to decide what your individual requirements are, it is mostly a counter balance between Performance and Reliability (reliability being anything from producing instability in the disk system by changing caching methods, through to the amount of data lost at power failure, or if using WAL log archiving how long between log writes and how often to write checkpoints). As always when changing any value that is suggested do your own research. For me I looked at the official Postgres documentation as well as just googling the setting name (ie shared_buffers) and reading information from the mail list archives and other tutorials.

Resources I used:

• http://wiki.postgresql.org/wiki/Tuning_Your_PostgreSQL_Server
• http://blog.gtuhl.com/2009/03/08/postgresql-setup-basics/
• http://blog.gtuhl.com/2009/05/13/dell-md1120-perc6e-performance/
• http://www.postgresql.org/docs/8.3/static/

In this three part series we will explore creating a repository for use with multiple systems, creating and customizing the AutoInstall file, and finally running through an install procedure.

Part 3 – Putting it all together and installing a server

In this part we will look at running through a machine install making use of our newly created repository and AutoInstall file.

Check out Part 1 – Creating a repository for use with multiple systems
Part 2 – Creating and Customizing the AutoInstall file

What you will need:

• OpenSUSE Installation Media
• Server/Desktop/VM that has access to DHCP and the HTTP server holding the Repository

OpenSUSE Installation Media

You will need to boot the new machine with some form of installation media. This can be either the full installation DVD or my preference is the small network install CD (I prefer this as not all my servers have DVD drives, plus it is an 88mb ISO to upload to my VM Servers rather then a 5gb ISO). To get a copy of the Network Install disk goto http://software.opensuse.org/ and select Network for the installation medium.

Installation

For the screenshots in this Demo I am using a VM hosted on VMWare ESXi 4.0.

The machine should boot into the following screen.

To start with press F4 to bring up the source menu.

Select HTTP. You will then need to enter the details of your repository server (remember the directory in this context is the directory relative to the webroot)

The next part you need to fill in some extra details for the boot options.

autoyast=http://192.168.0.65/autoyast/autoinstall/demo.xml

This will be the AutoInstall file you have created.

Next select Installation and press enter. For the rest just sit back and wait.

If you didn’t select an automated installation you will be stopped before the following screen presented with all the information regarding the installation (change what you need to then hit Install).

This is after the first reboot, when presented with the install screen again just select “Boot from Hard Disk” or let it time out. You could also remove the CD/DVD just before the machine reboots.

And that is it. The machine install is complete. That concludes the series on AutoYast/AutoInstall. Sometime in the next few weeks I will put together a short howto on creating your own local Update repository.

In this three part series we will explore creating a repository for use with multiple systems, creating and customizing the AutoInstall file, and finally running through an install procedure.

Part 2 – Creating and Customizing the AutoInstall file

In this part we will look at creating an AutoInstall file for use with AutoYast.

Check out Part 1 – Creating a repository for use with multiple systems

For this part you will need a machine with OpenSUSE installed, the latest version would be best (although I think it should work with older versions, however I haven’t tested this theory).

Required Packages:

• autoyast2
• autoyast2-installation

In the next section we look at many different options. There are lots of things to explore, but I will leave this up to you. We will go through the basics:

• Disk Partitioning
• Network Setup
• Package Selection
• Users and Groups
• System Services (Runlevels)
• Basic Post Install Scripting

Firstly access YaST from either a X Windows session or from the Command Line (The screen shots are all from the command line version but the same rules should apply).

Goto the “Miscellanous – Autoinstallation” screen

Disk Partitioning

In this section I found it best to use “Auto” for the device. This way the same file works between my test server in VM, through to my HP Servers with RAID configurations.

For the most part you will just need to enter the details you want/need. But there are two things that needs to be pointed out with regard to partition sizes. What you can do is build a partition with size Auto (this is best used on your swap partition) and also with size Max which will fill the rest of the drive’s free space. So the layout I commonly use is:

• /boot with 128mb – EXT2
• SWAP with Auto
• / with 15GB – EXT3
• /var or /home with Max, EXT3 (Depending on the role of the server, typically /home for development/webservers and /var for database/mail servers)

Unfortunately, if you need to use software RAID Autoyast/Autoinstall will not work for you.

Network Setup

In the Network Settings screen click on Add.

Leave the defaults on the Hardware Dialog Screen (unless you want to change from Eth0 to Eth1,2,3….N, change the number in the Configuration Name Section).

Fill in the Network Card Setup screen with the options you require (ie, DHCP, Static Address, Additional Interface aliases).

Now back in the Network Settings screen go to Hostname/DNS and fill in the appropriate details there (Please note, this is one of the only things that needs to be changed for each server build, or alternatively you could build your servers with a generic name and change them after build).

Next fill in the required details in Routing and your done.

Package Selection

This section is relatively easy, all you need to do is fill in the details as you would for any installation. (The catch is you will first of all need to use the filter to display “Patterns” then at a bare minimum select “Base System”).

Users and Groups

Go to “Security and Users – User and Group Management” and at a bare minimum edit the Root user and set a password (if you don’t do this then you will not be able to login to the machine after it has been built).

At this point you can make any customisations you require for Users and Groups, as a best practice measure you should also create a non-root user and set a password for that account too.

System Services

You will also need to change the default system runlevel if you want a console only server, to do this goto “System – System Services (Runlevel)” then click edit.

Set the default runlevel to “3: Full multiuser with network” and then click ok.

Custom Scripts

You can use AutoYast to install custom files. This allows you to copy down configuration files to the installed machine, inject ssh keys whatever your imagination lets you do.

The other part is Post Install scripts which run during installation, this is used to run custom installers (ie wget a custom module and compile, or rsync webfiles, etc).

Finishing Touches

The final thing to do is decide if you want the autobuild script to require confirmation, by default the installer will show you the proposed setup before installing requiring you to actually click the install button, if this is how you want it then by all means have a look at the other configuration options available to you before we head into the machine installation section in the next part of the series.

If you do want everything to be automatic then goto the “System – General Options” section then edit.

Just click next on the Mouse Configuration (unless you want to change something). Then set “Confirm installation?” to off. Click Next then Finished (F10 in CLI mode).

You can download my sample AutoInstall file – Demo SUSE AutoInstall XML File

The password set on the root account in the XML file is “crackme”.

And your done. Check back for the next part when we look at actually doing a machine install.

Part 3 – Putting it all together and installing a server