Download it from here.

Unpack it then run

$ sudo apxs -ci -I /usr/include/libxml2 mod_xml2enc.c
$ sudo apxs -ci -I /usr/include/libxml2 -I . mod_proxy_html.c

Then update you httpd.conf to include


LoadFile /usr/lib/libxml2.dylib
LoadModule proxy_html_module libexec/apache2/mod_proxy_html.so
LoadModule xml2enc_module libexec/apache2/mod_xml2enc.so


Also take a look at the proxy-httpd.conf file that ships with mod_proxy_html for additional config you may want.

Read on for links to some interesting articles and my conclusions.

First a quick word about terminology. Dual Parity raid is often implemented as RAID 6, this involves having 2 disks for parity rather than the single disk in RAID 5. I’m going to look at articles regarding two vendor implementations of dual parity raid, these differ from traditional RAID 6 in a number of ways. They are NetApp’s RAID-DP and Sun’s raidz2.

I came across this article from one of Sun’s engineers some time ago, and the project I’m working on caused me to review it again. The article compares two MTTDL models, one ‘traditional’ mode, which takes Mean Time Before Failure (MTBF) and Mean Time To Repair (MTTR) into account, and a second more interesting one that takes into account the unrecoverable error rate (UER).

It is the UER that I find particularly interesting, this is the rate at which a read for a bit will result in an unrecoverable error. In a single parity system such as RAID 1+0 or RAID5 you need your parity data to be 100% readable in the event of a disk failure,otherwise your going to be looking at data loss, which will inevitably be in the worst location.

Disk sizes continue their march towards ever larger capacities (a possible move to smaller, faster SSD drives excepted). However the UER has remained fairly flat. The linked Sun blog above has this to say

Typically, UER will be 1 per 10^14 bits read for consumer class drives and 1 per 10^15 for enterprise class drives. This can be alarming, because you could also say that consumer class drives should see 1 UER per 12.5 TBytes of data read. Today, 500-750 GByte drives are readily available and 1 TByte drives are announced. Most people will be unhappy if they get an unrecoverable read error once every dozen or so times they read the whole disk.

This NetApp white paper about their implementation of RAID-DP has this to say:

As disk drives have gotten larger, their reliability has not improved, and, more importantly, the bit error likelihood per drive has increased proportionally with the larger media.

As we push drives sizes higher and higher we increase the likelihood of finding an unreadable bit and have to consider the real possibility that we cannot recover a volume that was protected with single parity.

So what can we do about it. A three-way mirror is an option, but try explaining the cost of that to your customer. A more practical solution is to consider double parity RAID. With this configuration you can lose an entire drive, and if you hit an unreadable bit you still have one more location to retrieve that data from before it is lost.

“But what about write performance” I hear you cry! In a traditional RAID5/6 implementation if you make writes that are not across a full stripe then there performance can tank considerably, increasingly so as you add more spindles. The reason for this is that for a partial stripe write, the section of the stripe that is not being written to must first be read from disk in order for the parity to be calculated. Then the new data and parity can be written out to disk.

ZFS’s raidz2 has a solution for this and that is that all writes are made as full stripe writes. This is achieved by dynamically altering the width of the stripe to match the write being made. This also neatly works around the RAID5/6 write hole. Have a read of this document if you’d like more info about this.

NetApp’s RAID-DP is actually an extension of RAID4 rather than RAID5. This means that there a two dedicated columns for parity, rather than a distributed layout. The aforementioned white paper has more detail as to the specific layout used. NetApp also uses a file system called WAFL, I’ve been unable to easily find any documentation around this and partial stripe write performance, please comment if you know of any.

The whole issue with partial stripe write is obviated somewhat in an environment where you are supporting a single application, such as Oracle, where the block size is known and configurable. In this instance the Oracle block size can be configured in partnership with the storage to eliminate (or greatly reduce) the need for partial stripe writes.

Read performance from a raidz2 device is not as great as you might imagine, typically it will be limited to the performance of a single disk, for reasons that are explained in this article.

I expect that the read performance of RAID-DP matches that of a traditional RAID4/5 system much more closely, i.e. that reads can be striped across all the spindles. However I don’t have any stats to back this up.

I would like to have the opportunity to benchmark the systems against each other with a realistic workload but at the moment I don’t have that option. For me the increased data reliability of a dual parity system, when faced with an increasing UER per disk, is enough to covert me, even if it means a loss of some performance. Providing the performance the array can deliver is adequate then I see dual parity RAID as worthy of significant consideration.

• xVM VirtualBox
• xVM Server
• Linux Branded Zone

This post takes a look at the interesting world of the Linux branded zone. I’ve posted about VirtualBox before and I hope to take a look at xVM Server (Xen) in a future post. Read on for my first steps with Linux branded zones.

I’ve used Solaris Express (SXCE) Build 106 here as it delivers this fix which is pretty much required for running a 2.6 kernel in a branded zone. The OpenSolairs pkg repository does not yet appear to be updated to Build 106 at the moment.

Linux branded zones are perhaps the screwball option for running Linux binaries on Solaris. Rather than provide a fully self contained Linux environment it provides a Linux compatible zone in this case based on CentOS 5.2 i386. Currently the only install method for a Linux 2.6 based distribution is to install the OS somewhere else then create a tarball of the image and use that as the source media. Installing direct from the distribution iso files is not yet supported.

You must disable SELinux in the image you are going to create. You can either do this during the install or select ‘Administration’ / ‘Security Level and Firewall’ and disable SELinux.

So to start we need a running 32-bit CentOS 5.2 image, it’s trivial to install one in VirtualBox. Then create an tarball of the running image like so:

tar -jcf /net/192.168.0.10/iso/centos5.2.i386.tar.bz --exclude dev --exclude proc --exclude sys --exclude boot --exclude media --exclude net *

To begin configuring the zone you need to download a Linux 2.6 zone template file, simply do this

cd /etc/zones
wget http://www.opensolaris.org/os/community/brandz/files/SUNWlx26.xml

Then create the zone, the difference between this process and a normal Solaris zone is that the Linux 2.6 template must be specified

bash-3.2# zonecfg -z centos52
centos52: No such zone configured
Use 'create' to begin configuring a new zone.
zonecfg:centos52> create -t SUNWlx26
zonecfg:centos52> set zonepath=/vm/zones/centos52
zonecfg:centos52> add net
zonecfg:centos52:net> set address=192.168.0.50
zonecfg:centos52:net> set physical=e1000g0
zonecfg:centos52:net> end
zonecfg:centos52> commit
zonecfg:centos52> verify
zonecfg:centos52> exit

The install the archive into the zone

bash-3.2# zoneadm -z centos52 install -d /net/192.168.0.10/iso/centos/centos5.2.i386.tar
A ZFS file system has been created for this zone.
Installing zone 'centos52' at root directory '/vm/zones/centos52'
from archive '/net/192.168.0.10/mnt/usbdisk1/iso/centos/centos5.2.i386.tar'

This process may take several minutes.

Setting up the initial lx brand environment.
System configuration modifications complete.
Setting up the initial lx brand environment.
System configuration modifications complete.

Installation of zone 'centos52' completed successfully.

Details saved to log file:
    "/vm/zones/centos52/root/var/log/centos52.install.1615.log"

Check that it is installed

bash-3.2# zoneadm list -cv
  ID NAME             STATUS     PATH                           BRAND    IP
   0 global           running    /                              native   shared
   - centos52         installed  /vm/zones/centos52             lx       shared

The boot it up

bash-3.2# zoneadm -z centos52 boot
zone 'centos52': WARNING: e1000g0:1: no matching subnet found in netmasks(4) for 192.168.0.50; using default of 255.255.255.0.
bash-3.2# zlogin -C centos52
[Connected to zone 'centos52' console]
Entering non-interactive startup
--snip start up messages---
CentOS release 5.2 (Final)
Kernel 2.6.18 on an i686

centos52 login: root
Password:
Last login: Mon Jan 26 22:56:35 on :0
[root@centos52 ~]# uname -a
Linux centos52 2.6.18 BrandZ fake linux i686 i686 i386 GNU/Linux

Then processes running in the zone

[root@centos52 ~]# ps -ef
UID        PID  PPID  C STIME TTY          TIME CMD
xfs       3120  2688  0 Jan27 ?        00:00:00 xfs -droppriv -daemon
root      3228     1  0 Jan27 ?        00:00:00 /bin/login -- root
root         1     0  0 Jan27 ?        00:00:00 /sbin/init
root      3141  2688  0 Jan27 ?        00:00:00 /usr/sbin/atd
root      2891  2688  0 Jan27 ?        00:00:00 klogd -x
root      3229  3228  0 Jan27 ?        00:00:00 -bash
root      2688  2688  0 Jan27 ?        00:00:00 [zsched]
root      3253  3229  0 Jan27 ?        00:00:00 ps -ef
root      3131  2688  0 Jan27 ?        00:00:00 anacron -s
root      3052  2688  0 Jan27 ?        00:00:00 /usr/sbin/sshd
root      2888  2688  0 Jan27 ?        00:00:00 syslogd -m 0 -p /var/run/syslog
root      3063  2688  0 Jan27 ?        00:00:00 cupsd
root      3003  2688  0 Jan27 ?        00:00:00 pcscd
root      3089  2688  0 Jan27 ?        00:00:00 crond
root      3173  2688  0 Jan27 ?        00:00:00 /usr/bin/python -tt /usr/sbin/yu
root      3175  2688  0 Jan27 ?        00:00:00 /usr/libexec/gam_server
[root@centos52 ~]#

And the related processes in the global zone

# ps -efZ | grep centos52
centos52 0000043  3120     1   0 23:30:05 ?           0:00 xfs -droppriv -daemon
  global     root  2687     1   0 23:29:19 ?           0:00 zoneadmd -z centos52
centos52     root  3228  2702   0 23:31:06 ?           0:00 /bin/login -- root
centos52     root  2702  2688   0 23:29:19 ?           0:00 /sbin/init
centos52     root  3141     1   0 23:30:05 ?           0:00 /usr/sbin/atd
  global     root  2822  1551   0 23:29:21 pts/3       0:00 zlogin -C centos52
centos52     root  2891     1   0 23:29:32 ?           0:00 klogd -x
centos52     root  3229  3228   0 23:31:37 zoneconsole    0:00 -bash
centos52     root  2688     1   0 23:29:19 ?           0:00 zsched
centos52     root  3131     1   0 23:30:05 ?           0:00 anacron -s
centos52     root  3052     1   0 23:29:34 ?           0:00 /usr/sbin/sshd
centos52     root  2888     1   0 23:29:22 ?           0:00 syslogd -m 0 -p /var/run/syslog
centos52     root  3063     1   0 23:29:34 ?           0:00 cupsd
centos52     root  3003     1   0 23:29:33 ?           0:00 pcscd
centos52     root  3089     1   0 23:30:04 ?           0:00 crond
centos52     root  3173     1   0 23:30:05 ?           0:00 /usr/bin/python -tt /usr/sbin/yum-updatesd
centos52     root  3175     1   0 23:30:05 ?           0:00 /usr/libexec/gam_server

Notably you can see that ‘init’ in process ‘1′ in the zone but process ‘2702′ in the global zone, presumably to maintain compatibility.

Due to this being a zone rather than a full on emulation you still only have one running kernel, the Solaris one, so you can presumably manage resources for this processes like you can for any normal Solaris zone, notably the memory space is shared as are the file systems, this has the potential to provide better performance than adding a full virtualisation layer and running a Linux kernel. Hopefully I’ll have a chance to do some rudimentary tests in the next week or so.

So instead I decided to write my own exporting tool, you can see the end result at http://photos.tyr.org.uk the default view is a match of the project/folder/album structure within my Aperture library and the ‘quilt’ view (see top right) is just for fun.  To export photos to the website I just tag them with the ‘Web’ keyword then run the export/sync script, it couldn’t be quicker and certainly much less hastle than maintaining two libraries.

Read on if you want to know more about how it was put together

The project consists of two parts. Firstly a method to get information out of Aperture and secondly some method of displaying the exported photos to the end user. I’ll cover the process I used to extract the metadata and images from Aperture in some detail in the hope it is useful to others wanting to achieve the same thing, the end user web front end is more specific to my requirements so I won’t go into it in so much depth but I’ll give some pointers to some tools used.

The functionaility I wanted to achieve with the export was to export all images tagged with a specific keyword and create metadata describing the folder/album structure and the image metadata. After much playing about with the inbuilt Aperture export tools I couldn’t find anything that would do this for me.

However all is not lost as Aperture uses a SQLite database to store it’s catalogue. Whilst it is not inteded to be accessed directly it is possible to open it up and take a look inside and figure out what is going on. However the layout of the database is not published publically not is it likelly to be stable between Aperture releases, in fact much of this code was originally written for Aperture 1.5 and had to be rewritten when 2.0 was released.

The first step to using the data in the database was to figure out what it all means. I used SQLite Database Browser to interrogate the database. The database itself is called ‘Library.apdb’ and can be found within your Aperture library, to browse into the library option-click on it and choose ‘Show Package Contents’. You can then open up the Library.apdb in the database browser.

I wrote the data extraction script in Perl, handily OS X ships with the appropriate SQLite libraries for Perl DBI so opening up the database is easy.

my $dbh = DBI->connect ( "dbi:SQLite:Library.apdb") || die "Cannot connect: $DBI::errstr";

All keywords in Aperture are identified by a UUID called a ZUUID, so the first step is to extract the ZUUID associated with the Keyword, eg

select ZUUID from ZRKKEYWORD where ZNAME = 'Web'

Photos in Aperture consist of a ‘Master’ image, each of which can have multiple ‘Versions’. We are interesting in extracting the versions that have the assocated keyword, we can find the UUID’s for these versions like this

select ZVERSIONUUID from ZRKXKEYWORDVERSION where ZKEYWORDUUID = '$keyZUUID'

Each of these versions will be extracted and published on the web site. We need to find out two pieces of information for each image, its location in the library and its individual metadata.

As part of this design we assume that all tagged images are part of an album, we’re not allowing for images that exist only in the top level of a project. We can find the albums a version is a member of like so

select ZALBUMUUID from ZRKWALBUMVERSION where ZVERSIONUUID = '$versionUUID'

Note that a version can belong in multiple albums. At this point we just build a list of albums and the ZVERSIONUUID’s of the versions each album.

The metadata for each version is stored in two places. The most significant data is stored in ZRKVERSION which contains one row per version, the less significant information such as EXIF data is stored in a huge table called  ZRKSEARCHABLEPROPERTY, this contains one row per property, as such it can contain 30 rows or so per version.

Firstly the information from ZRKVERSION is extracted

select ZVERSIONID, ZFILENAME, ZPROCESSEDWIDTH, ZPROCESSEDHEIGHT
from ZRKVERSION
where ZUUID = '$ZVERSIONUUID'

With this query we get the filename of the image and its processed width and height (so we can determine orientation) and als the ZVERSIONID, this is a different ID to the ZVERSIONUUID and we need the ZVERSIONID to extract the metadata for the image.

Now we can extract the bulk of the metadata for the image. This is held in ZRKSEARCHABLEPROPERTY and can be extracted like this

select ZPROPERTYIDENTIFIERUUID, ZVALUETYPE, ZPROPERTYSPECIFICNUMBER, ZPROPERTYSPECIFICSTRING
from ZRKSEARCHABLEPROPERTY
where ZVERSIONID = $ZVERSIONID"

Each property is either a string or a number, only one of ZPROPERTYSPECIFICSTRING and ZPROPERTYSPECIFICNUMBER contains a value, if ZVALUETYPE is ‘3′ then it’s a string otherwise it can be assume to be a number of some sort, floating point or integer.

The type of property is identified by it’s ZPROPERTYIDENTIFIERUUID. The data to convert this into a human redable property description is found in ZRKPROPERTYIDENTIFIER. To enable this to work efficiently I extracted all the data from this table into a lookup hash. eg

select ZPROPERTYKEY, ZUUID from ZRKPROPERTYIDENTIFIER

Where the ZUUID in ZRKPROPERTYIDENTIFIER matches the ZPROPERTYIDENTIFIERUUID in ZRKSEARCHABLEPROPERTY.

That completes the data extraction for the individual photos. The last thing to do is to extract the folder/album structure. For each album its name and its containting folder can be found like this

select ZFOLDERUUID, ZNAME from ZRKPERSISTENTALBUM where ZUUID = '$ZALBUMUUID'

Then for each folder its name and containing folder can be found from ZRKFOLDER like this

select Z_PK, ZNAME, ZPARENTFOLDERUUID from ZRKFOLDER where ZUUID = '$ZFOLDERUUID'

This data can be merged together to give you the full tree structure.

Now all the data had been gathered it’s just a matter of spitting out some XML. One file called library.xml contains the library structure, i.e. references  to all the folders/albums and photos. Then for each photos a ’sidecar’ xml file is created, this contains the exif and other metadata for the photo [example]. This ’sidecar’ is then loaded when the user displays the photo, this increases performance and reduces the amount of data required for an initial load of the library to just a skeleton.

Now we can extract the photo itself from Aperture. Since Aperture can resize images on export we’ll use that functionality to create an image that’s suitable for display to the end user. Aperture can be scripted using AppleScript so we take advantage of that functionality, this can be accessed from a shell call via ‘osascript’ for example

/usr/bin/osascript -e '
   tell application "Aperture"
       set exportFolder to "MacDrive:Users:username:Documents:WebExport"
       export {image version id "$imageUUID"} using "JPEG - Web - Display" to alias exportFolder
   end tell
'

It should be noted in the above that the value set for ‘exportFolder’ is the old Classic style of directory naming with colons rather than slashes.
You must create an export profile within Aperture with the name you use here, in this case ‘JPEG – Web – Display’.

Now we can rsync the images and the XML up to the server leaving just one final step and that is to create some thumbnails.

I wanted nice square thumbnails that didn’t change the aspect ratio of the image. Imagine the largest square you could create entirely within a full size photo then shrink that to a 50×50 thumbnail. This ImageMagick magic sorted it out. I don’t have ImageMagick on my Mac so this was run on the server that serves the images.

convert $sourceFile -thumbnail x200 -resize '200x< ' -resize 25% -gravity center -crop 50x50+0+0 +repage -quality 90 $destFile

And that completes the extraction and upload process. Finally we’ll take a quick look at the actual display side of things.

The idea behind the web design was to create something nice and simple that presented the photos in a non-fussy but easy to navigate way. I used the Yahoo User Interface Library (YUI) to build the interface. The main advantage in using this library is that it hides most of the browser incompatibilities out there, for example YAHOO.util.Connect.asyncRequest provides an AJAX function.

Particularly usefully you can assign a single ‘onclick’ hander to the folder browser div, then using  YAHOO.util.Event.getTarget you can find the object that was actually clicked. This makes it much more efficient than having onclick handlers all over the place.

The ‘quilt’ view does some simple fadein/fadeout animation. This was achieved using the YUI: Effects Widgets from davglass.com. This creates a nice affect as the imgaes are swapped in the quilt.

If you’re interested you can browse the javascript on the site; [main page], or [quilt].

I hope the above info was interesting or useful, post a comment if you’ve got any queries.

Please feedback with any bug reports or feature requests.

Read on for some of the behind the scenes info.

There are two options facing a prospective iPhone application developer, to develop a web app or a full blown iPhone application. Prior to the release of the iPhone 2.0 OS web apps were the only option open to developers wanting to target users without jailbroken phones.

Writing a full iPhone application requires Cocoa and Objective-C fu, which whilst I’m learning I’m nowhere near the level required this task. Conversely web apps are much simpler and are just served as standard web pages to the local iPhone Safari browser. All the client side coding can be created in JavaScript.

Apple’s provided development environment for web apps is called Dashcode. I’d used it previously as a beta on Tiger to create the Dashboard widget and found it to be fairly buggy. This time however I’m pleased to say it performed admirably. To aid development a selection of framework apps are provided, in this instance the ‘browser’ framework was picked. This provides you with a simple interface that lets the user browse through multiple view in a tree structure. The only Javascript you need to provide is a controller for the view, a sufficient outline is given to get you going without too much work.

I don’t actually own an iPhone but an iPhone simulator is provided with the development tools. As soon as you click ‘Run’ in Dashcode the simulator loads up and launches your web app. You can simulate pretty much all the normal iPhone functionality including rotation and multi touch. It really helps make development a breeze.

iPhone OS 2 ships with a version of Safari 3.1.1, which supports client side SQL in JavaScript. Using this to store user settings is a breath of fresh air compared to creating bespoke functions to (un)pack data into cookies. Here’s the SQL used to create a database

 try {
     var shortName = 'yournextbus';
     var version = '1.0';
     var displayName = 'Your Next Bus Database';
     var maxSize = 2048;
     var db = openDatabase (shortName, version, displayName, maxSize);
     return db;
 } catch (e) {
     if ( e == INVALID_STATE_ERR ) {
         // Version number mismatch
         alert ("Invalid database version");
     } else {
         alert ("An error occured creating a database "+e+".");
     }
 }

And here’s the code to create a table and add a row:

 this._db.transaction (
    function (transaction) {
        transaction.executeSql('CREATE TABLE stops(id INTEGER NOT NULL PRIMARY KEY AUTOINCREMENT, name TEXT NOT NULL DEFAULT "A Bus Stop", number INTEGER NOT NULL)', [], nullDataHandler, killTransactionSilently);
        transaction.executeSql('INSERT INTO stops ( name , number ) VALUES (?,?) ', [ 'Demo Stop', 45013610 ], nullDataHandler, errorHandler );

     }
 )

Whilst the data being stored for this app is fairly trivial it’s simple to see how this could become very powerful.

A couple of pics of the app in action:

You can download it here (needs a Symantec login) and there’s a Flash video explaining how to import an existing cluster here.

First an overview of what I set out to achieve.

• Clustered MySQL server in a zone
• Clustered Apache zone providing phpmyadmin front end for MySQL
• On a two node Solaris Express cluster
• With nodes running in VirtualBox virtual machines on a single physical machine
• Shared storage provided over iSCSI from the host machine

VirtualBox does not support sharing the same (VDI) disk image between multiple hosts, unless the image is read-only. As such VirtualBox cannot natively provide storage appropriate for a clustered environment, so we’re going to prevent storage over iSCSI to the virtual nodes.

At the time of writing Solaris Cluster Express 6/08 has just been released, this has been updated to run on Solaris Express Community Edition (SXCE) Build 86, so we’ll be using that for our guest OS in the virtual machines.

Initially I was hoping to use OS X as the host platform for VirtualBox however the networking support is far from complete in the OS X release. Specifically is does not support ‘internal networking’ and that’s needed for the cluster interconnects. So instead I chose OpenSolaris 2008.05 for the host, this has the advantage that the host can be used to provide the iSCSI services along with a Quorum Server and has the required networking flexibility.

If you don’t have an OpenSolaris host, then any recent Solaris 10 or Solaris Express server on your network should work fine. Ensure that ZFS supports the ’shareiscsi’ options and that you can share 2 8GB volumes from it. Also you will need to install a Quorum Server to it. If you don’t have a suitable host on your network you could create a 3rd VirtualBox machine to provide the required services.

To build this environment the following downloads were used

• Solaris Express community edition, build 86, from here
• VirtualBox 1.6.2 from here
• Solaris Cluster Express 6/08 from here
• OpenSolaris 2008.05 from here
• phpMyAdmin 2.11.6 from here

You’ll need a machine with at least 2 GB of RAM, and around 50GB of hard disk space to follow this guide. If you have 2GB of RAM you’ll spend a fair amount of time swapping and may experience occasional cluster node panic, so more is recommended if you can get it.

Throughout this guide the prompt for the commands indicated the server it should be typed on.

• opensolhost# – The OpenSolaris host
• cxnode1# – The first Cluster Express node
• cxnode2# – The second Cluster Express node
• both-nodes# – Repeat on both the first and second cluster nodes
• apache-zone# – Within the Apache zone
• mysql-zone# – Within the MySQL zone

The following IP addresses were used, modify these to match your local network environment

• 192.168.0.111 – The IPMP failover IP for cxnode1
• 192.168.0.112 – The IPMP failover IP for cxnode2
• 192.168.0.118 – The IP for the Apache zone
• 192.168.0.119 – The IP for the MySQL zone
• 192.168.0.121 – The first IPMP test address on cxnode1
• 192.168.0.122 – The first IPMP test address on cxnode2
• 192.168.0.131 – The second IPMP test address on cxnode1
• 192.168.0.132 – The second IPMP test address on cxnode2

Sun provide extensive documentation for Sun Cluster 3.2, which for these purposes should match Cluster Express closely enough. You can find a good starting point at this documentation centre page.

Installing and configuring VirtualBox

To begin download VirtualBox from the links above, the steps below reference the 64 bit version but they should be the same for 32 bit users. Download the package and unzip/tar it, inside you’ll find two packages, they both need installing, ie:

opensol-host# pkgadd -d VirtualBoxKern-1.6.2-SunOS-r31466.pkg
opensol-host# pkgadd -d VirtualBox-1.6.2-SunOS-amd64-r31466.pkg

Before the virtual machines can be created some network configuration on the host is required. To enable the front interfaces of the cluster nodes to connect to the local network some virtual interfaces on the host are required. These will have the desired affect of bridging the local VLAN with the virtual machines.

The instructions in the VirtualBox documentation say to use /opt/VirtualBox/setup_vnic.sh to create these interfaces, however I had problems getting that to work. Not only is does it need some tweaks to get it to work with OpenSolaris 2008.05, I found I couldn’t get the interfaces going even after they were created. Fortunately I came across this blog post, which pointed me in the direction of the steps below.

2 public interfaces are required for each cluster node, these can then be configured with IPMP, just as you would do in a real cluster, whilst this doesn’t really provide more resiliency it’s a worthwhile exercise. To create these public interfaces a total of four virtual interfaces are required on the host. Each one needs a defined MAC address that you can choose yourself, or you can use the suggested ones here. To create the interfaces do this, replacing e1000g0 with the address of your physical interface on your host.

opensol-host# /usr/lib/vna e1000g0 c0:ff:ee:0:1:0
opensol-host# /usr/lib/vna e1000g0 c0:ff:ee:0:1:1
opensol-host# /usr/lib/vna e1000g0 c0:ff:ee:0:2:0
opensol-host# /usr/lib/vna e1000g0 c0:ff:ee:0:2:1

I’ve chosen these MAC addresses because c0:ff:ee is easy to remember, and the last two octets represent the cluster node number and then the interface number in that node.

Now plumb these interfaces, but don’t give them IP addresses.

opensol-host# ifconfig vnic0 plumb
opensol-host# ifconfig vnic1 plumb
opensol-host# ifconfig vnic2 plumb
opensol-host# ifconfig vnic3 plumb

These four ‘vnic’ interfaces will form the public interfaces of our cluster nodes. However these will not persist over a reboot so you may want to create a start up script to create these. There’s a sample script in the linked blog post above.

It’s now time to create the virtual machines, as your normal user run /opt/VirtualBox/VirtualBox and a window should pop up like this

To start creating the virtual machine that will serve as the first node of the cluster, click ‘New’ to fire up the new server wizard. Enter ‘cxnode1′ for the name of the first node (or choose something more imaginative), pick ‘Solaris’ as the OS. Page through the rest of the wizard choosing 1GB for the memory allocation (this can be reduced later, at least 700MB is recommended) and create the default 16GB disk image.

The network interfaces need to be configured to support the cluster. Click into the ‘Network’ settings area and change the ‘Adapter 0′ type to ‘Intel PRO/1000 MT Desktop’ and change the ‘Attached to’ to ‘Host Interface’. Then enter the first MAC address you configured for the VNICs, if you followed the example above this will be c0ffee000100, then enter an interface name of ‘vnic0′. This Intel PRO/1000 MT Desktop adapter will appear as e1000g0 within the virtual machine. Generally I’ve grown to like these adapters, not least because they use a GLDv3 driver.

Now for ‘Adapter 1′ enable the adapter and change the type as before, and set ‘Attached to’ to ‘Host Interface’, the MAC to c0ffee000101 (or as appropriate) and the interface name to ‘vnic1′. Then enable ‘Adapter 2′ and set ‘Attached to’ to Internal Network and set ‘Network Name’ to ‘Interconnect 1′. Repeat for ‘Adapter 2′ but set the ‘Network Name’ to ‘Interconnect 2′.

Finally point the CD/DVD-ROM to the ISO image you downloaded for Solaris Express Build 86. You need to add the ISO with the ‘Virtual Disk Manager’ to make it available to the machine. You can use the ‘/net’ NFS automounter to point to a NFS share where this image resides if required.

Finally change the boot order, in General / Advanced, so that ‘Hard Disk’ comes before ‘CD/DVD’. This means that it will initially boot the install media, but once installed will boot from the installed drive.

Repeat the above steps to create a second cluster node. Ensure that ‘Adapter 2′ and ‘Adapter 3′ are connected to the same networks as for the first cluster node. Adapters 1 and 2 should be connected to the 3rd and 4th VNICs created previously.

Installing Solairs

Solaris now needs to be installed to both the cluster nodes. Repeat the following steps for each node. To boot a virtual machine click ‘Start’ and the machine should boot and display the Grub menu. DON’T pick the default of ‘Solaris Express Developer Edition’ but rather choose ‘Solaris Express’. If you choose the Developer Edition option you’ll get the SXDE installer which does not offer the flexibility required around partition layout.

Pick one of the ‘Solaris Interactive’ install options as per your personal preference. If you’ve ever installed one of the main line Solaris releases that you’ll be at home here. Suggested settings for system identification phase:

• Networked
• Configure e1000g0 (leave the others for the time being)
  • No DHCP
  • Hostname: cxnode1 (or choose something yourself)
  • IP: 192.168.0.111 (or something else as appropriate)
  • Netmask 255.255.255.0 (or as appropriate)
  • No IPV6
  • Default Route – Specify
  • 192.168.0.1 (the IP of your default router)
  • Don’t enable Kerberos
  • ‘None’ for naming service
  • Default derived domain for NFSv4 domain
• Specify time zone as required
• Pick a root password

Then in the installation phase pick the following options

• Reboot automatically: no
• Eject additional CDs: yes (not that we’ll be using any)
• Media: CD/DVD
• Install Type: Custom
• Add additional locales as required, we’re using C as the default.
• Web Start Scan location: None
• Software group: Entire Group Plus OEM / Default Packages
• Leave the fdisk partition as the default, one single Solaris partition covering the whole disk.
• To support the cluster a specific filesystem layout is required. Click ‘Modify’ then set as this: (If space is needed for a live upgrade in the future then an additional virtual disk can be attached)
  • Slice 0: ‘/’ – 13735 MB
  • Slice 1: ’swap’ – 2048 MB
  • Slice 6 ‘/globaldevices’ – 512 MB
  • Slice 7 ‘(leave blank)’ – 32MB
• Now just confirm and start the install

The installer should now run through in due course, once complete reboot the machine and check it boots up fine for the first time.

By default Solaris will boot up into a GNOME desktop. If you want to disable the graphic login prompt from launching then do ’svcadm disable cde-login’

Before the cluster framework is installed the VirtualBox ‘Guest Additions’ need to be installed, these serve a similar role to VMWare Tools in that they provide better integration with the host environment. Specifically the ‘Time synchronization’ facilities are required to assist with keeping the cluster nodes in sync.

If you still have the SXDE DVD image mounted then ‘eject’ this in the guest and ‘Unmount’ it from the ‘Devices’ menu. Then choose ‘Install Guest Additions…’ from the VirtualBox menu. The Guest Additions iso should mount, then su to root and pkgadd the VBoxSolarisAdditions.pkg from the CD. If you’re running X you should logout and back in to activate the X11 features.

Repeat the above steps for both cluster nodes.

It’s worth considering taking a snapshot at this point so if you run into problems later you can just snap it back to this post install state.

Preparing for cluster install

Once the nodes are installed there are a few steps required to configure them before the cluster framework can be installed. Firstly the public network interfaces on the nodes need to be configured. To do this use the below /etc/hostname files modifying where appropriate to your local network.

• cxnode1:/etc/hostname.e1000g0

  • 192.168.0.121 netmask 255.255.255.0 broadcast 192.168.0.255 deprecated -failover group public up
    addif 192.168.0.111 up

• cxnode1:/etc/hostname.e1000g1

  • 192.168.0.131 netmask 255.255.255.0 broadcast 192.168.0.255 deprecated -failover group public up

• cxnode2:/etc/hostname.e1000g0

  • 192.168.0.122 netmask 255.255.255.0 broadcast 192.168.0.255 deprecated -failover group public up
    addif 192.168.0.112 up

• cxnode2:/etc/hostname.e1000g1

  • 192.168.0.132 netmask 255.255.255.0 broadcast 192.168.0.255 deprecated -failover group public up

Also check that /etc/defaultrouter is correct.

The RPC communication must be activated for the cluster framework to function. To do this do

opensol-host# svccfg
svc:> select network/rpc/bind
svc:/network/rpc/bind> setprop config/local_only=false
svc:/network/rpc/bind> quit
opensol-host# svcadm refresh network/rpc/bind:default
opensol-host# svcprop network/rpc/bind:default | grep local_only

The last command should return ‘false’.

Modify your path to include

/usr/bin
/usr/cluster/bin
/usr/sbin
/usr/ccs/bin

Also check your ‘umask’ is set to 0022 and change it if not.

Finally we need to ensure the cluster nodes exist in /etc/inet/hosts on both hosts. For example

192.168.0.111 cxnode1
192.168.0.112 cxnode2

After making the above changes bounce the nodes to check it all persists across a reboot.

Once the above has been repeated on both cluster nodes it is time to install the cluster framework.

Installing Cluster Express

Download and extract Solaris Cluster Express 6/08, inside the package cd into 'Solaris_x86' then run './installer'. If you are connected over a ssh or on the console then run './installer -nodisplay' instead. The steps listed here are for the GUI installer, but the text one is much the same.

Wait for the GUI to launch, click through and accept the license. Then from the list of available services choose ‘Solaris (TM) Cluster Express 6/08′ and ‘Solaris (TM) Cluster Express Agents 6/08′ (enter ‘4,6′ if you’re in the text installer). Leave the other options disabled, and clear ‘Install multilingual packages’ unless you want them. Click ‘Next’ to start the installer. You’ll be informed that some packages are being upgraded from their existing versions (namely SUNWant, SUNWjaf and SUNWjmail). The cluster will now perform some pre installation checks and they should all pass ‘OK’, then proceed with the install. Choose ‘Configure Later’ when prompted, as that will be done once all the installation steps are finished. Repeat the install process on the second node.

Now the cluster is installed it can be configured and established. This is started by running /usr/cluster/bin/scinstall. I prefer to do this on the second node of the cluster (cxnode2), as the installer configures the partner server first and it will be assigned node id 1, the server running the install will be assigned node id 2. It doesn’t really matter I just prefer it to follow the ordering of the hostnames.

Once the installer is running choose option 1 (Create a new cluster or add a cluster node), then option 1 (Create a new cluster), answer ‘yes’ to continue, choose ‘2′ for custom configuration. Pick a cluster name eg ‘DEV1′. Then when prompted for the other nodes in the cluster enter ‘cxnode2′ then ^D to complete. Confirm the list of nodes is correct, communication with the other node will now be tested and should complete fine. Answer ‘no’ for DES authentication.

Now you’ll be asked about the network configuration for the interconnect. The default network range is 172.16.0.0/255.255.248.0 but you can change this if required. Answer ‘yes’ to use at least two private networks, then ‘yes’ to the question about switches. Although there are no switches to configure we’re considering each private network configured in VirtualBox to be an unmanaged switch. Accept the default names for ’switch1′ and ’switch2′. You’ll now be asked to configure the transport adapters (effectively the network interfaces). Pick ‘1′ (e1000g2) for the first cluster transport adapter, answer ‘yes’ to indicate it is a dedicated cluster transport adapter. Answer ’switch1′ when asked about which switch it is connected to and accept the defaut name. Then pick ‘2′ (e1000g3) for the second adapter and again pick the default options. Answer ‘yes’ for auto discovery.

You’ll now be asked about Quorum configuration, this will be addressed later so choose ‘yes’ to disable automatic selection. The next quetion is about the global devices file system. The default is /globaldevices, accept it as this matches the layout created when initially installing Solaris. Accept this for cxnode2 as well.

You’ll be asked for final confirmation that it is ok to proceed so just answer ‘yes’ when you’re ready. You’ll now be asked if you want the creation to be interrupted if sccheck fails, go for ‘yes’ rather than the default of ‘no’. The cluster establishment will now start. You should see something similar to this when it discovers the cluster transports:

The following connections were discovered:
cxnode2:e1000g2  switch1  cxnode1:e1000g2
cxnode2:e1000g3  switch2  cxnode1:e1000g3

cdnode1 will reboot and establish the cluster with itself as the only node. Don’t worry about any errors about /etc/cluster/ccr/did_instances, the DID database hasn’t been created yet. The installer will then configure cxnode2 and reboot that. When it boots back up it will join the new cluster.

We now have an established cluster! However it’s in ‘installmode’ until a quorum device is configured.

Configuring a Quorum Server

To finish the base cluster configuration a quorum device must be assigned. Initially I was planning to do this by presenting an iSCSI LUN from the OpenSolaris host into the guests, then using that for the quorum device. However I found that it, although it could be added fine, it would show as ‘offline’ and not function properly.

As such a quorum server running on the OpenSolaris host will be used. Full documentation for this can be found here, but this overview should be enough to get you going. Fortunately it can be installed on it’s own, without requiring a full cluster install. To install it make the Cluster Express package available on the host and run the installer again, you’ll need to use the ‘-nodisplay’ option as some packages won’t be available for the graphical installer.

When running the installer choose ‘No’ when asked if you want to install the full set of components. Then choose option 2 for ‘Quorum Server’ and install that. Choose ‘Configure Later’ when asked.

The default install creates a configuration for one quorum server, running on port 9000. You can see the configuration in /etc/scqsd/scqsd.conf. Start the quorum server by running

opensol-host# /usr/cluster/bin/clquorumserver start 9000

The quorum server can now be configured into the cluster to get it fully operational.

One one of the cluster nodes run clsetup. Answer ‘yes’ to confirm you have finished the initial cluster setup and ‘yes’ to add a quorum device. Now pick option 3 (Quorum Server). Answer ‘yes’ to continue then give a name for the device, such as ‘opelsolhost’. Enter the IP of your host when prompted and 9000 as the port number. Allow it to proceed then choose ‘yes’ to reset installmode. The cluster is now properly established.

You can check the quorum registration on the OpenSolaris host by doing /usr/cluster/bin/clquorumserver show, you should see something like this:

opensol-host#/usr/cluster/bin/clquorumserver show
---  Cluster DEV1 (id 0x484E9DB9) Registrations ---
Node ID:                      1
Registration key:           0x484e9db900000001
Node ID:                      2
Registration key:           0x484e9db900000002

Provisioning storage

To enable the creation of the Apache and MySQL zones it’s necessary to present some storage to the cluster that can be shared between them.

As OpenSolaris is running on the host, take advantage of the in built iSCSI support in ZFS. First create some ZFS Volumes, one for each clustered service. eg.

opensol-host# zfs create -V 8g rpool/apache
opensol-host# zfs create -V 8g rpool/mysql

Now enable iSCSI export on them.

opensol-host# zfs set shareiscsi=on rpool/apache
opensol-host# zfs set shareiscsi=on rpool/mysql

And confirm this with iscsitadm list target -v

opensol-host# iscsitadm list target -v
Target: rpool/apache
iSCSI Name: iqn.1986-03.com.sun:02:29f26a9a-facb-4d29-adf8-adef67b21a00
Alias: rpool/apache
--snip--
Size: 8.0G
Backing store: /dev/zvol/rdsk/rpool/apache
Status: online
Target: rpool/mysql
iSCSI Name: iqn.1986-03.com.sun:02:1a782d05-c7c5-cd74-8ea3-b6c094cb3cf8
Alias: rpool/mysql
--snip--
Size: 8.0G
Backing store: /dev/zvol/rdsk/rpool/mysql
Status: online

Now configure the nodes to see the presented storage. Do this on both of the nodes, replacing 192.168.0.104 with the IP of your host.

both-nodes# iscsiadm modify discovery –sendtargets enable
both-nodes# iscsiadm add discovery-address 192.168.0.104
both-nodes# svcadm enable network/iscsi_initiator

And then to confirm:

both-nodes# iscsiadm list target -S
Target: iqn.1986-03.com.sun:02:1a782d05-c7c5-cd74-8ea3-b6c094cb3cf8
Alias: rpool/mysql
--snip--
OS Device Name: /dev/rdsk/c2t01000017F202642400002A00484FCCC1d0s2
Target: iqn.1986-03.com.sun:02:29f26a9a-facb-4d29-adf8-adef67b21a00
Alias: rpool/apache
--snip--
OS Device Name: /dev/rdsk/c2t01000017F202642400002A00484FCCBFd0s2

Make a note of the OS Device Name to Alias matching as you need to put the right LUN into the correct resource group.

You can also confirm the storage is available by running format, eg:

both-nodes# format
Searching for disks...done
AVAILABLE DISK SELECTIONS:
0. c0d0
        /pci@0,0/pci-ide@1,1/ide@0/cmdk@0,0
1. c2t01000017F202642400002A00484FCCBFd0
        /scsi_vhci/disk@g01000017f202642400002a00484fccbf
2. c2t01000017F202642400002A00484FCCC1d0
        /scsi_vhci/disk@g01000017f202642400002a00484fccc1

To make this storage available to the cluster you must populate the DID device database. This is performed via cldevice and only needs to be run on one of the nodes:

cxnode1# cldevice populate
Configuring DID devices
cldevice: (C507896) Inquiry on device “/dev/rdsk/c0d0s2″ failed.
did instance 5 created.
did subpath cxnode1:/dev/rdsk/c2t01000017F202642400002A00484FCCC1d0 created for instance 5.
did instance 6 created.
did subpath cxnode1:/dev/rdsk/c2t01000017F202642400002A00484FCCBFd0 created for instance 6.
Configuring the /dev/global directory (global devices)
obtaining access to all attached disks

Then list the device database to check the storage is available

cxnode1# cldevice list -v
DID Device          Full Device Path
----------          ----------------
d1                  cxnode1:/dev/rdsk/c1t0d0
d2                  cxnode1:/dev/rdsk/c0d0
d3                  cxnode2:/dev/rdsk/c1t0d0
d4                  cxnode2:/dev/rdsk/c0d0
d5                  cxnode1:/dev/rdsk/c2t01000017F202642400002A00484FCCC1d0
d5                  cxnode2:/dev/rdsk/c2t01000017F202642400002A00484FCCC1d0
d6                  cxnode1:/dev/rdsk/c2t01000017F202642400002A00484FCCBFd0
d6                  cxnode2:/dev/rdsk/c2t01000017F202642400002A00484FCCBFd0

Now the basic configuration of the cluster is fairly complete. It’s a good opportunity to shut it down and take a snapshot. When shutting down the entire cluster you must use cluster shutdown rather than just shutting down the individual nodes. If you don’t then you must bring up the nodes in the reverse order of shutting them down. For an immediate shutdown do cluster shutdown -y -g 0.

Configuring the Storage

Originally I was planning to create some SVM metasets and add the disks to those, however this doesn’t appear to be possible with iSCSI LUNs yet, not even if you use the VirtualBox built in iSCSI initiator support, which results in the storage appearing as local disks. So instead I settled on using ZFS-HA to manage the disks. The process for this, as with most zfs stuff, is fairly straightforward.

First create a ZFS storage pool for each clustered service. We’ll use the DID device number here, so make sure you follow list back via cldevice list -v and iscsiadm list target -S to ensure you put the correct target into the correct ZFS pool.

In order for the storage to be added to ZFS it needs to have a ‘fdisk’ partition added to it. One one of the nodes run fdisk against the two devices, accepting the default partition layout.:

cxnode1# fdisk /dev/rdsk/c2t01000017F202642400002A00484B5467d0p0
No fdisk table exists. The default partition for the disk is:
a 100% “SOLARIS System” partition
Type “y” to accept the default partition,  otherwise type “n” to edit the
partition table.

The standard SMI label needs to be replaced with an EFI one. When I’ve been working with ZFS previously this has always happened automatically but it didn’t work for me this time, possibly because we are going to add the DID device to the zpool rather than a traditional disk device. To change it manually run format with the -e option.

cxnode1# format -e
Searching for disks…done

AVAILABLE DISK SELECTIONS:
0. c0d0
        /pci@0,0/pci-ide@1,1/ide@0/cmdk@0,0
1. c2t01000017F202642400002A00484FCCBFd0
        /scsi_vhci/disk@g01000017f202642400002a00484fccbf
2. c2t01000017F202642400002A00484FCCC1d0
        /scsi_vhci/disk@g01000017f202642400002a00484fccc1
Specify disk (enter its number): 2
selecting c2t01000017F202642400002A00484FCCC1d0
[disk formatted]
Error occurred with device in use checking: No such device
format> label
Error occurred with device in use checking: No such device
[0] SMI Label
[1] EFI Label
Specify Label type[0]: 1
Warning: This disk has an SMI label. Changing to EFI label will erase all
current partitions.
Continue? y
format> p
partition> p
Current partition table (default):
Total disk sectors available: 16760798 + 16384 (reserved sectors)

Part      Tag    Flag     First Sector        Size        Last Sector
0        usr    wm                34       7.99GB         16760798
1 unassigned    wm                 0          0              0
2 unassigned    wm                 0          0              0
3 unassigned    wm                 0          0              0
4 unassigned    wm                 0          0              0
5 unassigned    wm                 0          0              0
6 unassigned    wm                 0          0              0
7 unassigned    wm                 0          0              0
8   reserved    wm          16760799       8.00MB         16777182

If you’d like to suppress the “Error occurred with device in use checking: No such device” warning then set ‘NOINUSE_CHECK’ as an environment variable. It seems to be this bug that’s causing the warning.

Now the disks can be added to a ZFS pool. Make sure you add the correct DID device here:

cxnode1# zpool create apache-pool /dev/did/dsk/d6s0
cxnode1# zpool create mysql-pool /dev/did/dsk/d5s0

Then confirm these are available as expected:

cxnode1# zpool list
NAME          SIZE   USED  AVAIL    CAP  HEALTH  ALTROOT
apache-pool  7.94G   111K  7.94G     0%  ONLINE  -
mysql-pool   7.94G   111K  7.94G     0%  ONLINE  -

Now create some filesystems in the pools. For each service three file systems are required

• zone – This will become the zone root
• data – This will become /data within the zone and be used to store application data
• params – This will be used to store a cluster parameter file

cxnode1# zfs create apache-pool/zone
cxnode1# zfs create apache-pool/data
cxnode1# zfs create apache-pool/params
cxnode1# zfs create mysql-pool/zone
cxnode1# zfs create mysql-pool/data
cxnode1# zfs create mysql-pool/params

To make the storage usable with the cluster it needs to be configured into a resource group. Create two resource groups, one for each service

cxnode1# clresourcegroup create apache-rg
cxnode1# clresourcegroup create mysql-rg

SUNW.HAStoragePlus is the resource type that can manage ZFS storage, along with SVM and VxVM. It needs to be registered with the cluster with clresourcetype register, this only needs to be performed on one node.

cxnode1# clresourcetype register SUNW.HAStoragePlus

Create clustered resources to manage the ZFS pools:

cxnode1# clresource create -g apache-rg -t SUNW.HAStoragePlus -p Zpools=apache-pool apache-stor
cxnode2# clresource create -g mysql-rg -t SUNW.HAStoragePlus -p Zpools=mysql-pool mysql-stor

The resource groups will currently be in ‘Unmanaged’ state, you can confirm this with clresourcegroup status. To bring them under cluster management bring them online on the first node:

cxnode1# clresourcegroup online -M -n cxnode1 apache-rg

Then check the file systems are available are available on the first node but not on the second.

cxnode1# zfs list
NAME               USED  AVAIL  REFER  MOUNTPOINT
apache-pool        158K  7.81G    21K  //apache-pool
apache-pool/data    18K  7.81G    18K  //apache-pool/data
apache-pool/params 18K  7.81G    18K  //apache-pool/params
apache-pool/zone    18K  7.81G    18K  //apahce-pool/zone

cxnode2# zfs list
no datasets available

Now move the pool to the other node and check that it becomes available on that node

cxnode1# clresourcegroup switch -n cxnode2 apache-rg

cxnode1# zfs list
no datasets available

cxnode2#  zfs list
NAME               USED  AVAIL  REFER  MOUNTPOINT
apache-pool        158K  7.81G    21K  //apache-pool
apache-pool/data    18K  7.81G    18K  //apache-pool/data
apache-pool/params 18K  7.81G    18K  //apache-pool/params
apache-pool/zone    18K  7.81G    18K  //apahce-pool/zone

When you’re happy switch it back to the first node:

cxnode1# clresourcegroup switch -n cxnode1 apache-rg

Then repeat the above online and failover tests for mysql-rg and mysql-pool

As the ZFS pools have now been added as cluster managed resources you must now use the cluster to mange them. Don’t perform export/import operations manually.

Preparing Apache and MySQL zones for clustering

To provide a clustered Apache and MySQL service two zones are going to be created. Whilst these services could equally well be clustered without the use of zones I decided to go down this path so that the benefits of zones could be enjoyed in tandem with the clustering.

A full example run through for configuring MySQL in a clustered zone is provided by Sun in the official documentation if you require further information.

It’s worth pointing out that if you are following this plan then it will create zones on ZFS devices, currently this is not supported for ‘Live Upgrade’ at present, so you are restricting your upgrade paths in the future. If you do decide you need to ‘Live Upgrade’ the cluster at some point in the future then you could remove the zones, do the upgrade, and then re-create the zones. If you don’t want to do this then consider using raw disk slices with the cluster rather than ZFS.

For each zone the pool/zone file system will be used for the zone root, then the pool/data file system will be delegated to the zone’s control which will be used for the application to store it’s data, i.e. the MySQL databases or the Apache document root.

To provide IP addressing for the zones SUNW.LogicalHostname resources will be used, rather than directly configuring the zone with an IP addres.

First entries need adding to /etc/hosts on both nodes, eg:

192.168.0.118 apache-zone
192.168.0.119 mysql-zone

Then create LogicalHostname resources for each address:

cxnode1# clreslogicalhostname create -g apache-rg -h apache-zone apache-addr
cxnode1# clreslogicalhostname create -g mysql-rg -h mysql-zone mysql-addr

Now it’s time to create the zones. This is a really simple zone configuration, you could add resource controls or other features as desired. autoboot must be set to false as the cluster will be managing the starting and stopping of the zone.

cxnode1# zonecfg -z apache
apache: No such zone configured
Use 'create' to begin configuring a new zone.
zonecfg:apache> create
zonecfg:apache> set zonepath=/apache-pool/zone
zonecfg:apache> set autoboot=false
zonecfg:apache> add dataset
zonecfg:apache:dataset> set name=apache-pool/data
zonecfg:apache:dataset> end
zonecfg:apache> verify
zonecfg:apache> commit
zonecfg:apache> exit
cxnode1# zonecfg -z mysql
mysql: No such zone configured
Use 'create' to begin configuring a new zone.
zonecfg:mysql> create
zonecfg:mysql> set zonepath=/mysql-pool/zone
zonecfg:mysql> set autoboot=false
zonecfg:mysql> add dataset
zonecfg:mysql:dataset>set name=mysql-pool/data
zonecfg:mysql:dataset> end
zonecfg:mysql> verify
zonecfg:mysql> commit
zonecfg:mysql> exit

To enable the zones to be installed we must change the permissions on the zone roots:

cxnode1# chmod 700 /apache-pool/zone
cxnode1# chmod 700 /mysql-pool/zone

Now install the zones:

# zoneadm -z apache install
Preparing to install zone <apache>.
Creating list of files to copy from the global zone.
Copying <9668> files to the zone.
Initializing zone product registry. Determining zone package initialization order.
Preparing to initialize <1346> packages on the zone.
Initialized <1346> packages on zone.
Zone <apache> is initialized.
Installation of these packages generated warnings:
<sunwvboxguest>
The file </apache-pool/zone/root/var/sadm/system/logs/install_log> contains a log of the zone installation.
# zoneadm -z mysql install
etc..

The documentation has this to say about configuring zones:

Caution: If the zone is to run in a failover configuration, each node being able to host that zone must have the exact same zone configuration for that zone. After installing the zone on the first node, the zone’s zone path already exists on the zones’s disk storage. Therefore it must get removed on the next node prior to successfully create and install the zone.[..] Only the zone’s zone path created on the last node will be kept as the final zone path for the failover zone. For that reason any configuration and customization within the failover zone should get performed after the failover zone is known to all nodes that should be able to host it.

To achieve this the newly created zone must be destroyed and recreated on the second node. To my mind this is a really ugly way of achieving this, the cluster should be able to manage this itself and make suitable configuration changes on a node when the zone is configured into the cluster.

In later releases of Sun Cluster 3.1 the recommended way to manage this configuration was that the /etc/zones files be hacked to replicate the configuration of the zones from one node to another. However this method is not supported any more so the official instructions will be followed.

To do this migrate the storage for the zones to the other node

cxnode1# clresourcegroup switch -n cxnode2 apache-rg
cxnode1# clresourcegroup switch -n cxnode2 mysql-rg

Then delete the previously installed zone roots on the second node

cxnode2# rm -rf /apache-pool/zone/*
cxnode2# rm -rf /mysql-pool/zone/*

Now repeat the zonecfg and zoneadm steps above to recreate both of the zones.

When the zone installs have completed again move the storage back to the first node.

cxnode1# clresourcegroup switch -n cxnode1 apache-rg
cxnode1# clresourcegroup switch -n cxnode1 mysql-rg

The zones can now be booted and configured. Repeat these steps for Apache and the the MySQL zone. Boot the zone:

cxnode1# zoneadm -z apache boot

I got this error when booting the zones Unable to set route for interface lo0 to *??9?x , I m not sure what this means but it doesn’t seem to impact anything.

Login to the zone’s console to configure it:

cxnode1# zlogin -C apache

You’ll be asked a few questions to configure the zone. Choose language, terminal type and time zone information as appropriate. Enter the same hostname as you used above, eg ‘apache-zone’ or ‘mysql-zone’. I received some alerts about avahi-bridge-dsd failing to start when booting, as far as I can tell it’s some sort of Bonjour networking thing, we don’t need it here so it’s ok to disable. You can also disable some other services that are not required to free up some resources

apache-zone# svcadm disable cde-login
apache-zone# svcadm disable sendmail
apache-zone# svcadm disable webconsole
apache-zone# svcadm disable avahi-bridge-dsd
apache-zone# svcadm disable ppd-cache-update

Now mount the zfs file systems that have delegated to the zone to an appropriate place. To do this on the Apache zone:

apache-zone# zfs set mountpoint=/data apache-pool/data

And on the MySQL zone:

mysql-zone# zfs set mountpoint=/data mysql-pool/data

Wait for the zone to finish booting and check you don’t have any failed services with svcs -xv. Then shut the zone down and repeat for the other zone.

Clustering the zones

Before proceeding further ensure the storage is available on the first node, fail it over if necessary. Also make sure the zones are shut down.

To enable clustering for the zones they must be registered with the cluster. To do this a script called sczbt_register is provided. To use this a configuration file must be completed and then registered. A sample configuration file is provided at /opt/SUNWsczone/sczbt/util/sczbt_config, this is also the file that will be read by default by sczbt_register. It is recommended to copy this file to some other place for future reference, then run sczbt_register against that. Comments are included in the file to explain the options, or see the official docs for more info.

Copy /opt/SUNWsczone/sczbt/util/sczbt_config to /etc/sczbt_config.apache and /etc/sczbt_config.mysql and edit as follows

 /etc/sczbt_config.apache:

RS=apache-zone
RG=apache-rg
PARAMETERDIR=/apache-pool/params
SC_NETWORK=true
SC_LH=apache-addr
FAILOVER=true
HAS_RS=apache-stor
Zonename="apache"
Zonebrand="native"
Zonebootopt=""
Milestone="multi-user-server"
LXrunlevel="3"
SLrunlevel="3"
Mounts=""

 /etc/sczbt_config.mysql

RS=mysql-zone
RG=mysql-rg
PARAMETERDIR=/mysql-pool/params
SC_NETWORK=true
SC_LH=mysql-addr
FAILOVER=true
HAS_RS=mysql-stor
Zonename="mysql"
Zonebrand="native"
Zonebootopt=""
Milestone="multi-user-server"
LXrunlevel="3"
SLrunlevel="3"
Mounts=""

The zones can now be registered. First you need to register the SUNW.gds resource type. On one node do:

cxnode1# clresourcetype register SUNW.gds

Then register the two zones

cxnode1# /opt/SUNWsczone/sczbt/util/sczbt_register -f /etc/sczbt_config.apache
sourcing /etc/sczbt_config.apache
Registration of resource apache-zone succeeded.
Validation of resource apache-zone succeeded.

cxnode1# /opt/SUNWsczone/sczbt/util/sczbt_register -f /etc/sczbt_config.mysql
sourcing /etc/sczbt_config.mysql
Registration of resource mysql-zone succeeded.
Validation of resource mysql-zone succeeded.

Then enable the Apache zone and log in to it. You should see the LogicalHostname resource has been assigned to the zone

cxnode1# clresource enable apache-zone
cxnode1# zoneadm list -cv
ID NAME             STATUS     PATH                           BRAND    IP
0 global           running    /                              native   shared
1 apache           running    /apache-pool/zone              native   shared
- mysql            installed  /mysql-pool/zone               native   shared
cxnode1# zlogin apache
[Connected to zone 'apache' pts/2]
Last login: Tue Jun 17 20:23:08 on pts/2
Sun Microsystems Inc.   SunOS 5.11      snv_86  January 2008
apache-zone# ifconfig -a
lo0:1: flags=2001000849 mtu 8232 index 1
inet 127.0.0.1 netmask ff000000
e1000g1:1: flags=201040843 mtu 1500 index 3
inet 192.168.0.118 netmask ffffff00 broadcast 192.168.0.255

Then test a failover and failback of the zone.

cxnode1# clresourcegroup switch -n cxnode2 apache-rg
cxnode1# clresourcegroup switch -n cxnode1 apache-rg

Repeat the same checks for the MySQL zone.

Installing and configuring MySQL

MySQL 5.0.45 is installed by default with the Solaris Express install we have performed. This consists of the SUNWmysql5r, SUNWmysql5u and SUNWmysql5test packages.

The installation can be used pretty much unmodified, the only modification needed is to repoint the MySQL data directory to the delegated zfs file system. This is done by modifying the SMF properties for the service, you cannot make this change by modifying my.cfg. Make the change with svccfg:

mysql-zone# svccfg
svc:> select svc:/application/database/mysql:version_50
svc:/application/database/mysql:version_50> setprop mysql/data = /data/mysql
svc:/application/database/mysql:version_50> refresh
svc:/application/database/mysql:version_50> end

Then create the directory for the databases and one for some logs

mysql-zone# mkdir /data/mysql
mysql-zone# mkdir /data/logs
mysql-zone# chown mysql:mysql /data/mysql
mysql-zone# chown mysql:mysql /data/logs

Now start up the database and check that it starts ok

mysql-zone# svcadm enable mysql:version_50
mysql-zone# svcs mysql:version_50
STATE          STIME    FMRI
online         10:38:54 svc:/application/database/mysql:version_50

Now set a password for the root users for the database, it’s set to ‘root’ in this case.

mysql-zone# /usr/mysql/5.0/bin/mysqladmin -u root password root
mysql-zone# /usr/mysql/5.0/bin/mysqladmin -u root -h localhost -p password root
Enter password:root

The /etc/hosts file in the zone needs to be modified so that ‘mysql-zone’ is the name for the clustered IP address for the zone rather than the localhost, also the address for the apache-zone needs to be added.

127.0.0.1       localhost       loghost
192.168.0.119   mysql-zone
192.168.0.118   apache-zone

Allow access to the root user to connect from the Apache zone.

mysql-zone# /usr/mysql/5.0/bin/mysql -p
Enter password: root
mysql> GRANT ALL ON *.* TO 'root'@'apache-zone' identified by 'root';

MySQL is now configured and ready to be clustered. We’ll be using a process loosely based on the one documented here. Alternatively it would be possible to use SMF to manage the service, you can see an example of that method in the Apache configuration later.

First a user for the fault monitor must be created along with a test database for it to use. A script is provided with the agent to do this for you. It will grant the fault monitor user ‘PROCESS, SELECT, RELOAD, SHUTDOWN, SUPER’ on all databases, then ALL privileges on the test database.

To create the required users you need to provide a config file. Copy the supplied template into /etc and edit it there

mysql-zone# cp /opt/SUNWscmys/util/mysql_config /etc
mysql-zone# vi /etc/mysql_config

Use these values, note that MYSQL_DATADIR is the location of the my.cnf, not the directory to the databases. The meaning of DATADIR changed in 5.0.3 to mean the location of the data and not the config directory, but for this configuration it should point to the config directory.

MYSQL_BASE=/usr/mysql/5.0
MYSQL_USER=root
MYSQL_PASSWD=root
MYSQL_HOST=mysql-zone
FMUSER=fmuser
FMPASS=fmpass
MYSQL_SOCK=/tmp/mysql.sock
MYSQL_NIC_HOSTNAME="mysql-zone"
MYSQL_DATADIR=/etc/mysql/5.0

Then run the registration script

mysql-zone# /opt/SUNWscmys/util/mysql_register -f /etc/mysql_config
sourcing /etc/mysql_config and create a working copy under /opt/SUNWscmys/util/mysql_config.work
MySQL version 5 detected on 5.11/SC3.2
Check if the MySQL server is running and accepting connections
Add faulmonitor user (fmuser) with password (fmpass) with Process-,Select-, Reload- and Shutdown-privileges to user table for mysql database for host mysql-zone
Add SUPER privilege for fmuser@mysql-zone
Create test-database sc3_test_database
Grant all privileges to sc3_test_database for faultmonitor-user fmuser for host mysql-zone
Flush all privileges
Mysql configuration for HA is done

Now shut down the database so it can be bought online by the cluster.

mysql-zone# svcadm disable mysql:version_50

Drop back to the global zone and copy the MySQL agent configuration template to /etc

cxndoe1# cp /opt/SUNWscmys/util/ha_mysql_config /etc/ha_mysql_config

Use these settings, this time the ‘DATADIR’ should be set to point to the actual data location and not the location of the config. Descriptions of the configuration is given in the file:

RS=mysql-server
RG=mysql-rg
PORT=3306
LH=mysql-addr
HAS_RS=mysql-stor
ZONE=mysql
ZONE_BT=mysql-zone
PROJECT=
BASEDIR=/usr/mysql/5.0
DATADIR=/data/mysql
MYSQLUSER=mysql
MYSQLHOST=mysql-zone
FMUSER=fmuser
FMPASS=fmpass
LOGDIR=/data/logs/
CHECK=NO

Now register this with the cluster:

cxnode1# /opt/SUNWscmys/util/ha_mysql_register -f /etc/ha_mysql_config
sourcing /etc/ha_mysql_config and create a working copy under /opt/SUNWscmys/util/ha_mysql_config.work
clean up the manifest / smf resource
sourcing /opt/SUNWscmys/util/ha_mysql_config
disabling the smf service svc:/application/sczone-agents:
removing the smf service svc:/application/sczone-agents:
removing the smf manifest /var/svc/manifest/application/sczone-agents/.xml
sourcing /tmp/ha_mysql_config.work
/var/svc/manifest/application/sczone-agents/mysql-server.xml successfully created
/var/svc/manifest/application/sczone-agents/mysql-server.xml successfully validated
/var/svc/manifest/application/sczone-agents/mysql-server.xml successfully imported
Manifest svc:/application/sczone-agents:mysql-server was created in zone mysql
Registering the zone smf resource
sourcing /opt/SUNWsczone/sczsmf/util/sczsmf_config
Registration of resource mysql-server succeeded.
Validation of resource mysql-server succeeded.
remove the working copy /opt/SUNWscmys/util/ha_mysql_config.work

Before bringing this online a tweak is needed to the supplied agent scripts. As mentioned briefly above the use of ‘DATADIR’ is a bit broken. If you try to bring MySQL online now it will fail as it won’t be able to find its configuration file. The agent scripts have this hard coded to ${MYSQL_DATADIR}/my.cnf which is no use for our purposes.

In the zone edit /opt/SUNWscmys/bin/functions and make this replacement, ensure you edit the copy in the MySQL zone and not the one in the global zone.

MYSQL_DEFAULT_FILE=${MYSQL_DATADIR}/my.cnf

with

MYSQL_DEFAULT_FILE=/etc/mysql/5.0/my.cnf

The mysql-server can now be enabled.

cxnode1# clresource enable mysql-server

Configuring Apache

Apache is going to be used to provide a web front end to the MySQL install, via the ubiquitous phpMyAdmin. The supplied Apache install (at /usr/apache2/2.2) is going to be used. As Apahce will be running in a zone it can be used unmodified, keeping the configuration in /etc and not worrying about any potential conflicts with other Apache installs.

At present the supplied Apache resource type does not directly support running the resource in a zone, or at least I couldn’t figure it out. So instead some of the provided zone monitoring tools are going to be used to ensure Apache is up and running. This uses a combination of SMF and a shell script.

To begin Apache must be configured. Bring the zone online on one of the nodes and log in to it. The configuration file for Apache is at /etc/apache2/2.2/httpd.conf. Only a small tweak is required to move the document root onto the zfs file system we have prepared for it. You could, if desired, also move other parts of the configuration, such as the log location. For this example just change DocumentRoot to be /data/htdocs and update the Directory stanza a page or so below it. Then do a mkdir on /data/htdocs. That completes our very simple Apache configuration.

So start it up ’svcadm enable apache22′. Download phpMyAdmin from here. Solaris now ships with p7zip to manage 7z files, so you could download that version to save a bit of bandwidth if you like. You can extract them with pzip -d [filename]. Once extracted move the extracted directory to /data/htdocs/phpmyadmin.

Add mysql-zone to /etc/hosts eg

192.168.0.119    mysql-zone

Modify config.inc.php setting these two values:

$cfg['Servers'][$i]['host'] = 'mysql-zone';
$cfg['blowfish_secret'] = 'enter a random value here';

To enable monitoring of the Apache instance we need a simple probe script. Make a directory /opt/probes in the zone and create a file called probe-apache.ksh with this content:

#!/usr/bin/ksh
if echo "GET; exit" | mconnect -p 80 > /dev/null 2>&1
then
exit 0
else
exit 100
fi

Then chmod 755 /opt/probes/probe-apache.ksh. All this does is a simple connect on port 80, it could be replaced with something more complex if needed. Finally disable Apache so that the cluster can start it:

apache-zone# svcadm disable apache22

Drop back to the global zone and copy /opt/SUNWsczone/sczsmf/util/sczsmf_config to /etc/sczsmf_config.apache and set the following settings

RS=apache-server
RG=apache-rg
SCZBT_RS=apache-zone
ZONE=apache
SERVICE=apache22
RECURSIVE=true
STATE=true
SERVICE_PROBE="/opt/probes/probe-apache.ksh"

 Now this can be registered with the cluster:

cxnode1# /opt/SUNWsczone/sczsmf/util/sczsmf_register -f /etc/sczsmf_config.apache
sourcing /etc/sczsmf_config.apache
Registration of resource apache-server succeeded.
Validation of resource apache-server succeeded.

Now enable Apache and check that it’s functioning correctly:

clresource enable apache-server

You can now browse to /phpmyadmin and check that everything it working!

Conclusions

When I started this work I wasn’t sure whether it was going to be possible or not, but despite a couple of bumps along the way I’m happy with the end result. Whilst it might not be a perfect match for a real cluster it certainly provides enough opportunity for testing and for use in training.

To give the Shader Model 2 a whirl I fired up Half Life 2. It had refused to run in any previous VMware Fusion release but this time it runs! Ok, it’s runs but it’s nowhere near playable as it’s getting about <5 FPS. However it’s a promising development.

I will look further into the workings of OpenSolaris in some future posts but for now I thought I’d take a quick look at how the install ran on a bare metal Mac Pro (2006 version).

OpenSolaris is distributed as a Live CD with an integrated installer to make a permanent install once you’ve had a chance to poke around and check things like hardware compatibility. You can download the CD from Sun or from one of these mirrors.

Previously I’d had some problems with Solaris Express not being able to create a suitable ‘fdisk’ partition on the disk I was trying to install to. To see if this was still an issue I erased my existing Solaris Express drive and partitioned it with a standard OS X EFI/GUID partition table and a single HFS+ volume.

Because I’m rather paranoid about my data I powered down and removed my OS X drives from the machine before trying the OpenSolaris install. Then I booted up from the Live CD which loaded a standard GRUB menu, the banner indicated that this release is on a base of snv_86. I was prompted to pick a keyboard layout and a desktop language, then after a short while a fairly normal looking Gnome desktop poped up.

An item on the desktop that caught my attention was the ‘Device Driver Utility’. This bought up a list all the detected devices in the system and whether an appropriate driver was found for them.

Most devices were detected fine with the following exceptions.

• Broadcom Corporation BCM4238 802.11/a/b/g/n
• ACPI SMBus 1.0 Host Controller
• Intel Corporation 6311ESB/6321ESB I/OxAPIC Interrupt Controller
• Standard PC COM port – Driver Misconfigured

From within the device driver tool there’s an option to submit the results to the central hardware compatibility list.

I have a dual monitor setup, connected to an ATI 1900XT graphics card. Whilst a full resolution display was shown on the larger monitor both monitors displayed the same image, with the lower resolution display showing the top-left section of the other monitor. The ATI card is showing as using the ‘vgatext’ driver which is, presumably, some generic video driver. Unfortunately ATI do not make Solaris drivers for their cards, one reason why I’m thinking about moving to the 8800GT card that is now available. 

Since the LiveCD booted ok it seems fairly likely that a full install should work, providing the disks could be detected properly. The ‘Install OpenSolaris’ app is a bright, unmissable icon on the desktop. Picking it bought up with installer which after the initial welcome page displayed the disk chooser.

The disk was displayed correctly, I picked the option to use the whole disk. Then I went through simple time zone and locale options followed by a screen to set a non-root user and some passwords. That was it for the installer, it really couldn’t have been any more simple

The install ran for about 20 minutes and completed without error. After it completed I rebooted the machine and it rebooted into a GRUB menu with ‘OpenSolaris 2008.05 snv_86_rc3 X86′ as the default, and only, option. This booted fine to a GNOME login prompt.

I’ll poke around in OpenSolaris more over the coming weeks and will hopefully post some further thoughts here.

I’d always assumed that the train WiFi was supported by a 3G network, or something similar. However my outbound IP address is 217.28.34.132 which a whois indicates belongs to

inetnum: 217.28.32.0 - 217.28.35.15
netname: NSAB-NET
descr: NSAB backbone and address-pools for VPN Services
##########################
In case of improper use, please contact:
<abuse@nsab.se>
##########################


nsab.se redirects to SES SIRIUS AB which is a satellite broadband provider. Well there you go.

Now if only the train was on time…

P.S. Google comes up in Swedish