I figured since I had to gather this information from all over the web that I will score some internets points by writing about how I did it and collecting it all in one place. Also it gives me something to write in a blog so now I'm cool.
Problem: Need a shared disk source for reading and writing to be used by a Linux cluster of 3 nodes.
I didn't want to use NFS or some of the higher level solutions due to speed and propagation delays, so I looked into some more advanced configurations. I settled on using open-iscsi for the shared disk and ocfs2 as the clustered filesystem. My servers are Ubuntu Hardy Heron (8.04.1 LTS). It wasn't as difficult as I first thought it would be, it was harder trying to decide what I should use.
iSCSI Server
Install on only 1 server, I didn't get into creating failover for this yet.
package: iscsitarget
config file: /etc/ietd.conf
I didn't configure any authentication, I figured its on its own subnet and I'm not one to lock down the security so much that it raises more problems. Plus I had no clue what I was doing and I didn't want authentication to be the thing to stop since there's a few parts that need to work together.
The only things I edited:
The target's ID which was something like
iqn:2006-03.com.example:hoth-esx
and all I did was change the domain and the cluster name after the :
The Lun (not a clue what that stands for) to something like
Lun 0 Path=/dev/sdb,Type=fileio
where /dev/sdb is the disk I was dedicating to this use.
Restart iscsitarget and you should be good to go
Each node that's going to be accessing the shared disk needs to have the rest installed
iSCSI Client
package: open-iscsi
config file: /etc/iscsi/iscsid.conf
The only thing I did was change: node.startup to automatic
There's a lot of settings that I haven't a clue what they do but it worked just fine like this. If you're doing authentication you need to set it up in here too.
Restart open-iscsi
Make the iscsi connection, this only needs to be done once to initially set it up and both commands need to be run on all connecting nodes, obviously change the ip
iscsiadm -m discovery -t sendtargets -p 192.168.1.60
Then copy what that returned and plug it into:
iscsiadm --mode node --targetname iqn:2006-03.com.example:hoth-esx --portal 192.168.1.60:3260 --login
Restart open-iscsi again check your messages log
tail -f /var/log/messages
You should be able to find which device was assigned to it, this can be different from the device on your iscsitarget server, mine was /dev/sdb
OCFS2
packages: ocfs2-tools ocfs2console
config file: /etc/ocfs2/cluster.conf
The config file and folder wasn't created by default, there's an example in
/usr/share/doc/ocfs2-tools/examples/cluster.conf
Copy this to /etc/ocfs2/cluster.conf
I found this pretty straight forward to customize for my needs, this file needs to be identical on all nodes.
After editing the file I told the ocfs2 package to reconfigure itself which brought up a series of prompts
dpkg-reconfigure ocfs2-tools
The only thing I changed was the Cluster Name to match what I put in the cluster.conf file and took the defaults on the rest.
Restart your OCFS2 services:
/etc/init.d/o2cb restart
/etc/init.d/ocfs2 restart
On only 1 node (I did it on the first node I set up) you need to set up the drive partition and format it for ocfs2, nothing complicated
fdisk /dev/sdc
mkfs.ocfs2 /dev/sdc1
All nodes again:
You should now be able to mount this drive
mount -t ocfs2 /dev/sdc1 /mnt/iscsi
I then added an entry to the /etc/fstab so the drive will mount on boot. NOTE: don't use the UUID, the iscsi and the physical device get the same ID and they become confused and crash the system
/dev/sdc1 /mnt/iscsi ocfs2 _netdev 0 0
Its important to have the _netdev option
That's it, your setup should be working. The only problem I have is since my iSCSI target server doesn't have a failover, if that is reboot all the nodes spike in cpu and become non-responsive for a few seconds. Then they can't regain the connection after the server is back online, when this happens either reboot each node or stop the ocfs2 and iscsi services then bring them back up in the proper order.
Shutdown order:
unmount drive if currently mounted
ocfs2
o2cb
open-iscsi
Reverse the order for start up.
