iSCSI With OCFS2

I figured since I had to gather this information from all over the web that I will score some internets points by writing about how I did it and collecting it all in one place. Also it gives me something to write in a blog so now I'm cool.

Problem: Need a shared disk source for reading and writing to be used by a Linux cluster of 3 nodes.

I didn't want to use NFS or some of the higher level solutions due to speed and propagation delays, so I looked into some more advanced configurations. I settled on using open-iscsi for the shared disk and ocfs2 as the clustered filesystem. My servers are Ubuntu Hardy Heron (8.04.1 LTS). It wasn't as difficult as I first thought it would be, it was harder trying to decide what I should use.

iSCSI Server

Install on only 1 server, I didn't get into creating failover for this yet.

package: iscsitarget

config file: /etc/ietd.conf

I didn't configure any authentication, I figured its on its own subnet and I'm not one to lock down the security so much that it raises more problems. Plus I had no clue what I was doing and I didn't want authentication to be the thing to stop since there's a few parts that need to work together.

The only things I edited:

The target's ID which was something like

iqn:2006-03.com.example:hoth-esx

and all I did was change the domain and the cluster name after the :

The Lun (not a clue what that stands for) to something like

Lun 0 Path=/dev/sdb,Type=fileio

where /dev/sdb is the disk I was dedicating to this use.

Restart iscsitarget and you should be good to go

Each node that's going to be accessing the shared disk needs to have the rest installed

iSCSI Client

package: open-iscsi

config file: /etc/iscsi/iscsid.conf

The only thing I did was change: node.startup to automatic

There's a lot of settings that I haven't a clue what they do but it worked just fine like this. If you're doing authentication you need to set it up in here too.

Restart open-iscsi

Make the iscsi connection, this only needs to be done once to initially set it up and both commands need to be run on all connecting nodes, obviously change the ip

iscsiadm -m discovery -t sendtargets -p 192.168.1.60

Then copy what that returned and plug it into:

iscsiadm --mode node --targetname iqn:2006-03.com.example:hoth-esx --portal 192.168.1.60:3260 --login

Restart open-iscsi again check your messages log

tail -f /var/log/messages

You should be able to find which device was assigned to it, this can be different from the device on your iscsitarget server, mine was /dev/sdb

OCFS2

packages: ocfs2-tools ocfs2console

config file: /etc/ocfs2/cluster.conf

The config file and folder wasn't created by default, there's an example in

/usr/share/doc/ocfs2-tools/examples/cluster.conf

Copy this to /etc/ocfs2/cluster.conf

I found this pretty straight forward to customize for my needs, this file needs to be identical on all nodes.

After editing the file I told the ocfs2 package to reconfigure itself which brought up a series of prompts

dpkg-reconfigure ocfs2-tools

The only thing I changed was the Cluster Name to match what I put in the cluster.conf file and took the defaults on the rest.

Restart your OCFS2 services:

/etc/init.d/o2cb restart

/etc/init.d/ocfs2 restart

On only 1 node (I did it on the first node I set up) you need to set up the drive partition and format it for ocfs2, nothing complicated

fdisk /dev/sdc

mkfs.ocfs2 /dev/sdc1

All nodes again:

You should now be able to mount this drive

mount -t ocfs2 /dev/sdc1 /mnt/iscsi

I then added an entry to the /etc/fstab so the drive will mount on boot. NOTE: don't use the UUID, the iscsi and the physical device get the same ID and they become confused and crash the system

/dev/sdc1 /mnt/iscsi ocfs2 _netdev 0 0

Its important to have the _netdev option

That's it, your setup should be working. The only problem I have is since my iSCSI target server doesn't have a failover, if that is reboot all the nodes spike in cpu and become non-responsive for a few seconds. Then they can't regain the connection after the server is back online, when this happens either reboot each node or stop the ocfs2 and iscsi services then bring them back up in the proper order.

Shutdown order:

unmount drive if currently mounted

ocfs2

o2cb

open-iscsi

Reverse the order for start up.