Oracle: Unbreak my Linux Heart

Testing my new kickstart file to install Linux on my old Poweredge 2950 dell servers is fine, but multipath presents me with a new problem. We boot from SAN, even though the servers have local disks available. Eventually we will move towards Blades, so the experience is useful. But each server has a different number of LUNs available to it. The boot LUN is always LUN id 0. Without multipathd , this gave me /dev/sdb as the boot device. With multipathd , this gives me /dev/mapper/mpathX , where X is the number of LUNs I have. You may see the problem: The kickstart file must list the boot drive order and designate a place to put the MBR. However, I can't think of a way to tell kickstart to use the last mpath device... I wish there'd be a way to make /dev/mapper/mpath0 the boot LUN... Update: Well, it seems to be a limitation of multipath support in anaconda. I've booted the OEL 5.3 DVD ISO on 3 different bare metals and each server has a different number of LUNs associated w...

With Red Hat and Oracle Linux supporting multipathed installation from 5 update 2 on, I thought they should have changed the Kickstart Options as well, so that anaconda can be given the right parameters. And I was right. ignoredisks has been added to let the installer ignore SAN disks as well as for cluster setups. multipath has been added to handle dual HBAs cases where the same disk will be seen over two different paths. Update: except ignoredisks , there is really not much you need to do but boot the Linux kernel with the option ' mpath '. The multipath option lets you set/configure the multipath.conf configuration, I believe. Not used yet. In my case, I use this line to boot Linux and start a kickstart installation: linux mpath text nousbstorage ksdevice=eth0 ks=http://server/mpathks.cfg That's it. The first activates multipath-aware option in anaconda. The second uses text mode. The third parameter skips any USB devices in my PowerEdge 2950 (virtual floppy, d...

While on the topic of Support cycles and End-of-Life (EOL) notices... EMC has given PowerPath 5.0.0 for Linux an EOL from May 31st, 2009. So I'm checking if we'll upgrade to PowerPath 5.1.x or switch over to a fresh Linux release and use dm-multipath (aka dm-mpio, native Linux multi-pathing. blog'o thnet has a nice Comparison: EMC PowerPath vs. GNU/Linux dm-multipath for your convenience.

Just received notice that Red Hat Enterprise Linux 5.3 Beta has been out since October 28th... Sjeez! Why didn't I know that sooner...? Anyway the bèta will last until January 9th, 2009, so when I get from holidays, I'll have my work cut of out for me. RHEL 5U3 brings many, many virtualization (Xen) improvements. A new kernel (kernel-2.6.18-120.el5), rebased versions of monitoring tools (lm-sensors, net-snmp, openIPMI, openldap), new versions of yum and rpm, updated Kerberos subsystem, newer SAMBA (3.0.32) with support for Vista and Windows Server 2008, a ton kernel fixes and features, new hardware drivers (bcm43xx, bnx2/bnx2x, cpufreq, e1000/e1000e, mptscsi, qla24xx, etc.) and some interesting Technology Previews! dm-multipath install, which should allow anaconda to install Linux on systems with 2 or more HBAs. Support for ext4 file systems. CIFS (aka SAMBA) enhancements in the form of kernel DFS support and Kerberos security. Check the complete Release Notes for a full upda...

While on the subject, I found instructions on Using Device-Mapper Multipath in Red Hat's RHEL 4 update 6 (4u6). May help getting it to work with 4u5 too, or help decide if you should update... Update: here is a guide for RHEL 5, update 1 (PDF)

I got a mail today from another group working with Linux servers on Dell PE2950's. They have HP EVA boxes as their SAN, instead of our EMC DMX-3/4, and wanted to know how we got multi-pathing to work in Oracle's Enterprise Linux 4, update 5. They had heard that it was terrible and were thinking of using RHEL/OEL 5u2 instead. I told them, we were using PowerPath for this reason, and because EMC would only support us iff we used PowerPath. dm-multipath was not supported. Hmmm. Fair enough... While dm-multipathing is available in OEL 4u5, you have to go through some tricks to get it to support your root volume over dm-multipath. TuxyTurvy has instructions and people report it works. The trick he uses is to make a custom initrd image. Another solution may be to use HP StorageWorks for Linux or HP's own dm-multipath . Update: We going a different route... We'll install ESX 3.5 on the Dell servers and install Linux in VMs only, virtualizing the whole setup. ESX is compatibl...

I accidently did an update of a test server using ' yum update ', realizing I'd get a new kernel but forgetting the new kernel would break my PowerPath setup. It's EMC's software for getting multi-pathing and we use version 5.0.0 (EMCpower.LINUX-5.0.0-157.rhel). I should have used up2date because it knows how to update and skip undesired packages, such as the kernel... Oh well. *Nix hacks had some tips to fix PowerPath after kernel updates . You simply need to reinstall the package and force a replacement.

As a side-effect of the Linux boot process, I suddenly realized that our boot process is not fault-tolerant! We have 100+ servers that boot from SAN using two Qlogic 2460 HBAs. We installed EMC PowerPath 5.0.0 on OEL 4u5 to get multipathing and automatic fail-over in case a path fails. The OS is pretty well off in case of hardware faults. However, the boot process is not! Not even close! In stage 1, the boot loader reads the MBR, loads stage 1.5 so it can read the /boot ext2 partition where the kernel and initrd image are located. In our case, this is /dev/sdb1. But what if my HBA dies and /dev/sdb1 doesn't exist? It may be smart enough to try a device using the other path, but probably not. Also, since the kernel hasn't loaded yet, PowerPath does not exist and there is no multipath awareness to save the day... So once systems are installed and the number of disk partitions is stable, I can enhance GRUB with boot fallback systems , so it will try /dev/sdb1 and it that fails swi...

After a boot disk crash, I wanted to make sure existing data disks were unaffected. We use Oracle ASM for all our bare metal database servers and in this case there wasn't a proper backup of the (development) database. ASM marks its disks as being part of an ASM disk group and checking the first block of a partition. This lets you verify that the data still exists. Upon reinstallation of the OS and ASM software, you can then reuse existing ASM disks and groups and restore the data. To verify the disks are still marked as ASM disk groups, check the first block of the partition: dd count=1 if=/dev/sdc1 | od -c sdc being our first data disk after local disks (sda) and the boot LUN (sdb). Note the text string on the third line in capital letters and the string " D G _ D A T A 1", naming the disk group. You can check the list of available partitions using ' cat /proc/parititons|less '.

Red Hat updates Enterprise Linux to version 5.2 . That means Oracle Enterprise Linux and CentOS are soon to follow... Main improvements are "extensive driver updates", enhanced capabilities including power usage, scalability, and manageability (whatever that means) and increased virtualization support (i.e. Xen). [update: improved PAE support]

During network installs of Linux, we have 2 HBA connected to the SAN. Each provides one or more LUNs for the OS and data, as we boot from SAN. However, we have had a lot of problems with the visibility of these LUNs to the (anaconda) installer of Oracle Enterprise Linux (OEL) 4 update 5 (4U5). The current version gets confused by disks being visible twice and can't distinguish between them. So it overwrites over the second path what it already did over the first. Our current work-around was to either shutdown the FC port of one of the HBA cards on the FC switch. Another is to hide LUNs on the second HBA during install and reactivate them afterwards. However, both these cause extra manual work and unnecessarily increase complexity of the whole setup. Another solution that we're testing is to only hide the boot LUN on the second HBA, not the whole set of LUNs. Anaconda (wiki) seems to have an issue writing to disks that are visible over more than path, no reading from them. App...

If you, like me, love LVM for your Linux installations and LVM's ability to add disk space to partitions before they fill up without reboot or unmounting... then you'll be glad to know you can also use LVM if you have EMC PowerPath for your multi-path issues. In a bug fix from 2005, rhn.redhat.com | Red Hat Support , Red Hat lists that you can use Powerpath pseudo-devices with LVM. That way, LVM is unaware of a path failure and will get the fail-over path from PowerPath. Update: What you should do, though, is tell LVM to ignore local devices and only focus on emcpower devices. I distinguish between VMs and bare metal (BM) servers that also have local SAS disks. Here is what I've put in the lvm.conf file: # add EMCpower devices and sdb, reject all other SCSI disks and any CDs #BM#filter = [ "a|/dev/emcpower.*|","a|/dev/sdb|","r|/dev/sd.*|","r|/dev/cd.*|" ] # add any SCSI devices, reject all IDE disks and CDs #VM#filter...

Calivia has a nice little "howto" about doing failover tests of your FibreChannel HBA cards on Linux, to test multipathing software.

Last week I tried to install Oracle Enterprise Linux, aka Unbreakable Linux, on our new PowerEdge 2950 servers. Installation went without a hitch! So that was good news. A look in dmesg showed the two Qlogic QLE2460 HBAs and the on-board Broadcom dual 1 Gb NICs were recognized perfectly. However, SAN people reported no activity or visibilty on the SAN fabric whatsoever. I started to read up on Linux multipath IO (MPIO) and found some documentation there. I will have to go back to the data center and try these thing onsite, as remote access is not working yet. The good news is that MPIO is built-in in Linux kernels since early 2.6 (something like 2.6.5) so by now, these things should be standard. LVM2 includes all the necessary device-mapper packages and utils, so everything should be in place. I simply may need to activate the device mapper with dmsetup or multipathd . We'll see...