Oracle: Unbreak my Linux Heart

On the Dell PowerEdge mailing list, someone complained in details about the performance of the PERC6 RAID controller that dell uses by default in all its servers. The guy makes a compelling case with ~30-50 Mbps performance to local disks where a comparable Arcea card reaches ~300-400 Mbps! Anyway, long story short, he has detailed instructions and photos documenting his swap online.

We discovered a serious issue when rescanning the SCSI bus on (Oracle/Red Hat) Linux to discover newly added LUNs and I thought I'd mention it here. Our (old) Dell PowerEdge 2950 come with virtual media to allow mounting virtual floppy images (for driver disks, e.g.) and virtual CDs (great for ISOs). Dell adds these devices to the USB SCSI bus and therefore they show up as /dev/sda and /dev/sdb . I.e. before any regular (boot) media. While it makes sense, it creates a problem for us when adding new SAN LUNs to an existing Oracle database server, because we need to add some ASM disks. When doing the bus rescan echo "- - -" > /sys/class/scsi_host/host3/scan the virtual media get enumerated again as sda and sdb , which throws off existing device mapping. Our boot LUN which was sdb is now sdd and that used to be an ASM disk. After initialization with ASM, my boot LUN gets wiped and things get ugly. No root device, read-only access and upon reboot grub fails because boo...

At work I run a lot of Linux servers with Oracle Linux (OEL, Red Hat-derived) and our Dell PowerEdge 2950 servers have no problems with them. At first, OEL 4u5 had a terribly confusing issue with device enumeration where the first on-board NIC (labeled 1) would end up as eth1 and the other (labeled 2) would become eth0. That was fixed later on and I've reported on it here, 2 years ago. Lately, I've been playing with other Linux distos and seeing how well the (old) Dell servers hold up new distributions. I always used netimages where I install the OS over the network. For one, this greatly speeds up the ISO download and it also ensures I have an updated system after installation ready for testing. While Red Hat and SUSE-based distos are not a problem, Debian-based versions can be. This mainly has to do with the strict philosophy of their maintainers that non-free drivers and firmware should not be included in the image. In my case, the lack of built-in Broadcom (bnx2) drivers ca...

Testing my new kickstart file to install Linux on my old Poweredge 2950 dell servers is fine, but multipath presents me with a new problem. We boot from SAN, even though the servers have local disks available. Eventually we will move towards Blades, so the experience is useful. But each server has a different number of LUNs available to it. The boot LUN is always LUN id 0. Without multipathd , this gave me /dev/sdb as the boot device. With multipathd , this gives me /dev/mapper/mpathX , where X is the number of LUNs I have. You may see the problem: The kickstart file must list the boot drive order and designate a place to put the MBR. However, I can't think of a way to tell kickstart to use the last mpath device... I wish there'd be a way to make /dev/mapper/mpath0 the boot LUN... Update: Well, it seems to be a limitation of multipath support in anaconda. I've booted the OEL 5.3 DVD ISO on 3 different bare metals and each server has a different number of LUNs associated w...

With Red Hat and Oracle Linux supporting multipathed installation from 5 update 2 on, I thought they should have changed the Kickstart Options as well, so that anaconda can be given the right parameters. And I was right. ignoredisks has been added to let the installer ignore SAN disks as well as for cluster setups. multipath has been added to handle dual HBAs cases where the same disk will be seen over two different paths. Update: except ignoredisks , there is really not much you need to do but boot the Linux kernel with the option ' mpath '. The multipath option lets you set/configure the multipath.conf configuration, I believe. Not used yet. In my case, I use this line to boot Linux and start a kickstart installation: linux mpath text nousbstorage ksdevice=eth0 ks=http://server/mpathks.cfg That's it. The first activates multipath-aware option in anaconda. The second uses text mode. The third parameter skips any USB devices in my PowerEdge 2950 (virtual floppy, d...

While on the topic of Support cycles and End-of-Life (EOL) notices... EMC has given PowerPath 5.0.0 for Linux an EOL from May 31st, 2009. So I'm checking if we'll upgrade to PowerPath 5.1.x or switch over to a fresh Linux release and use dm-multipath (aka dm-mpio, native Linux multi-pathing. blog'o thnet has a nice Comparison: EMC PowerPath vs. GNU/Linux dm-multipath for your convenience.

Network World : "Oracle has contributed data-integrity protection code, partly developed with the hardware vendor Emulex, to the Linux kernel, the vendors announced Tuesday. The code helps maintain 'comprehensive data integrity' as information 'moves from application to database, and from [the] Linux operating system to disk storage,' according to a statement. It also lowers the possibility that erroneous data will get written to disk. The companies' effort is meant to help data-center administrators track and address corrupted data quickly, lowering costs and downtime"

Just received notice that Red Hat Enterprise Linux 5.3 Beta has been out since October 28th... Sjeez! Why didn't I know that sooner...? Anyway the bèta will last until January 9th, 2009, so when I get from holidays, I'll have my work cut of out for me. RHEL 5U3 brings many, many virtualization (Xen) improvements. A new kernel (kernel-2.6.18-120.el5), rebased versions of monitoring tools (lm-sensors, net-snmp, openIPMI, openldap), new versions of yum and rpm, updated Kerberos subsystem, newer SAMBA (3.0.32) with support for Vista and Windows Server 2008, a ton kernel fixes and features, new hardware drivers (bcm43xx, bnx2/bnx2x, cpufreq, e1000/e1000e, mptscsi, qla24xx, etc.) and some interesting Technology Previews! dm-multipath install, which should allow anaconda to install Linux on systems with 2 or more HBAs. Support for ext4 file systems. CIFS (aka SAMBA) enhancements in the form of kernel DFS support and Kerberos security. Check the complete Release Notes for a full upda...

I have been getting some weird errors in some Linux VMs running on VMware ESX 3.0.x (and perhaps ESX 3.5, as we're migrating to that). The symptoms are always that some weird errors occur for users or some maintenance task, and when you check the VM's consoles, you discover that the file system is suddenly read-only. My dear friend MrVanes discovered there is a known issue with the VMware LSI driver mptscsi in VMware guests running Debian . "...one could get SCSI timeouts when there is massive workload on the host system. Some kernel versions these will get the file systems remounted read-only, which probably makes sense for real hardware, but doesn't make sense for emulated hardware. Instead it should just wait a bit longer." A savvy user called TuxyTurvy has thoroughly investigated this issue. He was using Red Hat RHEL 4 and 5 on Dell 1850 servers with a Clarion AX150i el-cheapo SAN. When I/O load got high, running under VMware, due to storage contention, SCSI t...

I got a mail today from another group working with Linux servers on Dell PE2950's. They have HP EVA boxes as their SAN, instead of our EMC DMX-3/4, and wanted to know how we got multi-pathing to work in Oracle's Enterprise Linux 4, update 5. They had heard that it was terrible and were thinking of using RHEL/OEL 5u2 instead. I told them, we were using PowerPath for this reason, and because EMC would only support us iff we used PowerPath. dm-multipath was not supported. Hmmm. Fair enough... While dm-multipathing is available in OEL 4u5, you have to go through some tricks to get it to support your root volume over dm-multipath. TuxyTurvy has instructions and people report it works. The trick he uses is to make a custom initrd image. Another solution may be to use HP StorageWorks for Linux or HP's own dm-multipath . Update: We going a different route... We'll install ESX 3.5 on the Dell servers and install Linux in VMs only, virtualizing the whole setup. ESX is compatibl...

I have up2date setup for my servers to update most packages but ignore kernel updates as well as oracleasm -related updates. Kernel updates break things and must be done manually, when needed. However, I prefer using yum over up2date. But until now I did not find an easy way to tell yum to ignore updates on some packages. An old Duke mailing list had an improved man yum text that eluded to add multiple exclude=package* statements to the repos in /etc/yum.repos.d/ . Thanks Duke! Here is my example yum repo configuration with the excluded packages: [el4_patch]name=Oracle Enterprise Linux - $releasever - $basearch - patchbaseurl=http://uln.example.com/yum/EL4/$releasever/patch/$basearch/exclude=kernel*exclude=oracleasm*gpgcheck=1enabled=1 Update: something is still not right for it just upgraded the kernel anyway. Update2: When I moved the 'exclude' lines before the 'enabled' line, the yum update process does indeed ignore kernel updates, as expected. Before, I had the...

As a side-effect of the Linux boot process, I suddenly realized that our boot process is not fault-tolerant! We have 100+ servers that boot from SAN using two Qlogic 2460 HBAs. We installed EMC PowerPath 5.0.0 on OEL 4u5 to get multipathing and automatic fail-over in case a path fails. The OS is pretty well off in case of hardware faults. However, the boot process is not! Not even close! In stage 1, the boot loader reads the MBR, loads stage 1.5 so it can read the /boot ext2 partition where the kernel and initrd image are located. In our case, this is /dev/sdb1. But what if my HBA dies and /dev/sdb1 doesn't exist? It may be smart enough to try a device using the other path, but probably not. Also, since the kernel hasn't loaded yet, PowerPath does not exist and there is no multipath awareness to save the day... So once systems are installed and the number of disk partitions is stable, I can enhance GRUB with boot fallback systems , so it will try /dev/sdb1 and it that fails swi...

We've found some really strange I/O errors (Qlogic qle2460, firmware 1.24) using LUNs on our DMX-3 SAN. One HBA was faulty so we replaced it. However upon restoring the OS and reinstalling it, more problems appeared. The new HBA would not boot at all using the existing disks. So we disabled it in the BIOS and booted from the other (original) HBA. Both HBAs have the same firmware, same settings. Upon booting anything involving the disks (we boot from SAN and have data disks there as well) is extremely sluggish. Letting the server do its thing, I got a ton of I/O errors first during disk discovery, then again during mounting of file systems. ERROR: ddf1: reading /dev/sdb[Input/output error] ERROR: hpt37x: reading /dev/sdb[Input/output error] ERROR: pdc: reading /dev/sdb[Input/output error] ERROR: pdc: reading /dev/sdb[Input/output error] ERROR: pdc: reading /dev/sdb[Input/output error] ERROR: pdc: reading /dev/sdb[Input/output error] ERROR: pdc: reading /dev/sdb[Input/output error...

While we ourselves did not have any performance issues of our Dell Poweredge 2950 using 2 Qlogic qle2460 HBAs to connect to an EMC DMX-3 SAN over Cisco FC switches, some of our colleagues at another subsidiary did have problems: only 50MB/s using the bonnie disk benchmark where we got 240MB/s... Clearly something was different. Investigations showed they were using LVM on the LUNs and created partitions within LVM. We did not use LVM but gave bare LUNs to OracleASM to manage and control. This led to believe that misalignment of the partition to physical disk cylinder may have caused additional overhead. Because LVM created an MBR on the disk's first set of cylinders, the first partition was created starting at cylinder 63. As this is not a power of 2, the disk subsystem experiences overhead as it translates the numbers back and forth. Research showed that both Oracle and EMC's best-practices both recommend to align your partitions on power of two cylinders! There is a best-pra...

During network installs of Linux, we have 2 HBA connected to the SAN. Each provides one or more LUNs for the OS and data, as we boot from SAN. However, we have had a lot of problems with the visibility of these LUNs to the (anaconda) installer of Oracle Enterprise Linux (OEL) 4 update 5 (4U5). The current version gets confused by disks being visible twice and can't distinguish between them. So it overwrites over the second path what it already did over the first. Our current work-around was to either shutdown the FC port of one of the HBA cards on the FC switch. Another is to hide LUNs on the second HBA during install and reactivate them afterwards. However, both these cause extra manual work and unnecessarily increase complexity of the whole setup. Another solution that we're testing is to only hide the boot LUN on the second HBA, not the whole set of LUNs. Anaconda (wiki) seems to have an issue writing to disks that are visible over more than path, no reading from them. App...

Our Dell PowerEdges were also giving us a headache with their built-in USB devices and virtual devices. While they are useful for maintenance and patching, they mess up the Linux kernel's device enumeration because they come first in that list, no matter where you put your local disk or HBA card in your boot order. The Dell Virtual Flash and Virtual Floppy always become /dev/sda, /dev/sdb , respectivally. The local PERC disk then becomes /dev/sdc and our boot LUN on the SAN ends up at /dev/sdd . Turning the Dell devices on or off on BIOS then shuffles your devices around and suddenly your boot LUN will be sdb or sdc . No real danger, but /etc/fstab will no longer be valid and booting the system either gives you the maintenance shell or X1 will fail and you get a text login prompt (i.e. runlevel 3). Two solutions... One, you can use device labelling or udev naming to have the kernel trace the partitions it needs. However, a quick solutions is - during production - to turn off USB ...

We've been optimizing our kickstart script for unattended Oracle Linux provisioning and found out some interesting tidbits... When rolling out Linux to new Dell PowerEdge 2950 boxes that boot from (DMX-3) SAN, the anaconda installer has issues with pre-existing LVM volume groups and physical volume signatures. The zerombr option alone is not enough. Appearantly, some traces remain on that disk/LUN and anaconda fails with a cryptic error message. The solution is to use the kickstart pre-install section to clear your partitions! However, a simple parted won't do. Here is what we ended up putting in kickstart to effectively clear out /dev/sda and /dev/sdb , resp. our local disks (PERC) and out boot LUN (qlogic qle2460), from all partitions and LVM housekeeping stuff: %pre #forcefully remove all primary partitions from sda parted /dev/sda rm 1 parted /dev/sda rm 2 parted /dev/sda rm 3 parted /dev/sda rm 4 #forcefully remove all primary partitions from sdb parted /dev/sdb rm 1 pa...

We had a small problem today, where we wanted to add a few LUNs to a (backup) Linux server without rebooting. The problem was that the qlogic qle2460 did see the LUN but the OS wouldn't assign it a SCSI device ID (i.e. /dev/sdg or /dev/sdr or so). We needed to figure out a way to rescan the LUNs on the HBA and force a detection of LUNs in the OS layer. A new colleague solved it by using /sys, issuing a LIP and doing what is also pointed out in the old mail archive from Dell: New LUN available on Linux without reboot . Good to know. Thanks Dell and Koen!

Finally got the scripted network install working as expected. Here is how... Dell BIOS is at 1.3.7 Qlogic BIOS is at 1.24, BIOS enabled, boot LUN assigned. I disabled the 2nd HBA so the server only has one path to the SAN. Mount the Oracle Linux CD1 through Dell's DRAC interface and (re)boot the machine. Typ " linux ks=http://10.1.1.1/oel/ks.cfg ksdevice=eth0 " to start the installation and load your script. Your kickstart script should be located in the root of your installation sources, i.e. where you stored the contents of all the CDs. The last parameters is needed in case your server has multiple NICs, like ours, and avoid being asked which network card you want to use for installation. Since DHCP is enabled on that subnet, that's all that is needed. Now there is one more tricky part... upon loading the kickstart script, before the installation can start, you MUST disconnect the CD image that is still mounted in DRAC. If not, you get an endless loop and installati...

I was going crazy trying to install Linux on our servers and upon reboot they would do absolutely nothing or give me the GRUB limited shell or simply hang and complain about missing partitions. I suspected something wrong with the boot device: a LUN on an EMC DMX-3 Symmetrix SAN. OEL 4U5 boot CD recognizes the Qlogic qle2460 HBAs perfectly. Kudos there! However, for some reason I noticed my LUN set through both paths, despite having disabled on path (aka HBA) in its BIOS. No matter what I did, the installation CD kept recognizing all LUNs over all paths. Detailed investigation by our EMC guy showed that the Cisco SAN switches were talking FC-AL (Arbitrated loop) , for some reason. After shutting down the second port/path on the SAN switch, the installation CD only saw one set of devices and was no longer confused by all the LUNs that seems to be the same (they were!). Lesson learned: limit your variables during installation and only use one SAN path. Activate the second path during pos...