Oracle: Unbreak my Linux Heart

On the Dell PowerEdge mailing list, someone complained in details about the performance of the PERC6 RAID controller that dell uses by default in all its servers. The guy makes a compelling case with ~30-50 Mbps performance to local disks where a comparable Arcea card reaches ~300-400 Mbps! Anyway, long story short, he has detailed instructions and photos documenting his swap online.

We discovered a serious issue when rescanning the SCSI bus on (Oracle/Red Hat) Linux to discover newly added LUNs and I thought I'd mention it here. Our (old) Dell PowerEdge 2950 come with virtual media to allow mounting virtual floppy images (for driver disks, e.g.) and virtual CDs (great for ISOs). Dell adds these devices to the USB SCSI bus and therefore they show up as /dev/sda and /dev/sdb . I.e. before any regular (boot) media. While it makes sense, it creates a problem for us when adding new SAN LUNs to an existing Oracle database server, because we need to add some ASM disks. When doing the bus rescan echo "- - -" > /sys/class/scsi_host/host3/scan the virtual media get enumerated again as sda and sdb , which throws off existing device mapping. Our boot LUN which was sdb is now sdd and that used to be an ASM disk. After initialization with ASM, my boot LUN gets wiped and things get ugly. No root device, read-only access and upon reboot grub fails because boo...

We've found some really strange I/O errors (Qlogic qle2460, firmware 1.24) using LUNs on our DMX-3 SAN. One HBA was faulty so we replaced it. However upon restoring the OS and reinstalling it, more problems appeared. The new HBA would not boot at all using the existing disks. So we disabled it in the BIOS and booted from the other (original) HBA. Both HBAs have the same firmware, same settings. Upon booting anything involving the disks (we boot from SAN and have data disks there as well) is extremely sluggish. Letting the server do its thing, I got a ton of I/O errors first during disk discovery, then again during mounting of file systems. ERROR: ddf1: reading /dev/sdb[Input/output error] ERROR: hpt37x: reading /dev/sdb[Input/output error] ERROR: pdc: reading /dev/sdb[Input/output error] ERROR: pdc: reading /dev/sdb[Input/output error] ERROR: pdc: reading /dev/sdb[Input/output error] ERROR: pdc: reading /dev/sdb[Input/output error] ERROR: pdc: reading /dev/sdb[Input/output error...