Oracle: Unbreak my Linux Heart

Dell TechCenter has released a LiveDVD image with OMSA to update firmware and BIOS on Dell hardware using a CentOS Linux distro.

The kind people of the German university of Göttingen have mirrored the Dell OMSA hardware and firmware repo for us. This way, we have a European mirror available that is also much faster than Dell's native server.

Dell has made the source code to iDRAC6 available online, at least somewhat...You can find the code on  Dell Support . That is code only, no build environment or firmware image creation scripts as required by GPLv3. New versions of iDRAC can be found on  http://ftp.us.dell.com/esm/ . Many thanks to yo61.com for finding it and publishing it on the Dell Linux mailing list . Update: Dell gave a better working Dell Support link. Update: All official Dell DRAC manuals

Dell Community notified us that you can now install Ubuntu Server Edition on its servers. While they won't factory-install it and Dell support itself will not support it, Dell has worked with Canonical for the past 2 years to support it. Canonical will be offering support for Ubuntu on Dell servers. To make support easier, Dell has built a native .deb package of OMSA 6.3 on Ubuntu 9.10.

Robin at yo61.com found a great and handy way to discover the MAC addresses of the embedded NICs of your Dell (PowerEdge) servers. Instead of going through DRAC, logging in, starting Java, etc. you can also get the same using the RAC command line interface: racadm racdump. Read the article on yo61.com for complete details. Excellent! If only I could find a similar command for my 500 PE2950 servers with DRAC5...

There are at least 2 version of the new iDRAC, Dell's remote server management and monitoring vehicle. Documentation from Dell lists the differences between the two and the older BMC interface.

I've been diving a bit into the RAM configuration of our new R710 Dell servers. They have new memory configuration settings. "Memory Optimized" mode can yield a 50% performance improvement at no cost other than to place DIMMs in appropriate slots. Dell has a white paper detailing the options a bit more: Optimizing DDR3 Memory Settings in New 11th-Generation Dell PowerEdge Servers . It also lists trade-offs of choosing one configuration over the other or when to go for a balanced intermediate.

In compliance with GPL license used by Dell to create DRAC, Dell finally made the source code of their web-based remote server management interface available on their website . It used to be available only after explicit requests. DRAC is built on BusyBox Linux , as reported here earlier.

I'm in the middle of creating a deployment image of OEL 5u3 (based on RHEL5u3) for new Dell PowerEdge R710 servers. After creating a generic and untested setup for the BIOS, my VMware guy came and informed me that vSphere 4 recommendations list to disable memory interleaving. Generally, letting NUMA do its thing yields better performance. So I'm now checking whether or not use interleaving for my 32 GB servers or to use NUMA out-of-the-box on Linux 2.6.18 as shipped with OEL5u3. I could switch to " Optimizer Mode " for the 32 GB but I'll need to have all servers physically checked and DIMMs reseated. So I opted for Interleaving to hope and try to gain some performance increase. Now I find out NUMA could be better and it is supported from RHEL5, which each update improving on it. I'll have to run more tests to be sure but for now I'm glad I discovered I could enable NUMA on OEL4u5 or higher as well, using a kernel option at boot time. Be sure to check nu...

Once again I am running into the weird issue where at install time my NICs are enumerated one way and after a reboot another. This first happened to me in 2007, while installing Oracle Enterprise Linux (OEL) 4u5 on a Dell PowerEdge 2950. The on-board NICs were numbered 'gb1' and 'gb2' but enumerated 'eth1' and 'eth0', respectively. Strangely, the problem went away silently and just now reappeared in OEL 5u3. Today I am using new R11 servers from Dell, the R710 with two dual on-board Broadcom NICs as well the quad-NIC add-on from Intel. At boot, my 'eth0' seems to be what I expect it but after a reboot it becomes 'eth4'. Found an updated white paper from Dell, listing a work-around: nic-enum-whitepaper-v4.pdf .

Where I had great difficulties trying to get OMSA to install let alone work in Oracle's Enterprise Linux 4 (u5 and above), I am happy to report that I got it working out-of-the-box after an install of my custom 5u3 image, following the instructions on the OMSA Repository Wiki . A simple http proxy setting, the wget boot strap line to configure yum and the dependencies and I was in business! Excellent work, Dell!

On the Dell PowerEdge mailing list, someone complained in details about the performance of the PERC6 RAID controller that dell uses by default in all its servers. The guy makes a compelling case with ~30-50 Mbps performance to local disks where a comparable Arcea card reaches ~300-400 Mbps! Anyway, long story short, he has detailed instructions and photos documenting his swap online.

We discovered a serious issue when rescanning the SCSI bus on (Oracle/Red Hat) Linux to discover newly added LUNs and I thought I'd mention it here. Our (old) Dell PowerEdge 2950 come with virtual media to allow mounting virtual floppy images (for driver disks, e.g.) and virtual CDs (great for ISOs). Dell adds these devices to the USB SCSI bus and therefore they show up as /dev/sda and /dev/sdb . I.e. before any regular (boot) media. While it makes sense, it creates a problem for us when adding new SAN LUNs to an existing Oracle database server, because we need to add some ASM disks. When doing the bus rescan echo "- - -" > /sys/class/scsi_host/host3/scan the virtual media get enumerated again as sda and sdb , which throws off existing device mapping. Our boot LUN which was sdb is now sdd and that used to be an ASM disk. After initialization with ASM, my boot LUN gets wiped and things get ugly. No root device, read-only access and upon reboot grub fails because boo...

At work I run a lot of Linux servers with Oracle Linux (OEL, Red Hat-derived) and our Dell PowerEdge 2950 servers have no problems with them. At first, OEL 4u5 had a terribly confusing issue with device enumeration where the first on-board NIC (labeled 1) would end up as eth1 and the other (labeled 2) would become eth0. That was fixed later on and I've reported on it here, 2 years ago. Lately, I've been playing with other Linux distos and seeing how well the (old) Dell servers hold up new distributions. I always used netimages where I install the OS over the network. For one, this greatly speeds up the ISO download and it also ensures I have an updated system after installation ready for testing. While Red Hat and SUSE-based distos are not a problem, Debian-based versions can be. This mainly has to do with the strict philosophy of their maintainers that non-free drivers and firmware should not be included in the image. In my case, the lack of built-in Broadcom (bnx2) drivers ca...

Never seen so many IPMI messages flying around on the Dell Linux mailing list. It's an open standard to monitor information of basic health parameters of a server. Here is a very good from by someone from Dell that sums up the usage and support of IPMI under Linux on Dell PowerEdge servers. Basically, you only need ipmitool to issue power on/off the servers. If you want to do more, you need to install the IPMI packages (yum install OpenIPMI)

Wouldn't it be great if you could use the LCD on a Dell PowerEdge to display the names of the VMs running on it? You may never to know where a VM actually, but if you do, you usually need it badly. Well Nickapedia wrote the script and the HOWTO for you: Dell Server LCD Update Script . It uses Dell OMSA on ESX and IPMI to get access to the VM names and then outputs these on the LCD. Brilliant! See video on YouTube ! BTW: won't work on ESXi because OMSA is not supported there.

HPC Community - Systems Management : "The Dell OpenManage Nagios Plug-in is a freely available, open source plug-in for the Nagios network monitor. By harnessing Dell OpenManage (OMSA), the plug-in provides the ability to monitor the overall health of supported Dell hardware models, all via the Nagios interface. Additionally, the Nagios notification mechanism may be used to provide alerts regarding system state based upon the information reported by the Dell OpenManage Nagios Plug-in."

Someone on the Dell mailing list was kind enough to point that although OMSA (OpenManage) on Ubuntu is not supported by Dell, the SARA institute in the Netherlands has a couple of OMSA Debian packages for Ubuntu ready for use. Check their site !

I've been seeing a weird error on my server console lately. We're using Dell PowerEdge 2950 and 2950 III running Oracle's Enterprise Linux 4U5, source code from RHEL AS 4u5, with no patches applied. Straight "base distribution" (I know, I know, bad idea but bare with me). I reported it to Dell's PE mailing list, but didn't get any conclusive answers. So I'm listing it here, for reference. Sep 1 15:36:26 kernel: hda: irq timeout: status=0xd0 { Busy } Sep 1 15:36:26 kernel: hda: irq timeout: error=0x00 Sep 1 15:36:26 kernel: hda: ATAPI reset complete I've found various reports of similar errors on different hardware but all seem to revolve around Red Hat's 4 release (OEL, RHEL, CentOS, Fedora) with various updates. See [ 1 ] [ 2 ] Some sources say it's a DMA-related issue. Others suggest to turn off ACPI using kernel boot parameters acpi=off and another says it's simply the HAL daemon probing your devices. You can configure the hal...

Very interesting tidbit on the PowerEdge mailing list today... Dell seems to have built their DRAC remote server management interface around BusyBox Linux . It's a minimal, low profile Linux platform, often used in embedded situations or for system rescues, maintenance, upgrades and so on. BusyBox is GPL, so accordingly DRAC must be GPL and therefore the source code must be available somewhere. Well, it is ! Just file a service request and mention " SKU 420-3178 ". And this makes for low level custom builds for your specific organization, additional drivers, etc. Update: The SKU number for the DRAC source code is also mentioned in the Dell PowerEdge Diagnostics readme.txt file. Update2: Someone has thoroughly investigated the SKU with Dell Australia and the ruling is that the SKU does not exist (anymore). It can't be found. Update3: Dell releases DRAC5 source code on website.