Oracle: Unbreak my Linux Heart

I knew I could use fdisk in Linux to create partitions for me from a script. This can come in handy during kickstart for (re)deployment of a server and you don't want to let anaconda handle partitions for you. Or you simply want to be absolutely sure things go as you want, instead of relying on software with unknown bugs or quirks. I knew you can use something like: fdisk /dev/sda to partition /dev/sda for you with one large partitions and give it type 8e (LVM). (The 'q' is so that those who simply copy this, don't immediately wipe some disk.) But I was looking for somewhat cleaner, more intuitive style. James Stephens pointed out the use of sfdisk from scripts . While I don't know if sfdisk is less reliable than fdisk or parted, or how to handle byte-aligning disks [ 1 ] [ 2 ] [ 3 ] from these scripts, it's a good place to start and I wanted to document it here.

While are many excellent tutorials about creating and using LVM on Linux, not may show you how you can remove disks from LVM Volume Groups (VG) and reclaim storage or how to remove a Logical Volume (LV) from your LVM set-up. Here is what I did: Use -t to TEST ANY LVM action first! We are going to release 1 TB from LVM. The Volume group was extended with 1 TB storage to serve as a cheap NFS/CIFS file server when setting up our data center. It is now deprecated and replaced by a NAS so it's no longer needed. 1) check LVM; note the four 256 GB LUNs [root@server ~]# pvscan -v Wiping cache of LVM-capable devices Wiping internal VG cache Walking through all physical volumes PV /dev/sdb1 VG vgdata lvm2 [50.00 GB / 0 free] PV /dev/sdc1 VG vgdata lvm2 [256.00 GB / 0 free] PV /dev/sdd1 VG vgdata lvm2 [256.00 GB / 0 free] PV /dev/sde1 VG vgdata lvm2 [256.00 GB / 0 free] PV /dev/sdf1 VG vgdata lvm2 [256.00 GB / 0 free] PV /dev/sdg ...

Oracle is pleased to announce the general availability of Enterprise Linux Release 5 Update 4 for x86 (32 bit) and x86_64 (64 Bit) architectures. This update includes the following kernel/driver changes: Bug Fixes added by Oracle * Check to see if hypervisor supports memory reservation change [orabug 7556514] * Add entropy support to igb [orabug 7607479] * Convert ENETUNREACH to ENOTCONN [orabug 7689332] * Add xen pv/bonding netconsole support [orabug 6993043] * Shrink zone patch [orabug 6086839] * Fix aacraid not to reset during kexec [orabug 8516042] * Fix failure of file creation from hpux client [orabug 7579314] Fixes/Additions from the upstream distribution provider * A new tunable parameter /proc/sys/vm/max_writeback_pages has been added to the kernel, allowing system administrators to change the maximum number of modified pages kupdate writes to disk per iteration each time it runs. It defaults to 1024 or 4MB ...

We discovered a serious issue when rescanning the SCSI bus on (Oracle/Red Hat) Linux to discover newly added LUNs and I thought I'd mention it here. Our (old) Dell PowerEdge 2950 come with virtual media to allow mounting virtual floppy images (for driver disks, e.g.) and virtual CDs (great for ISOs). Dell adds these devices to the USB SCSI bus and therefore they show up as /dev/sda and /dev/sdb . I.e. before any regular (boot) media. While it makes sense, it creates a problem for us when adding new SAN LUNs to an existing Oracle database server, because we need to add some ASM disks. When doing the bus rescan echo "- - -" > /sys/class/scsi_host/host3/scan the virtual media get enumerated again as sda and sdb , which throws off existing device mapping. Our boot LUN which was sdb is now sdd and that used to be an ASM disk. After initialization with ASM, my boot LUN gets wiped and things get ugly. No root device, read-only access and upon reboot grub fails because boo...

Testing my new kickstart file to install Linux on my old Poweredge 2950 dell servers is fine, but multipath presents me with a new problem. We boot from SAN, even though the servers have local disks available. Eventually we will move towards Blades, so the experience is useful. But each server has a different number of LUNs available to it. The boot LUN is always LUN id 0. Without multipathd , this gave me /dev/sdb as the boot device. With multipathd , this gives me /dev/mapper/mpathX , where X is the number of LUNs I have. You may see the problem: The kickstart file must list the boot drive order and designate a place to put the MBR. However, I can't think of a way to tell kickstart to use the last mpath device... I wish there'd be a way to make /dev/mapper/mpath0 the boot LUN... Update: Well, it seems to be a limitation of multipath support in anaconda. I've booted the OEL 5.3 DVD ISO on 3 different bare metals and each server has a different number of LUNs associated w...

Another note on using PowerPath in Linux... When using 2 Linux servers and PowerPath with a bunch of shared LUNs in order to build a cluster, e.g. Oracle RAC, you have to ensure that on both servers the LUNs are designated as the same /dev/emcpower* devices. Linux is a bit flaky in device enumeration and often, the order of devices gets mixed up. PowerPath can easily fix this . Use powermt save file=devmap.pp on one host of the cluster and use powermt unmanage dev=emcpower{a,b} and powermt load file=devmap.pp on the other, to make sure they both have the same mapping. Then do a ' powermt save ' to make the configuration persistent. Also, when using LVM in conjunction with PowerPath, you don't want regular SCSI devices (local disks or perhaps iSCSI) to intervene. You can tell LVM to ignore certain devices completely when it comes to LVM. In my case I've included the filters below to make sure Linux VMs and BMs (bare metals) get the proper config. Put this in /etc/lvm...

As a side-effect of the Linux boot process, I suddenly realized that our boot process is not fault-tolerant! We have 100+ servers that boot from SAN using two Qlogic 2460 HBAs. We installed EMC PowerPath 5.0.0 on OEL 4u5 to get multipathing and automatic fail-over in case a path fails. The OS is pretty well off in case of hardware faults. However, the boot process is not! Not even close! In stage 1, the boot loader reads the MBR, loads stage 1.5 so it can read the /boot ext2 partition where the kernel and initrd image are located. In our case, this is /dev/sdb1. But what if my HBA dies and /dev/sdb1 doesn't exist? It may be smart enough to try a device using the other path, but probably not. Also, since the kernel hasn't loaded yet, PowerPath does not exist and there is no multipath awareness to save the day... So once systems are installed and the number of disk partitions is stable, I can enhance GRUB with boot fallback systems , so it will try /dev/sdb1 and it that fails swi...

After a boot disk crash, I wanted to make sure existing data disks were unaffected. We use Oracle ASM for all our bare metal database servers and in this case there wasn't a proper backup of the (development) database. ASM marks its disks as being part of an ASM disk group and checking the first block of a partition. This lets you verify that the data still exists. Upon reinstallation of the OS and ASM software, you can then reuse existing ASM disks and groups and restore the data. To verify the disks are still marked as ASM disk groups, check the first block of the partition: dd count=1 if=/dev/sdc1 | od -c sdc being our first data disk after local disks (sda) and the boot LUN (sdb). Note the text string on the third line in capital letters and the string " D G _ D A T A 1", naming the disk group. You can check the list of available partitions using ' cat /proc/parititons|less '.

We've found some really strange I/O errors (Qlogic qle2460, firmware 1.24) using LUNs on our DMX-3 SAN. One HBA was faulty so we replaced it. However upon restoring the OS and reinstalling it, more problems appeared. The new HBA would not boot at all using the existing disks. So we disabled it in the BIOS and booted from the other (original) HBA. Both HBAs have the same firmware, same settings. Upon booting anything involving the disks (we boot from SAN and have data disks there as well) is extremely sluggish. Letting the server do its thing, I got a ton of I/O errors first during disk discovery, then again during mounting of file systems. ERROR: ddf1: reading /dev/sdb[Input/output error] ERROR: hpt37x: reading /dev/sdb[Input/output error] ERROR: pdc: reading /dev/sdb[Input/output error] ERROR: pdc: reading /dev/sdb[Input/output error] ERROR: pdc: reading /dev/sdb[Input/output error] ERROR: pdc: reading /dev/sdb[Input/output error] ERROR: pdc: reading /dev/sdb[Input/output error...

While backups are usually often considered when designing high-availability server parks, doing an actual restore is often overlooked. You simply don't want to pull a plug and rather wait for the actual disaster to happen. Having a full file system image of a server, while space and time consuming can be the fastest way to get a server back online after a failure. If you have a SAN available, it can clone LUNs for your transparently and you can swap them back in when a machine fails or burn down. However, not everyone has one. Using clonezilla , you can easily pull images off of Linux or even Windows servers and restore them (even using multicast!) in minutes. As it only copies used blocks, that 300 GB local disk can result in a 5 GB image. For a restore, Clonezilla restores the appropriate image based on MAC address.

I've become quite adapt with LVM but there is still one thing I haven't done yet: Reducing a logical volume . In other words, there are several LV inside your VG and you want to redistribute the disk space rather than add another disk or LUN and extend one. The latter is way easier, by the way. This is especially tricky if the LV you want to resize contains your root partition and is therefore mounted. Read the article above to find out how you can do it anyway. However, it will require a (re)boot and you'll need a rescue or installation disk that you can boot from.

A little petpeev of mine just got solved, I think, thanks to the Logical Volume Manager Cheatsheet by Martin Ankerl . One thing I was still doing in X11 (i.e. runlevel 5) was adding, creating and adjusting LVM disks. The GUI just makes things sooo much easier. Thanks to Martin's cheatsheet, though, I think I can easily add new LUNs to existing Volume Groups and extend my Logical Volumes... Cheers Martin!

I was just playing a test set-up for Nagios , to monitor some hosts, services and devices for me across the data center, when I ran into Splunk in a config file. Being naturally curious, I checked out Splunk... Splunk is basically your IT search repository. It has a search engine that collects everything you want to track from all your servers. Use Splunk to collect SNMP traps, mail queues, log from web servers, firewall or routers, transaction and database logs and more. It can index anything because it uses live tracking. What it doesn't know but occurs frequently, you can teach it. Best of all, use Splunk to trigger an SNMP alert for a host or service in Nagios, when say more than 10 HTTP 404 errors occur on your corporate website. Probably there has been an error that some other system may or not have tried to inform you about... Excellent! Some assembly required of course, as everyone has different logs, needs, wishes and demands...

While we ourselves did not have any performance issues of our Dell Poweredge 2950 using 2 Qlogic qle2460 HBAs to connect to an EMC DMX-3 SAN over Cisco FC switches, some of our colleagues at another subsidiary did have problems: only 50MB/s using the bonnie disk benchmark where we got 240MB/s... Clearly something was different. Investigations showed they were using LVM on the LUNs and created partitions within LVM. We did not use LVM but gave bare LUNs to OracleASM to manage and control. This led to believe that misalignment of the partition to physical disk cylinder may have caused additional overhead. Because LVM created an MBR on the disk's first set of cylinders, the first partition was created starting at cylinder 63. As this is not a power of 2, the disk subsystem experiences overhead as it translates the numbers back and forth. Research showed that both Oracle and EMC's best-practices both recommend to align your partitions on power of two cylinders! There is a best-pra...

We've been having a variety of problems with our LVM system partitioning and anaconda. Mind you, once it is configured and installed, the system runs fine. But during rollout, LVM has been a head breaker... Main issue is: dirty disks. Since we are booting from SAN, we have no control over the LUNs we get. Although we specify zerombr and clearpart --all in anaconda, LVM complains if there is anything on the disk. Especially if there are non-Linux partitions left on the disks. Altiris does it's boot "magic" using a DOS partition to create a known environment from which it launches stuff. This gave the anaconda installer headaches. Our work-around: don't do partitioning from anaconda, but use the pre-install phase (%pre). We force a clear of the local disk (sda) and our boot LUN (sdb) and clear partitions explicitly as well as remove any traces of LVM signatures. We remove all Logical Volumes (LV), Volume Groups (VG) and Physical Volumes (PV). While this is a lo...

During network installs of Linux, we have 2 HBA connected to the SAN. Each provides one or more LUNs for the OS and data, as we boot from SAN. However, we have had a lot of problems with the visibility of these LUNs to the (anaconda) installer of Oracle Enterprise Linux (OEL) 4 update 5 (4U5). The current version gets confused by disks being visible twice and can't distinguish between them. So it overwrites over the second path what it already did over the first. Our current work-around was to either shutdown the FC port of one of the HBA cards on the FC switch. Another is to hide LUNs on the second HBA during install and reactivate them afterwards. However, both these cause extra manual work and unnecessarily increase complexity of the whole setup. Another solution that we're testing is to only hide the boot LUN on the second HBA, not the whole set of LUNs. Anaconda (wiki) seems to have an issue writing to disks that are visible over more than path, no reading from them. App...

Our Dell PowerEdges were also giving us a headache with their built-in USB devices and virtual devices. While they are useful for maintenance and patching, they mess up the Linux kernel's device enumeration because they come first in that list, no matter where you put your local disk or HBA card in your boot order. The Dell Virtual Flash and Virtual Floppy always become /dev/sda, /dev/sdb , respectivally. The local PERC disk then becomes /dev/sdc and our boot LUN on the SAN ends up at /dev/sdd . Turning the Dell devices on or off on BIOS then shuffles your devices around and suddenly your boot LUN will be sdb or sdc . No real danger, but /etc/fstab will no longer be valid and booting the system either gives you the maintenance shell or X1 will fail and you get a text login prompt (i.e. runlevel 3). Two solutions... One, you can use device labelling or udev naming to have the kernel trace the partitions it needs. However, a quick solutions is - during production - to turn off USB ...

We've been optimizing our kickstart script for unattended Oracle Linux provisioning and found out some interesting tidbits... When rolling out Linux to new Dell PowerEdge 2950 boxes that boot from (DMX-3) SAN, the anaconda installer has issues with pre-existing LVM volume groups and physical volume signatures. The zerombr option alone is not enough. Appearantly, some traces remain on that disk/LUN and anaconda fails with a cryptic error message. The solution is to use the kickstart pre-install section to clear your partitions! However, a simple parted won't do. Here is what we ended up putting in kickstart to effectively clear out /dev/sda and /dev/sdb , resp. our local disks (PERC) and out boot LUN (qlogic qle2460), from all partitions and LVM housekeeping stuff: %pre #forcefully remove all primary partitions from sda parted /dev/sda rm 1 parted /dev/sda rm 2 parted /dev/sda rm 3 parted /dev/sda rm 4 #forcefully remove all primary partitions from sdb parted /dev/sdb rm 1 pa...

We had a small problem today, where we wanted to add a few LUNs to a (backup) Linux server without rebooting. The problem was that the qlogic qle2460 did see the LUN but the OS wouldn't assign it a SCSI device ID (i.e. /dev/sdg or /dev/sdr or so). We needed to figure out a way to rescan the LUNs on the HBA and force a detection of LUNs in the OS layer. A new colleague solved it by using /sys, issuing a LIP and doing what is also pointed out in the old mail archive from Dell: New LUN available on Linux without reboot . Good to know. Thanks Dell and Koen!

Finally got the scripted network install working as expected. Here is how... Dell BIOS is at 1.3.7 Qlogic BIOS is at 1.24, BIOS enabled, boot LUN assigned. I disabled the 2nd HBA so the server only has one path to the SAN. Mount the Oracle Linux CD1 through Dell's DRAC interface and (re)boot the machine. Typ " linux ks=http://10.1.1.1/oel/ks.cfg ksdevice=eth0 " to start the installation and load your script. Your kickstart script should be located in the root of your installation sources, i.e. where you stored the contents of all the CDs. The last parameters is needed in case your server has multiple NICs, like ours, and avoid being asked which network card you want to use for installation. Since DHCP is enabled on that subnet, that's all that is needed. Now there is one more tricky part... upon loading the kickstart script, before the installation can start, you MUST disconnect the CD image that is still mounted in DRAC. If not, you get an endless loop and installati...