Oracle: Unbreak my Linux Heart

While are many excellent tutorials about creating and using LVM on Linux, not may show you how you can remove disks from LVM Volume Groups (VG) and reclaim storage or how to remove a Logical Volume (LV) from your LVM set-up. Here is what I did: Use -t to TEST ANY LVM action first! We are going to release 1 TB from LVM. The Volume group was extended with 1 TB storage to serve as a cheap NFS/CIFS file server when setting up our data center. It is now deprecated and replaced by a NAS so it's no longer needed. 1) check LVM; note the four 256 GB LUNs [root@server ~]# pvscan -v Wiping cache of LVM-capable devices Wiping internal VG cache Walking through all physical volumes PV /dev/sdb1 VG vgdata lvm2 [50.00 GB / 0 free] PV /dev/sdc1 VG vgdata lvm2 [256.00 GB / 0 free] PV /dev/sdd1 VG vgdata lvm2 [256.00 GB / 0 free] PV /dev/sde1 VG vgdata lvm2 [256.00 GB / 0 free] PV /dev/sdf1 VG vgdata lvm2 [256.00 GB / 0 free] PV /dev/sdg ...

I have been getting some weird errors in some Linux VMs running on VMware ESX 3.0.x (and perhaps ESX 3.5, as we're migrating to that). The symptoms are always that some weird errors occur for users or some maintenance task, and when you check the VM's consoles, you discover that the file system is suddenly read-only. My dear friend MrVanes discovered there is a known issue with the VMware LSI driver mptscsi in VMware guests running Debian . "...one could get SCSI timeouts when there is massive workload on the host system. Some kernel versions these will get the file systems remounted read-only, which probably makes sense for real hardware, but doesn't make sense for emulated hardware. Instead it should just wait a bit longer." A savvy user called TuxyTurvy has thoroughly investigated this issue. He was using Red Hat RHEL 4 and 5 on Dell 1850 servers with a Clarion AX150i el-cheapo SAN. When I/O load got high, running under VMware, due to storage contention, SCSI t...

After a boot disk crash, I wanted to make sure existing data disks were unaffected. We use Oracle ASM for all our bare metal database servers and in this case there wasn't a proper backup of the (development) database. ASM marks its disks as being part of an ASM disk group and checking the first block of a partition. This lets you verify that the data still exists. Upon reinstallation of the OS and ASM software, you can then reuse existing ASM disks and groups and restore the data. To verify the disks are still marked as ASM disk groups, check the first block of the partition: dd count=1 if=/dev/sdc1 | od -c sdc being our first data disk after local disks (sda) and the boot LUN (sdb). Note the text string on the third line in capital letters and the string " D G _ D A T A 1", naming the disk group. You can check the list of available partitions using ' cat /proc/parititons|less '.

I've become quite adapt with LVM but there is still one thing I haven't done yet: Reducing a logical volume . In other words, there are several LV inside your VG and you want to redistribute the disk space rather than add another disk or LUN and extend one. The latter is way easier, by the way. This is especially tricky if the LV you want to resize contains your root partition and is therefore mounted. Read the article above to find out how you can do it anyway. However, it will require a (re)boot and you'll need a rescue or installation disk that you can boot from.

A little petpeev of mine just got solved, I think, thanks to the Logical Volume Manager Cheatsheet by Martin Ankerl . One thing I was still doing in X11 (i.e. runlevel 5) was adding, creating and adjusting LVM disks. The GUI just makes things sooo much easier. Thanks to Martin's cheatsheet, though, I think I can easily add new LUNs to existing Volume Groups and extend my Logical Volumes... Cheers Martin!

I've reported on strange disk errors before and wasn't able to pin point the problem. Well, luckily I found what the problem but sadly I still don't know caused the problem to begin with... Every now and then I ran into a server that won't (re)boot and fail with multiple reports of missing disks, ext2 or ext3 superblocks to be missing and to try e2fsck -b 8193. Result: the server cannot load some LVM Volume Group (VG) and the Logical Volume (LV) can't be found. Basically it means you've lost a disk. You have errors similar to this: "Couldn't find all physical volumes for volume group vgdata." "Couldn't find device with uuid '56pgEk-0zLS-cKBc-z9vJ-kP65-DUBI-hwZPSu'." 'Volume group "vgdata" not found' LVM has several backup methods so not all is lost and chances are only the LVM meta data is corrupt, so the LVM manager doesn't know what to do with a physical disk and can't find its LVM signatures eithe...

While we ourselves did not have any performance issues of our Dell Poweredge 2950 using 2 Qlogic qle2460 HBAs to connect to an EMC DMX-3 SAN over Cisco FC switches, some of our colleagues at another subsidiary did have problems: only 50MB/s using the bonnie disk benchmark where we got 240MB/s... Clearly something was different. Investigations showed they were using LVM on the LUNs and created partitions within LVM. We did not use LVM but gave bare LUNs to OracleASM to manage and control. This led to believe that misalignment of the partition to physical disk cylinder may have caused additional overhead. Because LVM created an MBR on the disk's first set of cylinders, the first partition was created starting at cylinder 63. As this is not a power of 2, the disk subsystem experiences overhead as it translates the numbers back and forth. Research showed that both Oracle and EMC's best-practices both recommend to align your partitions on power of two cylinders! There is a best-pra...

We've been having a variety of problems with our LVM system partitioning and anaconda. Mind you, once it is configured and installed, the system runs fine. But during rollout, LVM has been a head breaker... Main issue is: dirty disks. Since we are booting from SAN, we have no control over the LUNs we get. Although we specify zerombr and clearpart --all in anaconda, LVM complains if there is anything on the disk. Especially if there are non-Linux partitions left on the disks. Altiris does it's boot "magic" using a DOS partition to create a known environment from which it launches stuff. This gave the anaconda installer headaches. Our work-around: don't do partitioning from anaconda, but use the pre-install phase (%pre). We force a clear of the local disk (sda) and our boot LUN (sdb) and clear partitions explicitly as well as remove any traces of LVM signatures. We remove all Logical Volumes (LV), Volume Groups (VG) and Physical Volumes (PV). While this is a lo...

We've been optimizing our kickstart script for unattended Oracle Linux provisioning and found out some interesting tidbits... When rolling out Linux to new Dell PowerEdge 2950 boxes that boot from (DMX-3) SAN, the anaconda installer has issues with pre-existing LVM volume groups and physical volume signatures. The zerombr option alone is not enough. Appearantly, some traces remain on that disk/LUN and anaconda fails with a cryptic error message. The solution is to use the kickstart pre-install section to clear your partitions! However, a simple parted won't do. Here is what we ended up putting in kickstart to effectively clear out /dev/sda and /dev/sdb , resp. our local disks (PERC) and out boot LUN (qlogic qle2460), from all partitions and LVM housekeeping stuff: %pre #forcefully remove all primary partitions from sda parted /dev/sda rm 1 parted /dev/sda rm 2 parted /dev/sda rm 3 parted /dev/sda rm 4 #forcefully remove all primary partitions from sdb parted /dev/sdb rm 1 pa...

We had a small problem today, where we wanted to add a few LUNs to a (backup) Linux server without rebooting. The problem was that the qlogic qle2460 did see the LUN but the OS wouldn't assign it a SCSI device ID (i.e. /dev/sdg or /dev/sdr or so). We needed to figure out a way to rescan the LUNs on the HBA and force a detection of LUNs in the OS layer. A new colleague solved it by using /sys, issuing a LIP and doing what is also pointed out in the old mail archive from Dell: New LUN available on Linux without reboot . Good to know. Thanks Dell and Koen!

I tried to image our Linux servers, taking a "golden image" from one of our bare metal servers. I wanted to have a quick restore available in case I seriously cripple a serever during testing. In earlier runs, the restore worked but the Linux image had become unusable. GRUB wouldn't produce anything beyond its first 4 letters, so I created another server and imaged that. Same thing... Turns out I am using a Dell Poweredge 2950 with a local disk in RAID1 (mirror). I only use the local disk as swap area, creating a big LVM Volume Group (VG) on it called VGinternal. The actual boot disk is a LUN on the SAN. It also uses LVM and I have a VG in the LUN called VGsystem. VGsystem holds the actual system partitions /, /var and /tmp. When Altiris images this, it uses raw mode to capture the LVM partitions. Upon deploying the image containing LVM disk info, you must tell the DS to use the Linux environment . Not the WinPE or DOS modes... Testing it later...

If you, like me, love LVM for your Linux installations and LVM's ability to add disk space to partitions before they fill up without reboot or unmounting... then you'll be glad to know you can also use LVM if you have EMC PowerPath for your multi-path issues. In a bug fix from 2005, rhn.redhat.com | Red Hat Support , Red Hat lists that you can use Powerpath pseudo-devices with LVM. That way, LVM is unaware of a path failure and will get the fail-over path from PowerPath. Update: What you should do, though, is tell LVM to ignore local devices and only focus on emcpower devices. I distinguish between VMs and bare metal (BM) servers that also have local SAS disks. Here is what I've put in the lvm.conf file: # add EMCpower devices and sdb, reject all other SCSI disks and any CDs #BM#filter = [ "a|/dev/emcpower.*|","a|/dev/sdb|","r|/dev/sd.*|","r|/dev/cd.*|" ] # add any SCSI devices, reject all IDE disks and CDs #VM#filter...

I had another issue with Dell's Poweredge 2950 and the combination on Oracle Enterprise Linux (OEL) 4 update 5 (4U5), qlogic 2460 single port HBAs and an EMC DMX SAN. I was trying to install a Linux server to boot from SAN... It seems be caused by Dell's BIOS and Linux device enumeration, about which I've reported earlier. Dell uses DRAC to service its servers remotely. Using DRAC, you can mount virtual devices: a floppy and/or a CD (image). With these, you can install driver updates, install an OS or apply updates to a machine as if you're sitting directly with them. Very handy, but... These virtual media get recognized as USB/SCSI devices, or so it seems, and this causes Linux to assign device names to them, just like ordinary (SCSI) disks. In my case, the Dell Virtual Floppy gets enumerated as /dev/sda , which bumps up the internal PERC controller and my QLogic HBA. This results in a boot drive letter change from /dev/sda to /dev/sdb or sdb to sdc . Therefore, you...