Oracle: Unbreak my Linux Heart

At work we are in the process of upgrading our VMware infrastructure to vSphere with new Vmax storage from EMC. One of the new features is VAAI , vStorage APIs for Array Integration. This abstracts certain aspects of storage to the SAN layer, where e.g. cloning can be a mere functional call that is then handled transparantly for the layer above. Yellow Bricks has a great post on VAAI sweetness collected from twitter and gathered or our convenience. Awesome!

Also found this great but slightly dated resource on Tuning Journaling File Systems . We had a crash of a VM recently with 3 million small files in a single directory. Tuning some mounts with noatime and dir_index helped a lot. Still tinkering with journaling options, though.

Linux uses 512-byte sectors. Hard disks used to too. The emergence of huge TB-sized disks has caused manufacturers to changed to 4096-byte sectors. This change can cause performance degradation on Linux as it's partitions may not be aligned with these bigger sectors. The additional overhead for the disk causes a performance drop. This raises the need for aligning your Linux partitions to the new sector size. However, many tools such as fdisk and parted are not yet doing this for you easily. IBM DevWorks has a great in-depth article on the topic: Linux on 4KB-sector disks: Practical advice

I knew I could use fdisk in Linux to create partitions for me from a script. This can come in handy during kickstart for (re)deployment of a server and you don't want to let anaconda handle partitions for you. Or you simply want to be absolutely sure things go as you want, instead of relying on software with unknown bugs or quirks. I knew you can use something like: fdisk /dev/sda to partition /dev/sda for you with one large partitions and give it type 8e (LVM). (The 'q' is so that those who simply copy this, don't immediately wipe some disk.) But I was looking for somewhat cleaner, more intuitive style. James Stephens pointed out the use of sfdisk from scripts . While I don't know if sfdisk is less reliable than fdisk or parted, or how to handle byte-aligning disks [ 1 ] [ 2 ] [ 3 ] from these scripts, it's a good place to start and I wanted to document it here.

Uptake of native Linux ZFS port hampered by license conflict . But other than that, ZFS may soon arrive on Linux. See also ZFS Fuse .

On the Dell PowerEdge mailing list, someone complained in details about the performance of the PERC6 RAID controller that dell uses by default in all its servers. The guy makes a compelling case with ~30-50 Mbps performance to local disks where a comparable Arcea card reaches ~300-400 Mbps! Anyway, long story short, he has detailed instructions and photos documenting his swap online.

While are many excellent tutorials about creating and using LVM on Linux, not may show you how you can remove disks from LVM Volume Groups (VG) and reclaim storage or how to remove a Logical Volume (LV) from your LVM set-up. Here is what I did: Use -t to TEST ANY LVM action first! We are going to release 1 TB from LVM. The Volume group was extended with 1 TB storage to serve as a cheap NFS/CIFS file server when setting up our data center. It is now deprecated and replaced by a NAS so it's no longer needed. 1) check LVM; note the four 256 GB LUNs [root@server ~]# pvscan -v Wiping cache of LVM-capable devices Wiping internal VG cache Walking through all physical volumes PV /dev/sdb1 VG vgdata lvm2 [50.00 GB / 0 free] PV /dev/sdc1 VG vgdata lvm2 [256.00 GB / 0 free] PV /dev/sdd1 VG vgdata lvm2 [256.00 GB / 0 free] PV /dev/sde1 VG vgdata lvm2 [256.00 GB / 0 free] PV /dev/sdf1 VG vgdata lvm2 [256.00 GB / 0 free] PV /dev/sdg ...

Red Hat Enterprise Linux 5.4 Bèta is now available. Most interesting improvements, apart from the usual bug fixes, performance benefits and better hardware support, are the introduction of KVM virtualization (in addition to regular Xen-based virtualization). New hardware support in virtual environments through SRIOV ( Single Root I/O for Virtualization ), utilities for management of FCoE cards over Internet, FIPS-140 security certification and a libvirt Perl interface for virtualization. Read the release notes for full details. Better yet is this announcement: " An important feature of any Red Hat Enterprise Linux update is that kernel and user APIs are unchanged, so that Red Hat Enterprise Linux 5 applications do not need to be rebuilt or re-certified. This situation extends to virtualized environments: with a fully integrated hypervisor, the application binary interface (ABI) consistency offered by Red Hat Enterprise Linux means that applications certified to run on Red Hat ...

We discovered a serious issue when rescanning the SCSI bus on (Oracle/Red Hat) Linux to discover newly added LUNs and I thought I'd mention it here. Our (old) Dell PowerEdge 2950 come with virtual media to allow mounting virtual floppy images (for driver disks, e.g.) and virtual CDs (great for ISOs). Dell adds these devices to the USB SCSI bus and therefore they show up as /dev/sda and /dev/sdb . I.e. before any regular (boot) media. While it makes sense, it creates a problem for us when adding new SAN LUNs to an existing Oracle database server, because we need to add some ASM disks. When doing the bus rescan echo "- - -" > /sys/class/scsi_host/host3/scan the virtual media get enumerated again as sda and sdb , which throws off existing device mapping. Our boot LUN which was sdb is now sdd and that used to be an ASM disk. After initialization with ASM, my boot LUN gets wiped and things get ugly. No root device, read-only access and upon reboot grub fails because boo...

Dell has released OMSA 6.0.2 for monitoring and remotely managing Dell PowerEdge servers that run Linux. Plugins for Nagios, MOM and others exist, so you can tie your Linux servers to most corporate tools.

An interesting read from Slashdot mentioned a company YippieMove who switched from VMware to FreeBSD Jails when their infrastructure ran into so many problems doing I/O that something had to change. Interestingly, BSD Jails resembles Solaris' Zones and Linux vservers . Bottom-line is that, as the comments mention very well, you have to “know your workload before picking the technology.” Also, which hardware generation you run VMware on has a big impact. Newer hardware has a significantly better support for virtualization when compared to older generation hardware. We've had similar I/O problems with NFS shares exported to various bare metals (BMs) and VMs and we choose to move to an ECM NAS instead (based on the number and the volumes of the shares). I doubt installing vSphere 4 on our old PowerEdge 2950 would make any difference as the old hardware is simply already stretched to its limits.

Never seen so many IPMI messages flying around on the Dell Linux mailing list. It's an open standard to monitor information of basic health parameters of a server. Here is a very good from by someone from Dell that sums up the usage and support of IPMI under Linux on Dell PowerEdge servers. Basically, you only need ipmitool to issue power on/off the servers. If you want to do more, you need to install the IPMI packages (yum install OpenIPMI)

We are trying to create some common ground between Solaris and Linux environments. For example, auto mounting scripts of NFS file systems. One issue was that Sun uses the master map file called auto_master whereas Linux defaults to auto.master . Thankfully, you can tweak /etc/sysconfig/autofs and use MASTER_MAP_NAME to tell Linux to also use auto_master . Problem solved! Update: dont' forget to open /etc/nsswitch.conf and edit the line for autofs . I had to delete nisplus .

heise online : "A bug report posted in the bug tracker for the next version of Ubuntu 9.04 (Jaunty Jackalope) describes a massive data loss problem when using Ext4, the future standard file system for Linux, available as an option when installing Ubuntu 9.04. The report describes a crash occurring shortly after the KDE 4 desktop files had been loaded, resulting in the loss of all of the data that had been created, including many KDE configuration files." This may not be as relevant for a server setup but it could outline a major vulnaribility in ext4 for now. Read the article for details...

A Sun Solaris friend of mine looked at me in disgust when he saw my /etc/fstab file to mount NFS mounts, CDs and more. I was seriously hurt so when he explained the shiny happy possibilities of autofs on Solaris (and Linux), I was immediately convinced: Configuration of the Automounter Autofs ... and all was well!

With Red Hat and Oracle Linux supporting multipathed installation from 5 update 2 on, I thought they should have changed the Kickstart Options as well, so that anaconda can be given the right parameters. And I was right. ignoredisks has been added to let the installer ignore SAN disks as well as for cluster setups. multipath has been added to handle dual HBAs cases where the same disk will be seen over two different paths. Update: except ignoredisks , there is really not much you need to do but boot the Linux kernel with the option ' mpath '. The multipath option lets you set/configure the multipath.conf configuration, I believe. Not used yet. In my case, I use this line to boot Linux and start a kickstart installation: linux mpath text nousbstorage ksdevice=eth0 ks=http://server/mpathks.cfg That's it. The first activates multipath-aware option in anaconda. The second uses text mode. The third parameter skips any USB devices in my PowerEdge 2950 (virtual floppy, d...

Computerworld has a great read about Samba 4 and how it'll improve AD integration of Linux and MS AD. Although it's been possible for a long time, bundled SAMBA version in distos can run behind the main development line and become a pain when trying to read documentation and configure older versions. SAMBA4 will run most things out of the box: Bartlett admitted Samba has a "nasty" reputation for being "impossible to configure" and believes Samba 4 should "just work" without administrators needing to read through documentation first. "For example, in Samba 4 we generate the DNS configuration and have an optional OpenLDAP backend and we have configured this as we know how Samba needs this setup." Over the past year Samba 4 has added multi-master replication leveraging OpenLDAP, making Samba no longer a single-server implementation. "Our users have also demonstrated how smart card logins work. I barely had to do anything in Samba 4 to ...

Another note on using PowerPath in Linux... When using 2 Linux servers and PowerPath with a bunch of shared LUNs in order to build a cluster, e.g. Oracle RAC, you have to ensure that on both servers the LUNs are designated as the same /dev/emcpower* devices. Linux is a bit flaky in device enumeration and often, the order of devices gets mixed up. PowerPath can easily fix this . Use powermt save file=devmap.pp on one host of the cluster and use powermt unmanage dev=emcpower{a,b} and powermt load file=devmap.pp on the other, to make sure they both have the same mapping. Then do a ' powermt save ' to make the configuration persistent. Also, when using LVM in conjunction with PowerPath, you don't want regular SCSI devices (local disks or perhaps iSCSI) to intervene. You can tell LVM to ignore certain devices completely when it comes to LVM. In my case I've included the filters below to make sure Linux VMs and BMs (bare metals) get the proper config. Put this in /etc/lvm...

While on the topic of Support cycles and End-of-Life (EOL) notices... EMC has given PowerPath 5.0.0 for Linux an EOL from May 31st, 2009. So I'm checking if we'll upgrade to PowerPath 5.1.x or switch over to a fresh Linux release and use dm-multipath (aka dm-mpio, native Linux multi-pathing. blog'o thnet has a nice Comparison: EMC PowerPath vs. GNU/Linux dm-multipath for your convenience.

Network World : "Oracle has contributed data-integrity protection code, partly developed with the hardware vendor Emulex, to the Linux kernel, the vendors announced Tuesday. The code helps maintain 'comprehensive data integrity' as information 'moves from application to database, and from [the] Linux operating system to disk storage,' according to a statement. It also lowers the possibility that erroneous data will get written to disk. The companies' effort is meant to help data-center administrators track and address corrupted data quickly, lowering costs and downtime"