Oracle: Unbreak my Linux Heart

A few days ago, Oracle followed suit by Red Hat and released OEL6. Here are the Oracle Linux 6 Release Notes . At first glance, most notable changes are default ext4 file system. Better performance monitoring and tracking using perf and new user space "top" tools, and it no longer uses up2date but instead relies totally on yum. Yummie! Last but not least: the Unbreakable Kernel kernel-uek-2.6.32-100.28.5.el6 is installed and activated by default! This shouldn't matter much, unless you run the latest state-of-the-art hardware and use SSDs a lot. But 3rd party software vendors may like you stepping away from the Red Hat default kernel-2.6.32-71.el6 kernel. So check first! OL6 is the first Linux distribution from Oracle where they no longer patch the Red Hat kernel. Instead, they provide the Red Hat kernel as-is as an option to be configured for use in /etc/grub/menu.lst. The default kernel in OL6 is the Unbreakable Enterprise Kernel that Oracle maintains itself. As far...

Oracle Linux 5.6 makes Unbreakable Enterprise Kernel the default kernel after installation unless you perform a custom installation. Note that Unbreakable Enterprise Kernel is available for the x86_64 platform only. The Unbreakable Enterprise Kernel is delivered via the package kernel-uek .

At OpenWorld, Oracle announced an Oracle optimized Linux kernel that is upto 75% faster than the bundled Red Hat kernel. But what's the catch? Well, it's simply a 2.6.32-based kernel , tweaked and tuned for Oracle software. Here's how to get it , if you want to. And LWN has a more detailed technical description of the changes from the mainstream kernel . Basically, the kernel optimizes communication: network (InfiniBand, 10Gb Ethernet), storage (SSD, Flash memory) and memory ( NUMA ). It optimizes drivers and reduces complexity in the kernel (i.e. remove unused drivers and so on). And considering the compatibility issue, here is Oracle's claim: "Third-party applications that run on RHEL 5 should run unchanged on Oracle Linux with the Unbreakable Enterprise Kernel , while delivering significant performance and reliability improvements for end users." Good reads are CIOupdate and PCWorld

A recent article on Linux security showed that an old backdoor (cve-2007-4573) , patched in kernel 2.6.22.7, has resurfaced . Using the leak, any user that has a local user account on a 64 bit server, can easily get root rights, using the compatibility layer. While I don't fully understand the workings, I do understand that all OEL4/OEL5 64 bit servers are potentially harmed as they have kernels 2.6.9 and 2.6.18, resp. So I logged a SR on Metalink to see if I have a big security issue in the data center. Update: CVE-2007-4573 has been renamed CVE-2010-3301 and RHEL states : This issue did not affect the versions of Linux kernel as shipped with Red Hat Enterprise Linux 3, 4, 5, and Red Hat Enterprise MRG, as they do not contain the upstream commit d4d67150 that introduced this flaw. Update2: the issue is also listed as CVE-2010-3081 , which is slightly different and Red Hat does track this one, as it affects RHEL5 . Update3: Ksplice has a test available to verify your syste...

I'm in the middle of creating a deployment image of OEL 5u3 (based on RHEL5u3) for new Dell PowerEdge R710 servers. After creating a generic and untested setup for the BIOS, my VMware guy came and informed me that vSphere 4 recommendations list to disable memory interleaving. Generally, letting NUMA do its thing yields better performance. So I'm now checking whether or not use interleaving for my 32 GB servers or to use NUMA out-of-the-box on Linux 2.6.18 as shipped with OEL5u3. I could switch to " Optimizer Mode " for the 32 GB but I'll need to have all servers physically checked and DIMMs reseated. So I opted for Interleaving to hope and try to gain some performance increase. Now I find out NUMA could be better and it is supported from RHEL5, which each update improving on it. I'll have to run more tests to be sure but for now I'm glad I discovered I could enable NUMA on OEL4u5 or higher as well, using a kernel option at boot time. Be sure to check nu...

Oracle is pleased to announce the general availability of Enterprise Linux Release 5 Update 4 for x86 (32 bit) and x86_64 (64 Bit) architectures. This update includes the following kernel/driver changes: Bug Fixes added by Oracle * Check to see if hypervisor supports memory reservation change [orabug 7556514] * Add entropy support to igb [orabug 7607479] * Convert ENETUNREACH to ENOTCONN [orabug 7689332] * Add xen pv/bonding netconsole support [orabug 6993043] * Shrink zone patch [orabug 6086839] * Fix aacraid not to reset during kexec [orabug 8516042] * Fix failure of file creation from hpux client [orabug 7579314] Fixes/Additions from the upstream distribution provider * A new tunable parameter /proc/sys/vm/max_writeback_pages has been added to the kernel, allowing system administrators to change the maximum number of modified pages kupdate writes to disk per iteration each time it runs. It defaults to 1024 or 4MB ...

Red Hat Enterprise Linux 5.4 Bèta is now available. Most interesting improvements, apart from the usual bug fixes, performance benefits and better hardware support, are the introduction of KVM virtualization (in addition to regular Xen-based virtualization). New hardware support in virtual environments through SRIOV ( Single Root I/O for Virtualization ), utilities for management of FCoE cards over Internet, FIPS-140 security certification and a libvirt Perl interface for virtualization. Read the release notes for full details. Better yet is this announcement: " An important feature of any Red Hat Enterprise Linux update is that kernel and user APIs are unchanged, so that Red Hat Enterprise Linux 5 applications do not need to be rebuilt or re-certified. This situation extends to virtualized environments: with a fully integrated hypervisor, the application binary interface (ABI) consistency offered by Red Hat Enterprise Linux means that applications certified to run on Red Hat ...

Oracle Linux Tests are designed to verify Linux kernel functionality and stability essential for the Oracle Database. The Oracle Linux Test (OLT) kit , which is distributed as a set of rpms, provides an automated mechanism to define, execute and analyze tests. The kit contains tools such as DBT2 and DBT3 workloads against Oracle single instance or clustered databases, as well as specialized workload simulators. OLT can be used for running tests on Oracle Enterprise Linux, RHEL and SLES distributions and a variety of topologies (storage attached network, network attached storage).

With Red Hat and Oracle Linux supporting multipathed installation from 5 update 2 on, I thought they should have changed the Kickstart Options as well, so that anaconda can be given the right parameters. And I was right. ignoredisks has been added to let the installer ignore SAN disks as well as for cluster setups. multipath has been added to handle dual HBAs cases where the same disk will be seen over two different paths. Update: except ignoredisks , there is really not much you need to do but boot the Linux kernel with the option ' mpath '. The multipath option lets you set/configure the multipath.conf configuration, I believe. Not used yet. In my case, I use this line to boot Linux and start a kickstart installation: linux mpath text nousbstorage ksdevice=eth0 ks=http://server/mpathks.cfg That's it. The first activates multipath-aware option in anaconda. The second uses text mode. The third parameter skips any USB devices in my PowerEdge 2950 (virtual floppy, d...

Network World : "Oracle has contributed data-integrity protection code, partly developed with the hardware vendor Emulex, to the Linux kernel, the vendors announced Tuesday. The code helps maintain 'comprehensive data integrity' as information 'moves from application to database, and from [the] Linux operating system to disk storage,' according to a statement. It also lowers the possibility that erroneous data will get written to disk. The companies' effort is meant to help data-center administrators track and address corrupted data quickly, lowering costs and downtime"

Just received notice that Red Hat Enterprise Linux 5.3 Beta has been out since October 28th... Sjeez! Why didn't I know that sooner...? Anyway the bèta will last until January 9th, 2009, so when I get from holidays, I'll have my work cut of out for me. RHEL 5U3 brings many, many virtualization (Xen) improvements. A new kernel (kernel-2.6.18-120.el5), rebased versions of monitoring tools (lm-sensors, net-snmp, openIPMI, openldap), new versions of yum and rpm, updated Kerberos subsystem, newer SAMBA (3.0.32) with support for Vista and Windows Server 2008, a ton kernel fixes and features, new hardware drivers (bcm43xx, bnx2/bnx2x, cpufreq, e1000/e1000e, mptscsi, qla24xx, etc.) and some interesting Technology Previews! dm-multipath install, which should allow anaconda to install Linux on systems with 2 or more HBAs. Support for ext4 file systems. CIFS (aka SAMBA) enhancements in the form of kernel DFS support and Kerberos security. Check the complete Release Notes for a full upda...

I've been seeing a weird error on my server console lately. We're using Dell PowerEdge 2950 and 2950 III running Oracle's Enterprise Linux 4U5, source code from RHEL AS 4u5, with no patches applied. Straight "base distribution" (I know, I know, bad idea but bare with me). I reported it to Dell's PE mailing list, but didn't get any conclusive answers. So I'm listing it here, for reference. Sep 1 15:36:26 kernel: hda: irq timeout: status=0xd0 { Busy } Sep 1 15:36:26 kernel: hda: irq timeout: error=0x00 Sep 1 15:36:26 kernel: hda: ATAPI reset complete I've found various reports of similar errors on different hardware but all seem to revolve around Red Hat's 4 release (OEL, RHEL, CentOS, Fedora) with various updates. See [ 1 ] [ 2 ] Some sources say it's a DMA-related issue. Others suggest to turn off ACPI using kernel boot parameters acpi=off and another says it's simply the HAL daemon probing your devices. You can configure the hal...

I have been getting some weird errors in some Linux VMs running on VMware ESX 3.0.x (and perhaps ESX 3.5, as we're migrating to that). The symptoms are always that some weird errors occur for users or some maintenance task, and when you check the VM's consoles, you discover that the file system is suddenly read-only. My dear friend MrVanes discovered there is a known issue with the VMware LSI driver mptscsi in VMware guests running Debian . "...one could get SCSI timeouts when there is massive workload on the host system. Some kernel versions these will get the file systems remounted read-only, which probably makes sense for real hardware, but doesn't make sense for emulated hardware. Instead it should just wait a bit longer." A savvy user called TuxyTurvy has thoroughly investigated this issue. He was using Red Hat RHEL 4 and 5 on Dell 1850 servers with a Clarion AX150i el-cheapo SAN. When I/O load got high, running under VMware, due to storage contention, SCSI t...

Found a little note on Red Hat today, that mentioned to use network bonding mode numbers instead of names. Add settings to /etc/modprobe.conf but use mode={0,1,2,3,4,5,6} to enable load-balancing, active-passive, fail-over or others. Update: also check out documentation on network bonding from the Linux Foundation. Update2: Official Linux kernel documentation on network bonding .

Cyberciti writes: "Linux desktop user generally uses small hard disk and partition, but on server you need a large partition. For example, you cannot create 4TB partition size (RAID based) using fdisk command. It will not allow you to create a partition that is greater than 2TB. To solve this problem use GNU parted command with GPT. It supports Intel EFI/GPT partition tables. [...] By default Red Hat Enterprise Linux/CentOS comes with GPT kernel support. However, if you are using Debian or Ubuntu Linux, you need to recompile the kernel." Update: Check a good post on IDoLinux about using GPT or LVM to partition file systems, and if you use hardware RAID in your system, do read about optimizing the Linux file system to maximize performance.

Besides good ol' CVS and the de facto successor Subversion (SVN), the new invention and wide-spread use of distributed version control systems (DVCS) such as Mercurial, Bazaar and git (used for the Linux kernel itself) also have their advantages. For one, iBanjo says you can " suck down the entire history of a Subversion repository into a local, private repository, do all of your commits locally, make branches, experiment all you want, then “push” back up to Subversion again. For those interested in this functionality, there is now a very promising Mercurial to Subversion bridge called hgsubversion . Although still early, the bridge was done right and from the ground up using Subversion's own API and by someone with a good understanding of both svn and hg. The HOWTO is on iBanjo .

I have up2date setup for my servers to update most packages but ignore kernel updates as well as oracleasm -related updates. Kernel updates break things and must be done manually, when needed. However, I prefer using yum over up2date. But until now I did not find an easy way to tell yum to ignore updates on some packages. An old Duke mailing list had an improved man yum text that eluded to add multiple exclude=package* statements to the repos in /etc/yum.repos.d/ . Thanks Duke! Here is my example yum repo configuration with the excluded packages: [el4_patch]name=Oracle Enterprise Linux - $releasever - $basearch - patchbaseurl=http://uln.example.com/yum/EL4/$releasever/patch/$basearch/exclude=kernel*exclude=oracleasm*gpgcheck=1enabled=1 Update: something is still not right for it just upgraded the kernel anyway. Update2: When I moved the 'exclude' lines before the 'enabled' line, the yum update process does indeed ignore kernel updates, as expected. Before, I had the...

I accidently did an update of a test server using ' yum update ', realizing I'd get a new kernel but forgetting the new kernel would break my PowerPath setup. It's EMC's software for getting multi-pathing and we use version 5.0.0 (EMCpower.LINUX-5.0.0-157.rhel). I should have used up2date because it knows how to update and skip undesired packages, such as the kernel... Oh well. *Nix hacks had some tips to fix PowerPath after kernel updates . You simply need to reinstall the package and force a replacement.

US-CERT is aware of active attacks against Linux-based computing infrastructures using compromised SSH keys. The attack appears to initially use stolen SSH keys to gain access to a system, and then uses local kernel exploits to gain root access. Once root access has been obtained, a rootkit known as " phalanx2 " is installed. Read more at US-CERT.

As a side-effect of the Linux boot process, I suddenly realized that our boot process is not fault-tolerant! We have 100+ servers that boot from SAN using two Qlogic 2460 HBAs. We installed EMC PowerPath 5.0.0 on OEL 4u5 to get multipathing and automatic fail-over in case a path fails. The OS is pretty well off in case of hardware faults. However, the boot process is not! Not even close! In stage 1, the boot loader reads the MBR, loads stage 1.5 so it can read the /boot ext2 partition where the kernel and initrd image are located. In our case, this is /dev/sdb1. But what if my HBA dies and /dev/sdb1 doesn't exist? It may be smart enough to try a device using the other path, but probably not. Also, since the kernel hasn't loaded yet, PowerPath does not exist and there is no multipath awareness to save the day... So once systems are installed and the number of disk partitions is stable, I can enhance GRUB with boot fallback systems , so it will try /dev/sdb1 and it that fails swi...