Oracle: Unbreak my Linux Heart

EMC (VMware) has a great technical webcast and PDF online about Performance and Provisioning for Virtualized Oracle . I highly recommend you listen to it or read the sheets. Very informative if you want to use virtualized Oracle databases in a production environment. Most set-ups of virtualizing Oracle databases on VMware fail because storage I/O becomes a bottleneck. It turns out, 50% of all surveyed set-up had storage configured wrong. The problems has to do with storage queues in the VM and subsequently on the ESX host. Additionally, vSphere 4.1 introduces features ( SIOC ) to optimize the I/O better to the SAN and para virtualized SAN drivers can further reduce latency and increase throughput. Then the webcast also mentions Oracle's "native" NFS client ( dNFS , PDF) inside the database kernel. This reduces the CPU overhead and latency normally observed on NFS client in the OS. By using Oracle's native NFS client, you can use shared storage in a grid computing envi...

We had a strange issue during Oracle WebLogic development yesterday. Setting up a secure connection between a client and the WebLogic Admin server took really long on our shiny new hardware (Dell R710) compared the older servers (Dell 2950). After we made sure everything was exactly the same, people started to blame C1 and C1E states and other powersave or cpu throttling measures. However, given the different in setup times (2-6 sec vs 50-90 sec), I suspected it to be something else. Throttling down from 2.23 GHz to 1.6 GHz cannot cause such a great difference. Or at least be unlikely. Lo and behold, it turned out to be related to the seeding of the random number generator in Java. Normally this is linked to /dev/random but can be sped up by using /dev/urandom , which uses some entropy of your system. Still, the urandom device, while faster, still caused a 20-fold difference. As a workaround, you can tell your JVM which seed generator to use and prevent JVM from waiting or looking or ...

Besides having the option to download installation media from Oracle (for which you must register), you can use the Oracle Public Yum Server to get packages. For instance, to help solve ad-hoc dependencies. This alleviates you from the need to keep a local mirror available for all Linux releases.

A few days ago, Oracle followed suit by Red Hat and released OEL6. Here are the Oracle Linux 6 Release Notes . At first glance, most notable changes are default ext4 file system. Better performance monitoring and tracking using perf and new user space "top" tools, and it no longer uses up2date but instead relies totally on yum. Yummie! Last but not least: the Unbreakable Kernel kernel-uek-2.6.32-100.28.5.el6 is installed and activated by default! This shouldn't matter much, unless you run the latest state-of-the-art hardware and use SSDs a lot. But 3rd party software vendors may like you stepping away from the Red Hat default kernel-2.6.32-71.el6 kernel. So check first! OL6 is the first Linux distribution from Oracle where they no longer patch the Red Hat kernel. Instead, they provide the Red Hat kernel as-is as an option to be configured for use in /etc/grub/menu.lst. The default kernel in OL6 is the Unbreakable Enterprise Kernel that Oracle maintains itself. As far...

Oracle Linux 5.6 makes Unbreakable Enterprise Kernel the default kernel after installation unless you perform a custom installation. Note that Unbreakable Enterprise Kernel is available for the x86_64 platform only. The Unbreakable Enterprise Kernel is delivered via the package kernel-uek .

Dell Community : "RHEL 6 provides SR-IOV functionality on supported hardware which provides near native performance for virtualized guests. Single-Root I/O Virtualization (SR-IOV) specification, introduced by PCI-SIG details how a single PCIe device can be shared between various virtualization guests. Devices capable of SR-IOV functionality support multiple virtual functions on top of the physical function. Virtual Function is enabled in hardware as a light weight PCIe function. Operating System cannot discover this function as it does not respond to the PCI bus scan and requires support in the host’s driver. As in PCIe pass-through, a Virtual function of a SR-IOV capable card can be directly assigned to the guest operating system. A virtual function driver running in the guest manages this device."

Like many others, we are running into issues with Oracle licensing. Oracle charges a lot for its software and using state-of-the-art hardware you may even end up paying more, rather than less. The thing is, Oracle refuses to see VMware virtualization as "hard partitioned" but instead as "soft partitioned". That is, a single core VM may run on all cores that are available to it. So Oracle licensing requires you to factor in every core. Oracle's own Oracle VM (OVM) for x86 and SPARC (Xen-based) uses hard partitioning, but this comes at the cost of loss of overcommitment. That is, you cannot assign more cores than you have and increase the CPU utilization of your hosts. This is not necessarily bad, but means you can run less VMs on a given host. Reference: Oracle Storage Guy and Weinshenker.net

At OpenWorld, Oracle announced an Oracle optimized Linux kernel that is upto 75% faster than the bundled Red Hat kernel. But what's the catch? Well, it's simply a 2.6.32-based kernel , tweaked and tuned for Oracle software. Here's how to get it , if you want to. And LWN has a more detailed technical description of the changes from the mainstream kernel . Basically, the kernel optimizes communication: network (InfiniBand, 10Gb Ethernet), storage (SSD, Flash memory) and memory ( NUMA ). It optimizes drivers and reduces complexity in the kernel (i.e. remove unused drivers and so on). And considering the compatibility issue, here is Oracle's claim: "Third-party applications that run on RHEL 5 should run unchanged on Oracle Linux with the Unbreakable Enterprise Kernel , while delivering significant performance and reliability improvements for end users." Good reads are CIOupdate and PCWorld

A recent article on Linux security showed that an old backdoor (cve-2007-4573) , patched in kernel 2.6.22.7, has resurfaced . Using the leak, any user that has a local user account on a 64 bit server, can easily get root rights, using the compatibility layer. While I don't fully understand the workings, I do understand that all OEL4/OEL5 64 bit servers are potentially harmed as they have kernels 2.6.9 and 2.6.18, resp. So I logged a SR on Metalink to see if I have a big security issue in the data center. Update: CVE-2007-4573 has been renamed CVE-2010-3301 and RHEL states : This issue did not affect the versions of Linux kernel as shipped with Red Hat Enterprise Linux 3, 4, 5, and Red Hat Enterprise MRG, as they do not contain the upstream commit d4d67150 that introduced this flaw. Update2: the issue is also listed as CVE-2010-3081 , which is slightly different and Red Hat does track this one, as it affects RHEL5 . Update3: Ksplice has a test available to verify your syste...

I've had a long feud at work with business units who work with outside contractors who insist on writing web apps using the very latest version of PHP, Apache and MySQL. While I welcome the use of open source, state-of-the-art technology, the one thing these contractors often overlook is the need for stability and continuity in enterprise-grade applications. Even web applications. It's useless to use the latest greatest feature set in PHP or MySQL, if the organizations you're targeting are all running RHEL 4u8, 5u2 or even RHEL3! And many do. The Enterprise Distributions for Linux lag behind public version for up to 18 months! And they won't easily upgrade just because you ask or because your application was built using the most recent version of an open source package. The company may be able to give you a special server in some DMZ but you may lack access to certain key servers, because the security status of your app and server have not (yet) been approved. Inste...

Oracle's Virtualization Blog informed me that they've expanded the use of Oracle VM (OVM) significantly by supporting visualization on the SPARC platform. "Oracle VM Server for SPARC, previously called Sun Logical Domains, leverages the built-in SPARC hypervisor to subdivide supported platforms' resources (CPUs, memory, network, and storage) by creating partitions called logical (or virtual) domains. Each logical domain can run an independent operating system. Oracle VM Server for SPARC provides the flexibility to deploy multiple Oracle Solaris operating systems simultaneously on a single platform. Oracle VM Server also allows you to create up to 128 virtual servers on one system to take advantage of the massive thread scale offered by the CMT architecture."

Oracle has released an update to Unbreakable Linux, shortly after Red Hat released RHEL 5 update 5. Oracle's Linux portal ULN should have OEL5.5 available by now. Oracle's Linux OEL is downloadable by anyone, provided you register first. Only updates cost money.

OraclePartnerNetwork has created a series of videos on YouTube about Oracle VM 2.2 . They vary from installing Oracle VM, creating a VM, to live migration, enabling HA, etc.

I'm in the middle of creating a deployment image of OEL 5u3 (based on RHEL5u3) for new Dell PowerEdge R710 servers. After creating a generic and untested setup for the BIOS, my VMware guy came and informed me that vSphere 4 recommendations list to disable memory interleaving. Generally, letting NUMA do its thing yields better performance. So I'm now checking whether or not use interleaving for my 32 GB servers or to use NUMA out-of-the-box on Linux 2.6.18 as shipped with OEL5u3. I could switch to " Optimizer Mode " for the 32 GB but I'll need to have all servers physically checked and DIMMs reseated. So I opted for Interleaving to hope and try to gain some performance increase. Now I find out NUMA could be better and it is supported from RHEL5, which each update improving on it. I'll have to run more tests to be sure but for now I'm glad I discovered I could enable NUMA on OEL4u5 or higher as well, using a kernel option at boot time. Be sure to check nu...

How does ordinary virtualization relate to virtualization used in a cloud computing environment? That was the topic of a talk at the Cloud Computing Expo in Santa Clara last week. Is there a difference? Or is it merely a logical or organizational distinction but the techniques used are still the same? Adam Hawley talked about this , slides linked to in the article, and how Oracle's products do or will fit into this paradigm. Database services in the cloud, Application Server services in the cloud, middleware provisioning in the cloud for yourself and your customers, isolation, privacy, security, replication, disaster management. Everything is still the same yet different.

Once again I am running into the weird issue where at install time my NICs are enumerated one way and after a reboot another. This first happened to me in 2007, while installing Oracle Enterprise Linux (OEL) 4u5 on a Dell PowerEdge 2950. The on-board NICs were numbered 'gb1' and 'gb2' but enumerated 'eth1' and 'eth0', respectively. Strangely, the problem went away silently and just now reappeared in OEL 5u3. Today I am using new R11 servers from Dell, the R710 with two dual on-board Broadcom NICs as well the quad-NIC add-on from Intel. At boot, my 'eth0' seems to be what I expect it but after a reboot it becomes 'eth4'. Found an updated white paper from Dell, listing a work-around: nic-enum-whitepaper-v4.pdf .

On Wim Coekaerts Blog I read that Zend and Oracle are working closer together. This makes PHP/Zend Server more integrated with Oracle Enterprise Linux (OEL). Zend Server is a complete, Oracle-enabled, enterprise-ready Web application Server for running and managing PHP applications that require a high level of reliability, performance and security. Connectivity to Oracle databases is delivered out of the box. We have a long history of working with Zend and the PHP community to ensure that the latest and greatest Oracle features are available in the PHP database connectivity libraries. Oracle now hosts an rpm package on their ULN servers called zend-server-repo , which updates yum to also look at the software repositories hosted by Zend. (Note: you may need to use a proxy server to get to it, if you have ULN locally mirrored, or mirror the Zend repo as well!) When a user wants to install Zend Server, just a simple command will install the zend-server product directly. FYI, the zend rp...

Where I had great difficulties trying to get OMSA to install let alone work in Oracle's Enterprise Linux 4 (u5 and above), I am happy to report that I got it working out-of-the-box after an install of my custom 5u3 image, following the instructions on the OMSA Repository Wiki . A simple http proxy setting, the wget boot strap line to configure yum and the dependencies and I was in business! Excellent work, Dell!

Oracle's announced Oracle VM 2.2 , their latest Xen-based general purpose virtualization technology platform for their Red Hat-based Enterprise Linux version (OEL). Coupled with a new ocfs2 version and a few enhancements, you get ThinProvisioning for VMs and snapshots on cluster-aware files systems. That means they can span across data centers or locations. Update: wording changed as suggested by comment Update2: Oracle VM 2.2 CLI available too

Oracle is pleased to announce the general availability of Enterprise Linux Release 5 Update 4 for x86 (32 bit) and x86_64 (64 Bit) architectures. This update includes the following kernel/driver changes: Bug Fixes added by Oracle * Check to see if hypervisor supports memory reservation change [orabug 7556514] * Add entropy support to igb [orabug 7607479] * Convert ENETUNREACH to ENOTCONN [orabug 7689332] * Add xen pv/bonding netconsole support [orabug 6993043] * Shrink zone patch [orabug 6086839] * Fix aacraid not to reset during kexec [orabug 8516042] * Fix failure of file creation from hpux client [orabug 7579314] Fixes/Additions from the upstream distribution provider * A new tunable parameter /proc/sys/vm/max_writeback_pages has been added to the kernel, allowing system administrators to change the maximum number of modified pages kupdate writes to disk per iteration each time it runs. It defaults to 1024 or 4MB ...