Oracle: Unbreak my Linux Heart

EMC (VMware) has a great technical webcast and PDF online about Performance and Provisioning for Virtualized Oracle . I highly recommend you listen to it or read the sheets. Very informative if you want to use virtualized Oracle databases in a production environment. Most set-ups of virtualizing Oracle databases on VMware fail because storage I/O becomes a bottleneck. It turns out, 50% of all surveyed set-up had storage configured wrong. The problems has to do with storage queues in the VM and subsequently on the ESX host. Additionally, vSphere 4.1 introduces features ( SIOC ) to optimize the I/O better to the SAN and para virtualized SAN drivers can further reduce latency and increase throughput. Then the webcast also mentions Oracle's "native" NFS client ( dNFS , PDF) inside the database kernel. This reduces the CPU overhead and latency normally observed on NFS client in the OS. By using Oracle's native NFS client, you can use shared storage in a grid computing envi...

At work we are in the process of upgrading our VMware infrastructure to vSphere with new Vmax storage from EMC. One of the new features is VAAI , vStorage APIs for Array Integration. This abstracts certain aspects of storage to the SAN layer, where e.g. cloning can be a mere functional call that is then handled transparantly for the layer above. Yellow Bricks has a great post on VAAI sweetness collected from twitter and gathered or our convenience. Awesome!

Oracle's Virtualization Blog informed me that they've expanded the use of Oracle VM (OVM) significantly by supporting visualization on the SPARC platform. "Oracle VM Server for SPARC, previously called Sun Logical Domains, leverages the built-in SPARC hypervisor to subdivide supported platforms' resources (CPUs, memory, network, and storage) by creating partitions called logical (or virtual) domains. Each logical domain can run an independent operating system. Oracle VM Server for SPARC provides the flexibility to deploy multiple Oracle Solaris operating systems simultaneously on a single platform. Oracle VM Server also allows you to create up to 128 virtual servers on one system to take advantage of the massive thread scale offered by the CMT architecture."

A few days ago we released Oracle VM 2.1.5 which is available for download (registration free but required). This release contains the following : bugfixes on both Oracle VM Manager and Oracle VM Server side the new webservices interface which I wrote a blog entry about, a few weeks ago external data-collector cronjob/thread moved into the Oracle VM Manager instance minor cosmetic improvements on the Oracle VM Manager side This is a full CD release. So it is possible to do an install from scratch starting with 2.1.5, both the management and server side. Source: Oracle VM 2.1.5 (Wim Coekaerts Blog)

Although I'm using Oracle's Linux in the data center at work, I still keep up with other distributions. Just to check out what's going on and keep up. Lately, I've been playing with a new MediaWiki virtual appliance (VA) at work - where we keep most rough documentation and work-in-progress before moving it over to MS Word offial reports - and I must say turnkeylinux.org builds some mean VAs using Ubuntu JeOS. A 180MB ISO file is all you need. You create a VM, mount the ISO, boot, enter some root passwords and your set. It's integrated with Webmin, so you can completely configure the VM using your browser. No need to ssh into anything. :) Kudos! Anyway, I was going to install VMwareTools into the VM and remembered VMware had opened an apt-style repository for VMwareTools for use with Debian-based Linux distributions. So I made a few changes to the VM and installed it using apt-get! Wow! I love Debian, did I mention that before. ;) Here is what I did: wget http://p...

An interesting read from Slashdot mentioned a company YippieMove who switched from VMware to FreeBSD Jails when their infrastructure ran into so many problems doing I/O that something had to change. Interestingly, BSD Jails resembles Solaris' Zones and Linux vservers . Bottom-line is that, as the comments mention very well, you have to “know your workload before picking the technology.” Also, which hardware generation you run VMware on has a big impact. Newer hardware has a significantly better support for virtualization when compared to older generation hardware. We've had similar I/O problems with NFS shares exported to various bare metals (BMs) and VMs and we choose to move to an ECM NAS instead (based on the number and the volumes of the shares). I doubt installing vSphere 4 on our old PowerEdge 2950 would make any difference as the old hardware is simply already stretched to its limits.

Wouldn't it be great if you could use the LCD on a Dell PowerEdge to display the names of the VMs running on it? You may never to know where a VM actually, but if you do, you usually need it badly. Well Nickapedia wrote the script and the HOWTO for you: Dell Server LCD Update Script . It uses Dell OMSA on ESX and IPMI to get access to the VM names and then outputs these on the LCD. Brilliant! See video on YouTube ! BTW: won't work on ESXi because OMSA is not supported there.

There's a big discussion going on at my workplace about virtualizing Oracle databases. It used to be not done because of performance, but it offers many really useful features such as hardware independence, fail-over, VMotion and cloning, vmdk copies, etc. Oracle had previously told us they supported Oracle in VMs. When they announced Oracle VM Server based on Xen, they retracted this statement. We weren't impressed. Microsoft tried to do the same things while they were working on Hyper-V and so on. Eventually, anti-monopoly laws forced them to support competitors in the same way as they support their own virtualization flavor. So I expect Oracle to do the same. In the mean time... we're playing with Oracle Xen thingy as well as investigating virtual databases running on VMware 4.0 aka vSphere .

If you care about clustering and currently use VMware, you should read Wim Coekaerts' blog about vCenter Server Heartbeat and Oracle VM Manager with Oracle Clusterware ... I was gonna look into Oracle VM (and Xen) later this year anyway, but may move it up because of this.

Just discovered a fabulous resource for installing and configuring Oracle databases on various OS platforms: ORACLE-BASE . Oracle 9i, 10g, 11g but also RAC on Linux, Microsoft, VMware, Xen, etc. Recommended!

As we run 700+ VMs in our data center, 40% of which Linux BTW, this news is pretty relevant to me: VMware decided to make their Advanced Technical Trainings (PDF) - mainly used for in-house support staff and engineers - available for free! Previous VCP certification is recommended, or at least thorough experience with VMware. Source: VMware - VMware Partner News

I'm back from a 3 week vacation, so that explains the radio silence here. This year we'll upgrade from OEL 4.5 to OEL 5.x, upgrade OEM to OEM 10.2.4 and start using Oracle RAC on our Dell Poweredge 2950 servers. We'll also upgrade to ESX 3.5 everywhere and use EMC Networker backups. And finally, management of some serious Sun hardware (T2000, M5000, M9000) will fall under our responsibility so all our deployment scripts will be updated to fit that platform as well. Lots of other minor tweaks coming up too, so it should be an interesting year again. Subversion is almost fully integrated and we've been working towards release cycles for this year, to help streamline releases management and new features.

Here is a useful tip from the Dell Linux-PowerEdge mailing list : If you run RHEL3, perhaps because your software still won't support a higher release or your app was built on it, and when moving your physical servers over to VMware for better support (P2V?), you notice that the network won't come up, try removing the VLAN ID from the ifcfg file, i.e. /etc/sysconfig/network-scripts/ifcfg-eth0 .

I have been getting some weird errors in some Linux VMs running on VMware ESX 3.0.x (and perhaps ESX 3.5, as we're migrating to that). The symptoms are always that some weird errors occur for users or some maintenance task, and when you check the VM's consoles, you discover that the file system is suddenly read-only. My dear friend MrVanes discovered there is a known issue with the VMware LSI driver mptscsi in VMware guests running Debian . "...one could get SCSI timeouts when there is massive workload on the host system. Some kernel versions these will get the file systems remounted read-only, which probably makes sense for real hardware, but doesn't make sense for emulated hardware. Instead it should just wait a bit longer." A savvy user called TuxyTurvy has thoroughly investigated this issue. He was using Red Hat RHEL 4 and 5 on Dell 1850 servers with a Clarion AX150i el-cheapo SAN. When I/O load got high, running under VMware, due to storage contention, SCSI t...

With the whole green motion going around, I was happy to join lesswatts.org to try and limit the power consumed by my servers. Although we use VMware extensively to bundle VMs onto ESX hosts, there still are many bare metal servers for one reason or another. We're experimenting with VMware's experimental DPM (Dynamic Power Management) to collect and pack idle VMs onto a single host. DPM then suspends the host and saves power. In the morning, when loads increase , the VMs are VMotioned off the hosts again, after these are reawakened. For Linux, I'm gonna give PowerTOP a spin and test it in the coming year to try and optimize my Oracle Enterprise Linux servers.

On and off we have timekeeping issues with certain applications in Linux VMs. Now I know this is a common problem and nothing new. Bottom line is that you should leave the time keeping to VMware Tools and sync all ESX hosts to the same time source via NTP. However, it still can go wrong. There is a whole list of sources with tips, hints and solutions, but your mileage may vary. I was just pointed to a very detailed white paper by VMware that may help to solve your application specific problems, whether in VMware, Xen or Hyper-V. Check the 27 page PDF : " Timekeeping in VMware Virtual Machines " Update: Read the VMware KnowledgeBase on time keeping in Linux VMs or search for KB ID# 1006427.

ZDnet reports the Oracle is going high profile at the Linuxworld conference , positioning it as a world league player for Enterprise Linux... :yawn: But it made an interesting announcement : the introduction of the Oracle VM Template , pre-configured software stacks designed to reduce installation, configuration and maintenance costs by eliminating the need to install and test from the ground up. Always a big time saver, so I'm definitely going to look into this. Hope I can convert the Xen VMs to VMware format, but I think so.

Had an issue with an attempted upgrade of VMware Tools using the supplied script. It failed miserably and left my VM without network and IP address. Going to the console, I tried to remove (rpm -e VMwareTools) or update (rpm -Uhv VMwareTools) the RPM, but the script had crippled the installation. HowToGeek had some helpful commands to manually remove most of the VMware Tools files from the file system, so a new clean installation could overwrite all existing traces.

Since we are virtualizing many of our Linux servers, this post was of interest to us: "VMmark 1.0 utilized only 32-bit workloads, which was a reasonable mix when the benchmark was first defined roughly three years ago. However, 64-bit applications and OSes are becoming much more prevalent and we need the ability to characterize this more complex reality. To address this, [ for vmware 1.1 ] we have transitioned three of the VMmark workloads - Java server, database server, and web server - to 64-bit. In order to maintain comparability with the existing version 1.0 results, we have retained the underlying virtual hardware definitions and load levels for each workload." This will let us benchmark our specific ESX host setup and test Oracle Linux VM servers under various settings...

Solved an obscure error for some Windows 2003 R2 servers that were unable to map a drive to a SAMBA share. SAMBA is running on Oracle Enterprise Linux 4U6 with configs as defined by a wiki over at SAMBA . Error logs showed repeated " Failed to verify incoming ticket " errors. Problem turned out to be related to Kerberos. Not so much the version or the encryption (MIT vs. Heimdal), as many sources mention, but to plain and simple time keeping! The Windows servers giving the errors were VMs running on ESX hosts. VMware Tools is supposed to do the time keeping and time sync the VMs to ESX, however that had slipped a bit too much. Kerberos accepts a maximum of 5 minutes. Ours was 9 minutes off...