Oracle: Unbreak my Linux Heart

My OEL4 servers still have duplicate RPM packages on 64 bit servers. Libs for 32 bit and 64 bit are often installed, even though they may not always be used. When upgrading a package, such as glibc, you run into the error that yum won't touch a package because it finds duplicates. In this case you can try Removing Duplicate RPM Packages or remove a package using an explicit architecture tag: yum package.{i386,i686,x86_64} such as yum remove glibc.i686 . If you have RHEL/OEL5 or higher, you're in luck because you can try to install the package yum-utils and run package-cleanup --cleandupes . yum-utils offer a bunch of useful tools for working with yum repos and so on.

At OpenWorld, Oracle announced an Oracle optimized Linux kernel that is upto 75% faster than the bundled Red Hat kernel. But what's the catch? Well, it's simply a 2.6.32-based kernel , tweaked and tuned for Oracle software. Here's how to get it , if you want to. And LWN has a more detailed technical description of the changes from the mainstream kernel . Basically, the kernel optimizes communication: network (InfiniBand, 10Gb Ethernet), storage (SSD, Flash memory) and memory ( NUMA ). It optimizes drivers and reduces complexity in the kernel (i.e. remove unused drivers and so on). And considering the compatibility issue, here is Oracle's claim: "Third-party applications that run on RHEL 5 should run unchanged on Oracle Linux with the Unbreakable Enterprise Kernel , while delivering significant performance and reliability improvements for end users." Good reads are CIOupdate and PCWorld

I've been diving a bit into the RAM configuration of our new R710 Dell servers. They have new memory configuration settings. "Memory Optimized" mode can yield a 50% performance improvement at no cost other than to place DIMMs in appropriate slots. Dell has a white paper detailing the options a bit more: Optimizing DDR3 Memory Settings in New 11th-Generation Dell PowerEdge Servers . It also lists trade-offs of choosing one configuration over the other or when to go for a balanced intermediate.

Got a question about Linux name caching today. Some servers are requesting the same hostname to the DNS server every 2 seconds... Looked into it and the name caching daemon is not running. Got all the answers I needed here: alt.os.linux.redhat: Re: DNS cache on a RH9 system? . Also found a separate package called dnscache . And a HOWTO for bind to create a local DNS cache on your workstation or server. Simply starting nscd and editing nscd.conf seems sufficient for me now. BTW, here is my updated /etc/nscd.conf . If anyone has more pointers, please comment. logfile /var/log/nscd.log # use 1-10 for more verbose debugging into log debug-level 0 # 3 minimum, 5 default filethreads 5 # 32 is default max-threads 32 # restart intervals may fail if non-root server-user nscd # user granted to check nscd stats using 'nscd -g' stat-user nagios # 5 is default reload-count 5 # restart periodically paranoia yes # restart every 4 hrs, default is 1 hour #rest...

Anyone using a subversion server heavily needs good backups of their code. svnadmin dump is a good way to make regular backups. Or the mirroring method may work well for you. I found a really useful, simple, elegant and clean tool to backup SVN repos with: svnbackup by Doug Hellmann (don't let the name fool you!). His script utilizes the existing tools but adds value, which is excellent in my book. He dumps the svn repo using svnadmin , splits the repo into commits sets (100 by default) and compress them in the process. The greatest feature, IMHO, is however the ability to let the scripts scp the backups to a different server! Genius! Works likes a charm in 30 seconds! Here is my daily cron job: #!/bin/bash # http://code.google.com/p/svnautobackup/ # create dump of main SVN repo(s) /usr/local/bin/svnbackup.sh --scp userid@server:/data/users/userid /var/svn/repositories/svnrepo/ echo "***SVN io_se dumped" >> /var/log/messages If you have more than one repo, build ...

A Sun Solaris friend of mine looked at me in disgust when he saw my /etc/fstab file to mount NFS mounts, CDs and more. I was seriously hurt so when he explained the shiny happy possibilities of autofs on Solaris (and Linux), I was immediately convinced: Configuration of the Automounter Autofs ... and all was well!

Dell's OpenManage tools keep some logs under /var/log and do not clean them up. You also can't turn them off and they contain important info from the PERC controllers. Anyway, the mailing list mentions this often, so I thought I post a little solution using logrotate: Dell OpenManage and their TTY logs . /var/log/TTY_*.log { daily notifempty missingok rotate 7 # how many you want to save compress delaycompress sharedscripts postrotate /usr/sbin/invoke-rc.d dsm_om_shrsvc restart >/dev/null 2>&1 || true endscript } See also Planet Malaysia .

With the whole green motion going around, I was happy to join lesswatts.org to try and limit the power consumed by my servers. Although we use VMware extensively to bundle VMs onto ESX hosts, there still are many bare metal servers for one reason or another. We're experimenting with VMware's experimental DPM (Dynamic Power Management) to collect and pack idle VMs onto a single host. DPM then suspends the host and saves power. In the morning, when loads increase , the VMs are VMotioned off the hosts again, after these are reawakened. For Linux, I'm gonna give PowerTOP a spin and test it in the coming year to try and optimize my Oracle Enterprise Linux servers.

While working with Subversion for our code, tools, scripts and so on, some people also wanted to put documentation inside SVN. That way, we can release code and documentation in one go. Easy, clear, practical! Or is it? It seems Subversion is not really good at handling large(r) binary files, such as executables, Word or even image files. While on paper it should be capable of doing just that, in practice you often get severe performance penalties when checking in and out binary files.

I just came across this: Configure BASH, KSH, TCSH, ZSH Shell To Logout User Automatically After Idle Time . I was looking for that months ago but couldn't find it. Then I forgot about it, until now! All you need to do is set and export an environment variable, which can be set system-wide in /etc/profile !

On and off we have timekeeping issues with certain applications in Linux VMs. Now I know this is a common problem and nothing new. Bottom line is that you should leave the time keeping to VMware Tools and sync all ESX hosts to the same time source via NTP. However, it still can go wrong. There is a whole list of sources with tips, hints and solutions, but your mileage may vary. I was just pointed to a very detailed white paper by VMware that may help to solve your application specific problems, whether in VMware, Xen or Hyper-V. Check the 27 page PDF : " Timekeeping in VMware Virtual Machines " Update: Read the VMware KnowledgeBase on time keeping in Linux VMs or search for KB ID# 1006427.

Just an interesting tid-bit that I had been wondering about for a while: why is there a "boot stage 1.5" during Linux boot? I see a "stage 1" notification, then stage 1.5, but nothing else... How come? Well, IBM has the answer in " Inside the Linux boot process ". Linux uses a 2 stage boot process, stage 1 loads the boot loader, stage loads the kernel. But to allow Linux to load the kernel from a native file system such as ext2 or ext3 (or Reiser, XFS, ZFS, etc.), GRUB introduces an additional stage 1.5 that understands those file systems. Stage 2 then still loads the kernel, only now it can reside in a normal Linux partition instead of a raw disk sectors, such as with LILO. Neat!

Through an explanation about memory management in Linux on the Gentoo Wiki, I was lead to an article on KernelTrap about High Memory In The Linux Kernel . Though I'm still not sure, this leads me to believe Oracle compiles the kernel with the CONFIG_HIGHMEM4G=y switch on and thus the 3 GB RAM for user space application should be in place. However, I'm forced to test this to make sure. :)

I recently had a question about Linux that created more questions the longer I thought about it. The question was: "Given the app doesn't care, would you recommend 32 bit or 64 bit Oracle Linux for a server?" Driver compatibility plays a part, as does the amount of RAM the app needs. If you need more than 4 GB of RAM, you'll have to use a 64bit kernel... or do you...? I'm still not absolutely sure about it as information is a bit ambiguous. While some say you cannot allocate more than 3.7 GB of RAM to any one process on a 32 bit machine, using PAE , others say to just check free -m or vmstat . Red Hat lists /proc/meminfo as a good source of information and explains what you see there. All said and done, I still don't know for sure. I have a VM with 4 cores and 8 GB of RAM, and it says: [root@server ~]# uname -a Linux server 2.6.9-55.0.0.0.2.ELsmp #1 SMP Wed May 2 14:59:56 PDT 2007 i686 i686 i386 GNU/Linux [root@server ~]# cat /proc/meminfo MemTotal: 805...

Just recovered from a boot disk crash... Murphy's Law dictates that the crash happened on our backup server! 6 TB of data potentially useless. Needless to say I'm glad that we managed to recover from it... In the process of recovering, I detected the need to refer to my boot partition in a location independent way. Because my Dell PowerEdge 2950 tends to insert or delete USB partitions and devices "whenever it feels like it", which causes my boot disk to be /dev/sdb1 or /dev/sdc1 or /dev/emcpowere1 , I want to be able to identify it no matter what it is physically called. That's were boot labels come in handy. See the Linux Partition HOWTO on the why and the how. (Note: actually the server inserts/deletes USB device when you add/remove the nousbstorage from the linux kernel boot parameters.

Sometimes you need a logout script to do some housekeeping, such as clean up after processes or users... Some Linux shells have means for this, such as /bin/csh. Bash shell has a .bash_logout. But you may need something stronger. Blogger Digest showed a neat trick to Enable Linux Login Logout Scripts to use a specific script of your choice. Great! Thanks!

While researching ways to create a custom boot image , so I can provision new systems automatically with a CD or a mounted ISO image, I also learned how to create a custom splash.lss image . Most web pages out there instruct you to install ancient, obscure giftoppm tools - which you may not have or know how to get. I found out Irfanview (for Windows) easily lets you open and edit JPG or GIF images, rework them, add logos and text and save the result as a PPM file. Then just scp the .ppm file to Linux and use the ppmtolss16 tool (obtained through 'yum install syslinux') to convert the PPM file to LSS. Build the custom ISO image and you're done!

During the (scripted) updates of ESX using Altiris, we discovered a ton of timeouts on the ESX hosts. The problem was the NFS server was getting very slow and Altiris scripts were failing due to connection losses and timeouts. Although we updated /etc/init.d/nfs and increased the number of threads/servers from 8 to 16 (default is 8 per core, the VM has 2 cores), this didn't change the behavior we observed on the ESX hosts. Still timeouts... So I dug a little deeper and found NFS for clusters , an excellent tuning/testing document for NFS servers. I tested the server settings using the suggested nfsstat -s and observed next to nothing. All was fine. However, the /proc/net/rpc/nfsd file (using watch -d cat /proc/net/rpc/nfsd ) still showed at least 6 out of 8 processes (aka threads/servers) with 20+ seconds of 100% busy. Clearly something was wrong, but it wasn't the NFS server. More testing showed that manual actions (of what was going on in the scripts) pushed upto 100MB/s t...

Did you know that you can often save real power on your machine, even without updating any of the software? The site LessWatts.org contain a set of tips and tricks, contributed by community members and Intel engineers , to help you find the best power tuning for your laptop, desktop, or server. Highly recommended!