Oracle: Unbreak my Linux Heart

Howto convert a LiveCD image to a PXE Boot image . To convert the root of the ISO image to an NFS root and save on the RAM requirements when booting the image [RAM=2x(size of ISO in MB) + 60 MB], read this on PXE booting in general .

While are many excellent tutorials about creating and using LVM on Linux, not may show you how you can remove disks from LVM Volume Groups (VG) and reclaim storage or how to remove a Logical Volume (LV) from your LVM set-up. Here is what I did: Use -t to TEST ANY LVM action first! We are going to release 1 TB from LVM. The Volume group was extended with 1 TB storage to serve as a cheap NFS/CIFS file server when setting up our data center. It is now deprecated and replaced by a NAS so it's no longer needed. 1) check LVM; note the four 256 GB LUNs [root@server ~]# pvscan -v Wiping cache of LVM-capable devices Wiping internal VG cache Walking through all physical volumes PV /dev/sdb1 VG vgdata lvm2 [50.00 GB / 0 free] PV /dev/sdc1 VG vgdata lvm2 [256.00 GB / 0 free] PV /dev/sdd1 VG vgdata lvm2 [256.00 GB / 0 free] PV /dev/sde1 VG vgdata lvm2 [256.00 GB / 0 free] PV /dev/sdf1 VG vgdata lvm2 [256.00 GB / 0 free] PV /dev/sdg ...

Oracle is pleased to announce the general availability of Enterprise Linux Release 5 Update 4 for x86 (32 bit) and x86_64 (64 Bit) architectures. This update includes the following kernel/driver changes: Bug Fixes added by Oracle * Check to see if hypervisor supports memory reservation change [orabug 7556514] * Add entropy support to igb [orabug 7607479] * Convert ENETUNREACH to ENOTCONN [orabug 7689332] * Add xen pv/bonding netconsole support [orabug 6993043] * Shrink zone patch [orabug 6086839] * Fix aacraid not to reset during kexec [orabug 8516042] * Fix failure of file creation from hpux client [orabug 7579314] Fixes/Additions from the upstream distribution provider * A new tunable parameter /proc/sys/vm/max_writeback_pages has been added to the kernel, allowing system administrators to change the maximum number of modified pages kupdate writes to disk per iteration each time it runs. It defaults to 1024 or 4MB ...

An interesting read from Slashdot mentioned a company YippieMove who switched from VMware to FreeBSD Jails when their infrastructure ran into so many problems doing I/O that something had to change. Interestingly, BSD Jails resembles Solaris' Zones and Linux vservers . Bottom-line is that, as the comments mention very well, you have to “know your workload before picking the technology.” Also, which hardware generation you run VMware on has a big impact. Newer hardware has a significantly better support for virtualization when compared to older generation hardware. We've had similar I/O problems with NFS shares exported to various bare metals (BMs) and VMs and we choose to move to an ECM NAS instead (based on the number and the volumes of the shares). I doubt installing vSphere 4 on our old PowerEdge 2950 would make any difference as the old hardware is simply already stretched to its limits.

We are trying to create some common ground between Solaris and Linux environments. For example, auto mounting scripts of NFS file systems. One issue was that Sun uses the master map file called auto_master whereas Linux defaults to auto.master . Thankfully, you can tweak /etc/sysconfig/autofs and use MASTER_MAP_NAME to tell Linux to also use auto_master . Problem solved! Update: dont' forget to open /etc/nsswitch.conf and edit the line for autofs . I had to delete nisplus .

We currently use SVN to enable us to do release management of deployments. The SVN server holds all our kickstart, post installation and configuration scripts, as well as distribution scripts for software packages (Oracle database, application server, OEM, the OEM agent and more). The actual packages are not in SVN but on an NFS server. CollabNet released a new version of Subversion (SVN) a few days ago. And I'm excited! Subversion 1.6 introduces several new features, including improved authentication (Apache + LDAP) and file system storage as well as better handling of tree conflicts. Check the release notes for full details. The new version also introduces a new public way to reference the history of the repository. You can now read older versions of files through the web interface without the need for a SVN client. This enables 3rd party tools do innovative things through an easy to use interface into SVN.

A Sun Solaris friend of mine looked at me in disgust when he saw my /etc/fstab file to mount NFS mounts, CDs and more. I was seriously hurt so when he explained the shiny happy possibilities of autofs on Solaris (and Linux), I was immediately convinced: Configuration of the Automounter Autofs ... and all was well!

Due to the lack of some resources, I was asked to create an NFS server on Solaris. They figured all *nix flavors are alike and I should be able to do it. I was able to get it running, especially due to help the SunHelp NFS FAQ . I still have an unresolved issue concerning mountd ... That daemon keeps getting killed on the NFS server which results in no client being able to mount NFS shares any longer. *sigh* Update: turns out Local Zones cannot run an NFS server. Only the Global Zone (on a physical host) can have an NFS server. Other zones can use an NFS client to connect to it, even from other physical hosts. Alternatively, you can replicate NFS shares between physical hosts in the Global Zone and use clients to connect to it. Or loop back to the Global Zone from the Local Zone using LOFS.

I got NIS working last week, but simply following HOWTO guide from Linux Home Networking. Thanks guys! Only difference was that I explicitly defined /etc/securenets and did not need to set the NIS_DOMAIN_NAME as the server opted, but used my existing domain name instead. So instead of setting /etc/sysconfig/network to contain something like NIS_DOMAIN_NAME=MY-NIS-STUFF , I used the same name as my subdomain, i.e. NIS_DOMAIN_NAME=sub.domain.tld . After that, I started the NFS auto mounter on the clients, ran authconfig to set authentication to NIS (don't enable shadow passwords over NIS!) All that remains to be done, is testing, testing and testing, securing NIS for now, setting up a good, solid sudoers file and try to get Oracle's OEM working with NIS... (Hint: test or debug sudo with sudo -l ) Update: do yourself a favor and userdel and groupdel all local accounts or groups (except root) on your to-be NIS client(s), or at least prevent a NIS user from having the same nam...

After political and time pressure not to pursue any commercial products to integrate Linux user authentication into MS AD, I'm falling back to my original plan to set-up NIS(+) for now and deal with AD later... I followed instructions at Linux Home Networking and got the NFS server running but would get an error when logging in as a user: "unable to chdir to home directory". Fortunately, it seems a common error when using NIS and NFS (and Kerberos). It turns out my NFS client wasn't correctly configured. The automount file /etc/auto.home for /home listed my subnetwork (10.X.0.0/16). That's what I had the NFS server configured for. The automount file should just list '*' there... Next is NIS...

During the (scripted) updates of ESX using Altiris, we discovered a ton of timeouts on the ESX hosts. The problem was the NFS server was getting very slow and Altiris scripts were failing due to connection losses and timeouts. Although we updated /etc/init.d/nfs and increased the number of threads/servers from 8 to 16 (default is 8 per core, the VM has 2 cores), this didn't change the behavior we observed on the ESX hosts. Still timeouts... So I dug a little deeper and found NFS for clusters , an excellent tuning/testing document for NFS servers. I tested the server settings using the suggested nfsstat -s and observed next to nothing. All was fine. However, the /proc/net/rpc/nfsd file (using watch -d cat /proc/net/rpc/nfsd ) still showed at least 6 out of 8 processes (aka threads/servers) with 20+ seconds of 100% busy. Clearly something was wrong, but it wasn't the NFS server. More testing showed that manual actions (of what was going on in the scripts) pushed upto 100MB/s t...

Just wanted to share an update about my clamav setup with you... I currently have autoupdating of the data files working properly. But I still watch and download Dag Wieers' site for new packages of the engine. I tried swiping his site with wget for changes and automatically copying (aka mirroring) them, but the filenames are different each time. So I rely on the update emails from clamav and then check Dag Wieers' site. He usually has them pretty quickly anyway. (Thanks for that, BTW!) I roll out the clamav RPMs in a script from a shared NFS mount, install them, copy my custom .conf files over the default and do a freshclam to get the latest data files. I get the data files from a local web server, where I run a check every 6 hours for updates and put them in the root of the web server . The last thing is important, because it took me a while to get that right. The .cvd files must be in the web server's root, not in a subdir or so.

In any modern enterprise environment, security is a major issue. NFS and Linux/Unix have a long history together, but back then, security was that big a deal. Admins were few and servers too. So you know how dunnit. These days, when you configure NFS, security is important. So a local firewall on a server is no luxury. However, NFS uses several other daemons to do its duty and among these is RPC. RPC uses a random port numner in the high ranges, so you must first fix these to a static port (of your choice). Lowth.com has detailed instructions on the how and where, as well as a handy iptables firewall configurator wizard you can use in your browser. Red Hat has their own knowledgebase entry on the subject...

The docs on Red Hat Network Installations can also be applied to Oracle Enterprise Linux (OEL)... Easily. Just make sure that you may have to edit the file TRANS.TBL in Enterprise/RPMS . You can also use Altiris to setup a Linux ftp deployment server. Just boot the OEL cd with the prompt: linux askmethod Then, the OEL CD image will ask you if you'd like to use the CD-ROM or opt for an NFS, ftp or http install source. Before you hit the OK key, make sure that the local CD image disk 1 is unmounted . If you don't, the network install will fail and you'll see the file product.img appearing in your logs. Try again, this unmount the boot CD image from which you started and then hit OK to start the network install. This fixed all my problems (after several hours).

Today, we noticed a CPU hungry gam_server process using 100% CPU in some (32 bit) VM running Oracle Enterprise Manager. Not knowing what is did, we did some research. gam_server belongs to the gamin file alteration monitor , a part of FAM . By default, it monitors all /mnt/* and /media/* mounts. We happen to have an NFS share for common configs, sources and ISOs and such. So it makes some sense. Searching Google turns up a surprising amount hits for the same problem , dating back as far as 2004. There were some memory leaks with versions older than 0.1.7 but most problems should be fixed starting with that version. So update gamin if you don't have that version. The next issue is the default config for gamin. Two seperate solutions ( 1 | 2 ) both outline the solution: adjust the /etc/gamin/.gaminrc and set appropriate poll intervals for the file systems of your choice. In our case, we'll adjust NFS polls to every 30 seconds, ext3 to 10 secs. fsset nfs poll 30 fsset ext3 poll 10...

Kevin Closson’s Oracle Blog has an interesting blog about Mount Options for Oracle over NFS . It's not as hard as it may seem, but you have to pay attention to the option you use when mounting the NFS share. There are also little differences between various ports of Oracle for certain platforms. This, of course, makes troubleshooting nearly impossible. Kevin sheds his idea of this too. He may blog about this later in more detail...