Oracle: Unbreak my Linux Heart

Howto convert a LiveCD image to a PXE Boot image . To convert the root of the ISO image to an NFS root and save on the RAM requirements when booting the image [RAM=2x(size of ISO in MB) + 60 MB], read this on PXE booting in general .

Robin at yo61.com found a great and handy way to discover the MAC addresses of the embedded NICs of your Dell (PowerEdge) servers. Instead of going through DRAC, logging in, starting Java, etc. you can also get the same using the RAC command line interface: racadm racdump. Read the article on yo61.com for complete details. Excellent! If only I could find a similar command for my 500 PE2950 servers with DRAC5...

I've been playing a bit with the nscd now and want to share some tips related to tuning the nscd.conf file. To see how the DNS cache is doing, use nscd -g. nscd configuration: 0 server debug level 26m 57s server runtime 5 current number of threads 32 maximum number of threads 0 number of times clients had to wait yes paranoia mode enabled 3600 restart internal passwd cache: no cache is enabled [other zero output removed] group cache: no cache is enabled [other zero output removed] hosts cache: yes cache is enabled yes cache is persistent yes cache is shared 211 suggested size 216064 total data pool size 1144 used data pool size 3600 seconds time to live for positive entries 20 seconds time to live for negative entries 66254 cache hi...

Got a question about Linux name caching today. Some servers are requesting the same hostname to the DNS server every 2 seconds... Looked into it and the name caching daemon is not running. Got all the answers I needed here: alt.os.linux.redhat: Re: DNS cache on a RH9 system? . Also found a separate package called dnscache . And a HOWTO for bind to create a local DNS cache on your workstation or server. Simply starting nscd and editing nscd.conf seems sufficient for me now. BTW, here is my updated /etc/nscd.conf . If anyone has more pointers, please comment. logfile /var/log/nscd.log # use 1-10 for more verbose debugging into log debug-level 0 # 3 minimum, 5 default filethreads 5 # 32 is default max-threads 32 # restart intervals may fail if non-root server-user nscd # user granted to check nscd stats using 'nscd -g' stat-user nagios # 5 is default reload-count 5 # restart periodically paranoia yes # restart every 4 hrs, default is 1 hour #rest...

Dell has released OMSA 6.0.2 for monitoring and remotely managing Dell PowerEdge servers that run Linux. Plugins for Nagios, MOM and others exist, so you can tie your Linux servers to most corporate tools.

An article on Debian Administration got my attention today, since it listed something my colleague just showed to me the other day. Often, when debugging network issues, you call up some support guy and he wants to know which switch ports your NIC is using. Often, you're in the office and the server is in a distant data center. So getting to the cables is next impossible or very time-consuming at the very least. And often, documentation is not trusted because you are having problems to begin with. Using CDPR or the less brand-specific lldpd can become a god-sent tool. It allows you to monitor Cisco Discovery Protocol packets from the command line and tell the network admin exactly what he needs to know. :) Solaris admins have snoop to make use of. Linux guys can also use tcpdump: tcpdump -nn -v -i -s 1500 -c 1 ‘ether[20:2] == 0×2000’

Never seen so many IPMI messages flying around on the Dell Linux mailing list. It's an open standard to monitor information of basic health parameters of a server. Here is a very good from by someone from Dell that sums up the usage and support of IPMI under Linux on Dell PowerEdge servers. Basically, you only need ipmitool to issue power on/off the servers. If you want to do more, you need to install the IPMI packages (yum install OpenIPMI)

An age-old problem in mixed IT environments is that Windows servers can register themselves dynamically in a (Windows) DNS server and Unix/Linux servers can't (easily). Solaris has a way to fix this and I just discovered that Red Hat has one too. The RHEL 4 Reference Guide mentions adding the parameter DHCP_HOSTNAME in the ifcfg-eth# files. That will tell the DHCP client to specify a hostname when acquiring an IP address. Another search led to the use of nsupdate which seems to register a Linux host using Kerberos in the AD so that it can update its hostname on future reboots. I haven't tested both methods yet, but I will soon and let you know.

If you have configured bonding of two or more physical interfaces in a logical bonded interface capable of fail-overs, and you want to know the status of the bond , check /proc! /proc/net/bonding/bond0 shows the status of the bond: $ cat /proc/net/bonding/bond0 Ethernet Channel Bonding Driver: v2.6.3-rh (June 8, 2005) Bonding Mode: fault-tolerance (active-backup) Primary Slave: None Currently Active Slave: eth0 MII Status: up MII Polling Interval (ms): 100 Up Delay (ms): 0 Down Delay (ms): 0 Slave Interface: eth0 MII Status: up Link Failure Count: 1 Permanent HW addr: 00:1a:a0:34:73:dc Slave Interface: eth4 MII Status: up Link Failure Count: 1 Permanent HW addr: 00:15:17:38:b9:2a

Here is a useful tip from the Dell Linux-PowerEdge mailing list : If you run RHEL3, perhaps because your software still won't support a higher release or your app was built on it, and when moving your physical servers over to VMware for better support (P2V?), you notice that the network won't come up, try removing the VLAN ID from the ifcfg file, i.e. /etc/sysconfig/network-scripts/ifcfg-eth0 .

Network World : "Oracle has contributed data-integrity protection code, partly developed with the hardware vendor Emulex, to the Linux kernel, the vendors announced Tuesday. The code helps maintain 'comprehensive data integrity' as information 'moves from application to database, and from [the] Linux operating system to disk storage,' according to a statement. It also lowers the possibility that erroneous data will get written to disk. The companies' effort is meant to help data-center administrators track and address corrupted data quickly, lowering costs and downtime"

Found a little note on Red Hat today, that mentioned to use network bonding mode numbers instead of names. Add settings to /etc/modprobe.conf but use mode={0,1,2,3,4,5,6} to enable load-balancing, active-passive, fail-over or others. Update: also check out documentation on network bonding from the Linux Foundation. Update2: Official Linux kernel documentation on network bonding .

Just as a pointer for everyone looking to do the same, here is an article on the Dell PowerEdge mailing list about VLAN tagging on Red Hat Linux . It should also work for Oracle Enterprise Linux (OEL). The main tip was not to setup IP addressing for the native interface (eth2 in the example), but to add it for the sub interfaces eth2.vlan1 and the2.vlan2 instead.

Found an add-on for Nagios , the well-known open source monitoring tool based on SNMP. NagViz adds visualization to Nagios, in a way that - as far as I understand it - lets you create a kind of dashboards to show everything is up. For instance, all devices related to Email delivery or a web application farm or your extranet portal. Should help convince board members that Nagios is worthwhile as well as providing people with easy to use screens. Check out SourceForge.net .

After political and time pressure not to pursue any commercial products to integrate Linux user authentication into MS AD, I'm falling back to my original plan to set-up NIS(+) for now and deal with AD later... I followed instructions at Linux Home Networking and got the NFS server running but would get an error when logging in as a user: "unable to chdir to home directory". Fortunately, it seems a common error when using NIS and NFS (and Kerberos). It turns out my NFS client wasn't correctly configured. The automount file /etc/auto.home for /home listed my subnetwork (10.X.0.0/16). That's what I had the NFS server configured for. The automount file should just list '*' there... Next is NIS...

During the (scripted) updates of ESX using Altiris, we discovered a ton of timeouts on the ESX hosts. The problem was the NFS server was getting very slow and Altiris scripts were failing due to connection losses and timeouts. Although we updated /etc/init.d/nfs and increased the number of threads/servers from 8 to 16 (default is 8 per core, the VM has 2 cores), this didn't change the behavior we observed on the ESX hosts. Still timeouts... So I dug a little deeper and found NFS for clusters , an excellent tuning/testing document for NFS servers. I tested the server settings using the suggested nfsstat -s and observed next to nothing. All was fine. However, the /proc/net/rpc/nfsd file (using watch -d cat /proc/net/rpc/nfsd ) still showed at least 6 out of 8 processes (aka threads/servers) with 20+ seconds of 100% busy. Clearly something was wrong, but it wasn't the NFS server. More testing showed that manual actions (of what was going on in the scripts) pushed upto 100MB/s t...

Been looking some more into the user administration issues I face when maintaining a few hunderd Linux servers for various people, customers and developers... Centrify seems to have a nice product that allows me to authenticate Linux users against an (existing) MS Active Directory Server (AD). While that is nice, it is not enough. Granted, their DirectControl family of products covers a lot and is nice, what I miss is the ability to control which user can issue what commands on a Linux/Unix level. That is, DirectControl does not seem to link into the su/sudo/sudoers facilities of Linux. NIS is a common way to centralize Linux user management on a network and control access to servers, do password management and so on. However, I thought NIS was largely incompatible with more common, cross-platform directory solutions such as LDAP. Even MS AD can talk LDAP! ;) So I was more than happy to discover that PADL provides free toosl to help migrate from flat access files (passwd, NIS, etc.) ...

" Oracle Validated Configurations are pre-tested, validated architectures with software, hardware, storage, and networking components with included documented best practices for deployment. Oracle and its strategic partners offer and recommend these configurations to enable end-users to deploy fully tested solutions to achieve standardization with high performance, scalability, and reliability while lowering infrastructure costs."

Slashdot | Oracle Contributes Linux Code, Expands Hardware Support : "Oracle expanded the list of hardware compatible with its Linux distribution and added support for Novell's YAST administration tool. They have now certified six hardware configurations able to run Oracle Enterprise Linux. Certified products include those made by Compellent Technologies, Dell, Egenera, EMC, Hewlett-Packard, Pillar Data Systems and Unisys. Oracle also said it is releasing an open-source version of the YAST Linux installation and configuration tool for Oracle Enterprise Linux and Red Hat Enterprise Linux under the General Public License."

In any modern enterprise environment, security is a major issue. NFS and Linux/Unix have a long history together, but back then, security was that big a deal. Admins were few and servers too. So you know how dunnit. These days, when you configure NFS, security is important. So a local firewall on a server is no luxury. However, NFS uses several other daemons to do its duty and among these is RPC. RPC uses a random port numner in the high ranges, so you must first fix these to a static port (of your choice). Lowth.com has detailed instructions on the how and where, as well as a handy iptables firewall configurator wizard you can use in your browser. Red Hat has their own knowledgebase entry on the subject...