Oracle: Unbreak my Linux Heart

Once again I am running into the weird issue where at install time my NICs are enumerated one way and after a reboot another. This first happened to me in 2007, while installing Oracle Enterprise Linux (OEL) 4u5 on a Dell PowerEdge 2950. The on-board NICs were numbered 'gb1' and 'gb2' but enumerated 'eth1' and 'eth0', respectively. Strangely, the problem went away silently and just now reappeared in OEL 5u3. Today I am using new R11 servers from Dell, the R710 with two dual on-board Broadcom NICs as well the quad-NIC add-on from Intel. At boot, my 'eth0' seems to be what I expect it but after a reboot it becomes 'eth4'. Found an updated white paper from Dell, listing a work-around: nic-enum-whitepaper-v4.pdf .

At work I run a lot of Linux servers with Oracle Linux (OEL, Red Hat-derived) and our Dell PowerEdge 2950 servers have no problems with them. At first, OEL 4u5 had a terribly confusing issue with device enumeration where the first on-board NIC (labeled 1) would end up as eth1 and the other (labeled 2) would become eth0. That was fixed later on and I've reported on it here, 2 years ago. Lately, I've been playing with other Linux distos and seeing how well the (old) Dell servers hold up new distributions. I always used netimages where I install the OS over the network. For one, this greatly speeds up the ISO download and it also ensures I have an updated system after installation ready for testing. While Red Hat and SUSE-based distos are not a problem, Debian-based versions can be. This mainly has to do with the strict philosophy of their maintainers that non-free drivers and firmware should not be included in the image. In my case, the lack of built-in Broadcom (bnx2) drivers ca...

An article on Debian Administration got my attention today, since it listed something my colleague just showed to me the other day. Often, when debugging network issues, you call up some support guy and he wants to know which switch ports your NIC is using. Often, you're in the office and the server is in a distant data center. So getting to the cables is next impossible or very time-consuming at the very least. And often, documentation is not trusted because you are having problems to begin with. Using CDPR or the less brand-specific lldpd can become a god-sent tool. It allows you to monitor Cisco Discovery Protocol packets from the command line and tell the network admin exactly what he needs to know. :) Solaris admins have snoop to make use of. Linux guys can also use tcpdump: tcpdump -nn -v -i -s 1500 -c 1 ‘ether[20:2] == 0×2000’

Never seen so many IPMI messages flying around on the Dell Linux mailing list. It's an open standard to monitor information of basic health parameters of a server. Here is a very good from by someone from Dell that sums up the usage and support of IPMI under Linux on Dell PowerEdge servers. Basically, you only need ipmitool to issue power on/off the servers. If you want to do more, you need to install the IPMI packages (yum install OpenIPMI)

If you have configured bonding of two or more physical interfaces in a logical bonded interface capable of fail-overs, and you want to know the status of the bond , check /proc! /proc/net/bonding/bond0 shows the status of the bond: $ cat /proc/net/bonding/bond0 Ethernet Channel Bonding Driver: v2.6.3-rh (June 8, 2005) Bonding Mode: fault-tolerance (active-backup) Primary Slave: None Currently Active Slave: eth0 MII Status: up MII Polling Interval (ms): 100 Up Delay (ms): 0 Down Delay (ms): 0 Slave Interface: eth0 MII Status: up Link Failure Count: 1 Permanent HW addr: 00:1a:a0:34:73:dc Slave Interface: eth4 MII Status: up Link Failure Count: 1 Permanent HW addr: 00:15:17:38:b9:2a

Here is a useful tip from the Dell Linux-PowerEdge mailing list : If you run RHEL3, perhaps because your software still won't support a higher release or your app was built on it, and when moving your physical servers over to VMware for better support (P2V?), you notice that the network won't come up, try removing the VLAN ID from the ifcfg file, i.e. /etc/sysconfig/network-scripts/ifcfg-eth0 .

Network World : "Oracle has contributed data-integrity protection code, partly developed with the hardware vendor Emulex, to the Linux kernel, the vendors announced Tuesday. The code helps maintain 'comprehensive data integrity' as information 'moves from application to database, and from [the] Linux operating system to disk storage,' according to a statement. It also lowers the possibility that erroneous data will get written to disk. The companies' effort is meant to help data-center administrators track and address corrupted data quickly, lowering costs and downtime"

Found a little note on Red Hat today, that mentioned to use network bonding mode numbers instead of names. Add settings to /etc/modprobe.conf but use mode={0,1,2,3,4,5,6} to enable load-balancing, active-passive, fail-over or others. Update: also check out documentation on network bonding from the Linux Foundation. Update2: Official Linux kernel documentation on network bonding .

Just as a pointer for everyone looking to do the same, here is an article on the Dell PowerEdge mailing list about VLAN tagging on Red Hat Linux . It should also work for Oracle Enterprise Linux (OEL). The main tip was not to setup IP addressing for the native interface (eth2 in the example), but to add it for the sub interfaces eth2.vlan1 and the2.vlan2 instead.

" Oracle Validated Configurations are pre-tested, validated architectures with software, hardware, storage, and networking components with included documented best practices for deployment. Oracle and its strategic partners offer and recommend these configurations to enable end-users to deploy fully tested solutions to achieve standardization with high performance, scalability, and reliability while lowering infrastructure costs."

I am having very slow downloads (up2date and yum) from my Oracle Linux boxes. A little research shows many (Fedora/RedHat) Linux users having the same problem. I'm guessing Oracle's Enterprise Linux (OEL) has similar issues. I am following steps outlined on a thread on LinuxQuestions.org and hopefully can report back with positive news. The thread lists one interesting sysctl switch I hadn't seen before...

The Dell white paper outlining the problem with NIC emumeration under Linux on a Poweredge 2950 also writes that starting with Linux: 2.6.19 Kernel , the kernel has been equipped with appropriate code to correctly detect and solve this issue.

As we'll be using teaming of network cards (NICs) in Windows and Linux, I was searching for a manual for teaming (Word file) of the Broadcom Corporation NetXtreme II BCM5708 Gigabit Ethernet cards. At the same time, I'm including a link to the Broadcom Driver FAQ ...

After severals days of banging head against wall, one of our network technicians discovered that Dell's PowerEdge 2950 switches eth0 and eth1 under Linux . *sigh* Fortunately, Dell has a solution or workaround for this problem that occurs with the on-board Broadcom NetXtreme II 5708 GigabitEthernet NICs. We have also found that after prolonged periods of inactivity, the Broadcom NIC seem to go down as if in powersave mode or so. I can't however confirm this issue, as we're still resolving and confirming it.

I've been working on the Linux servers the last couple of days and while the Intel PRO/1000 PT quad port server adapter works fine with Intel's lastest drivers and the on-board dual port Broadcom NetExtreme II was detected properly by Oracle Enterprise Linux (version 4.4 at this time), I still have some problems with network trunking and VLANs in Linux . A colleague found a brilliant start-up script that simply processes the /etc/sysconfig/networking directory and starts your interfaces in the proper order and activates VLANs prior to bringing up the interface (eth0, eth1, etc), using the vconfig command. Oh one note: you may have to activate the 802.1q trunking kernel module using 'modprobe 8021q' or it won't work. I had to...