Solved: Mystery DHCP Requests from Dell R815 Broadcom NeXtreme II NICs

Listen, my friends, and I will tell you a tale of mystery MAC addresses, DCHP, and Broadcom woes.

It all started when we got a new server. A Dell R815, to be exact. The specs looked great, it could hold a ton of RAM for some of our bioinformatics projects, and the cost was low. Everyone was happy.

Then one day someone called and said, hey, there are a ton of DHCP requests and we've traced them back to [room that server is in]. Here's the MAC address: 002xxxxxxxd1.

This seemed weird to me because we had set up static IPs on the server's two live NICs, both on the primary NIC and on the iDRAC enterprise NIC. To my knowledge, there were no NICs set up with DHCP.

Even more weird was the fact that the MAC address 002xxxxxxxd1 did not actually exist. The R815 had five NICs, numbered like this:

002xxxxxxxd0 (NIC 1)
002xxxxxxxd2 (NIC 2)
002xxxxxxxd4 (NIC 3)
002xxxxxxxd6 (NIC 4)
002xxxxxxxd8 (iDRAC)

We pored over the NIC configurations in Windows 2008 R2 server and its virtual switch and virtual machines (since it was running the Hyper-V role). Nothing. Meanwhile, our sniffer showed that we were not dreaming...indeed, DHCP requests were emanating from that MAC address.

We pulled the network cable and the DHCP requests stopped. We pulled out our hair and ceremoniously waved a Windows 98 boot disk over the server -- we keep one around for just these sorts of occasions.

Finally we turned up something...in the iDRAC remote management screen we found a list of MAC addresses which included the mysterious one we were looking for.

I'll cut to the chase. The Broadcom NeXtreme II 5906 NICs have iSCSI capability built-in. The embedded iSCSI HBA has an IPv4 configuration setting that defaults to DHCP. This setting was on by default, and means that DHCP requests were emanating from the server, receiving a 10.11.x.x address.

The DHCP requests cannot be disabled without installing the BroadCom Managed Applications Control Suite and then following the instructions here.

It is very probable that I'm not very bright and other people expect DHCP requests to come from the iSCSI hardware on their NICs by default. Just thought I'd share. And I note that this is only happens on Windows, not Linux on the same hardware.

tl;dr DHCP requests are issued by default on Broadcom NeXtreme II network adapters on Windows servers unless you download a special tweaker to turn them off, and it took me a while to figure this out.

Reference: Broadcom iSCSI offload engine defaults to DHCP enabled (IBM, 2011-03-03)

Red Hat Enterprise Linux 5 on Dell OptiPlex 990 Initial Impressions

I received a Dell OptiPlex 990 and installed RHEL5 on it. Some preliminary observations:

• I wanted a RAID 1, so I reached for a trusty 3ware 8006-2LP, which is a 64-bit-capable SATA RAID PCI card that I typically run in a 32-bit PCI slot. But the OptiPlex 990 has placed a heatsink directly behind the PCI slot. The card can't physically fit in the slot because the heatsink is in the way. That led to my next adventure.

• I've tried several times over the years to run software RAID on OptiPlex systems both with FreeBSD and Linux. Everything would set up fine but under load the computer would reboot and rebuild. I suspect that the SATA controllers on the motherboard were to blame. But I thought hey, this whole Sandy Bridge thing might have improved things -- let's try again. So I put in the Red Hat installer. That led to my next adventure.
• The onboard Sandy Bridge video is not compatible with the installer. After X tries to start the result is a black screen. Not too surprising, really. Fortunately the installer offers a handy VNC option which worked fine.
• I created two RAID partitions, mirrored them, and assigned the /boot mountpoint to the mirror. Then I created a large LVM volume group and created logical volumes for the rest of my mountpoints. A brief summary of operations:

Create RAID partition of 100MB on /dev/sda
Create RAID partition of 100MB on /dev/sdb
Create RAID device on /dev/md0 using the above two partitions in RAID1. Mount point is /boot.

# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/VolGroup00-LogVol01
65G 2.3G 59G 4% /
/dev/mapper/VolGroup00-LogVol02
9.5G 152M 8.9G 2% /var/log
/dev/md0 99M 13M 82M 14% /boot
tmpfs 902M 0 902M 0% /dev/shm


Now on to test whether the onboard SATA controllers would fall down. I ran two simultaneous instances of bonnie++, one in /var/bonnie and one in /var/log/bonnie (separate logical volumes, if you've been paying attention and haven't fallen asleep yet). No rebooting. I was impressed.

My final observation: the onboard NIC is an Intel 82579LM which does not seem to have an onboard TCP offload engine. Though the 82577LM does in this matrix so I'm not sure. Let's see what ethtool has to say:

# ethtool -k eth0
Offload parameters for eth0:
Cannot get device udp large send offload settings: Operation not supported
rx-checksumming: on
tx-checksumming: on
scatter-gather: on
tcp segmentation offload: on
udp fragmentation offload: off
generic segmentation offload: off
generic-receive-offload: off


Update 2011-06-15: The 990 is definitely noisier than the 980 or the 7xx series due to fan placement and lack of hooded airflow internally. Also, with RHEL6 and BIOS A03 the OptiPlex 990 will shut down but will hang when restarted. In order for it to reboot cleanly it is necessary to add reboot=pci to the kernel options in /etc/grub.conf.

Multiple monitors with Windows 7 and an ATI FirePro 2460 (FireMV)

When planning a triple or quad monitor setup with Windows 7 and an ATI FirePro™ 2460 Multi-View Professional Graphics card, read the small print.

In my case, reading the following small print from the data sheet carefully would have saved some time:

Important: When using mini DisplayPort-to-singlelink DVI adapters, quad output is divided into two pairs (displays 1 and 2, and displays 3 and 4). The displays used in each pair must be identical (with the same manufacturer and model number) and must be identically configured (with the same resolution)...

In my case, I had two identical Dell 19" displays and one of the new energy-efficient G2210 displays. I paired the two 19-inchers on DisplayPort connectors 1 and 2, and put the G2210 on the third DisplayPort connector. All used the DisplayPort-to-singlelink DVI connectors that came in the box with the card.

Problem: In Windows 7 x64, when trying to configure multiple monitors in the control panel (Control Panel\Appearance and Personalization\Display\Screen Resolution), I kept getting the message "Unable to save display settings" when I tried to extend the desktop.

Solution: What was meant by that cryptic message was really "the two displays you think are identical really aren't". Sure enough, one of the 19-inch displays was an UltraSharp 1901 and the other was an UltraSharp 1905. They look identical and have the same native resolution and connectors. I scuttled around, found another 1905, and swapped it in. When I powered up, sure enough, with two UltraSharp 1905's the control panel doesn't complain at all.

Did I mention the 2460 is fanless? Yum.