Solved: Mystery DHCP Requests from Dell R815 Broadcom NeXtreme II NICs

Listen, my friends, and I will tell you a tale of mystery MAC addresses, DCHP, and Broadcom woes.

It all started when we got a new server. A Dell R815, to be exact. The specs looked great, it could hold a ton of RAM for some of our bioinformatics projects, and the cost was low. Everyone was happy.

Then one day someone called and said, hey, there are a ton of DHCP requests and we've traced them back to [room that server is in]. Here's the MAC address: 002xxxxxxxd1.

This seemed weird to me because we had set up static IPs on the server's two live NICs, both on the primary NIC and on the iDRAC enterprise NIC. To my knowledge, there were no NICs set up with DHCP.

Even more weird was the fact that the MAC address 002xxxxxxxd1 did not actually exist. The R815 had five NICs, numbered like this:

002xxxxxxxd0 (NIC 1)
002xxxxxxxd2 (NIC 2)
002xxxxxxxd4 (NIC 3)
002xxxxxxxd6 (NIC 4)
002xxxxxxxd8 (iDRAC)

We pored over the NIC configurations in Windows 2008 R2 server and its virtual switch and virtual machines (since it was running the Hyper-V role). Nothing. Meanwhile, our sniffer showed that we were not dreaming...indeed, DHCP requests were emanating from that MAC address.

We pulled the network cable and the DHCP requests stopped. We pulled out our hair and ceremoniously waved a Windows 98 boot disk over the server -- we keep one around for just these sorts of occasions.

Finally we turned up something...in the iDRAC remote management screen we found a list of MAC addresses which included the mysterious one we were looking for.

I'll cut to the chase. The Broadcom NeXtreme II 5906 NICs have iSCSI capability built-in. The embedded iSCSI HBA has an IPv4 configuration setting that defaults to DHCP. This setting was on by default, and means that DHCP requests were emanating from the server, receiving a 10.11.x.x address.

The DHCP requests cannot be disabled without installing the BroadCom Managed Applications Control Suite and then following the instructions here.

It is very probable that I'm not very bright and other people expect DHCP requests to come from the iSCSI hardware on their NICs by default. Just thought I'd share. And I note that this is only happens on Windows, not Linux on the same hardware.

tl;dr DHCP requests are issued by default on Broadcom NeXtreme II network adapters on Windows servers unless you download a special tweaker to turn them off, and it took me a while to figure this out.

Reference: Broadcom iSCSI offload engine defaults to DHCP enabled (IBM, 2011-03-03)

Slides from DrupalCamp Iowa 2011

Here are the slides from my morning presentation at yesterday's "DrupalCorn" DrupalCamp in Des Moines, Iowa. The presentation was designed to be a brief high-level overview of Drupal.

Introduction to Drupal 7 Architecture (PDF, 1.8MB)

Solution to Invalid command 'PubcookieAppID'

When setting up Pubcookie, you may encounter an "Internal Server Error" in your browser and the following error in /var/log/httpd/ssl_error_log (or wherever you're keeping your SSL error log):

[alert] [client] /var/www/html/foo/bar/baz/.htaccess: Invalid command 'PubcookieAppID', perhaps misspelled or defined by a module not included in the server configuration


This might be a headscratcher for a while, since you've probably been working hard to make sure that your Pubcookie configuration is nice and tidy and your .htaccess file is set up with PubcookieAppID, like this:

AuthType NetID
PubcookieAppID fribble
require valid-user


In fact, Apache is telling you exactly what you need to know: it can't make sense of the PubcookieAppID directive because the module that interprets that directive is not loading and thus is "not included in the server configuration."

To solve this, make sure that a line like this is actually somewhere in your configuration, normally somewhere like /etc/httpd/conf.d/pubcookie.conf:

LoadModule pubcookie_module modules/mod_pubcookie.so

In my case, I had commented it out while getting SSL to work. Duh.

Installing rpy and rpy2 on RHEL6

I wanted to install rpy and rpy2 on Red Hat Enterprise Linux 6. Here's how I did it. There have been several fixes in the rpy SVN repository that have not shown up in the version downloadable from SourceForge. Hopefully that's been fixed by now, but here's how I installed it by retrieving rpy directly from the repository.

First I made sure that the python-devel package was installed to avoid the error src/RPy.h:63:20: error: Python.h: No such file or directory:

yum install python-devel

Then I installed R and R-devel from EPEL:

rpm -Uvh http://download.fedora.redhat.com/pub/epel/6/x86_64/epel-release-6-5.noarch.rpm
yum install R R-devel


Then I downloaded rpy and extracted it:

curl http://rpy.svn.sourceforge.net/viewvc/rpy/trunk/rpy/?view=tar > rpy.tar.gz
tar xvzf rpy.tar.gz
cd rpy
python setup.py install


Lo and behold, it imports without error:

python -c "import rpy"

Yay!

Besides R and R-devel, rpy2 requires that the readline-devel package be installed, otherwise installation fails with ./rpy/rinterface/_rinterface.c:79:31: error: readline/readline.h: No such file or directory.

yum install readline-devel

After that, rpy2 installs easily:

curl http://pypi.python.org/packages/source/r/rpy2/rpy2-2.2.0.tar.gz#md5=a42a7f1e6ddb10dc3a1886c2f4309fab > rpy2-2.2.0.tar.gz
tar xvzf rpy2-2.2.0.tar.gz
cd rpy2-2.2.0
python setup.py install

Solution to nodereference_autocomplete_access error

We recently rolled out a site to testing and got a mysterious error:

warning: call_user_func_array() [function.call-user-func-array]: First argument is expected to be a valid callback, 'nodereference_autocomplete_access' was given in /var/www/html/drupal/includes/menu.inc on line 453.

We were using Acquia Drupal, so the nodereference module was right where one would expect. What was Drupal's problem?

Turned out to be totally unrelated. The developer had missed checking Drupal's .htaccess file into the repository, so this copy of Drupal was missing its .htaccess file, and all sorts of havoc (including the message above) resulted. The tip was when clicking on the login form resulted in a 404 Page Not Found, which led to the realization that clean URLs weren't working, which pointed the big fat Drupal finger directly at .htaccess problems.

Adding threads support to perl built by MacPorts

I had installed perl5.12 from MacPorts. sudo port installed listed

perl5 @5.12.3 lang/perl5
perl5.12 @5.12.3 lang/perl5.12
perl5.8 @5.8.9 lang/perl5.8


To enable threading support, I used


sudo port upgrade --enforce-variants perl5.12 +threads

Mail Forwarding on OS X Server 10.5 Leopard: Using /etc/aliases

I like to have my mailserver set up so that mail sent to apache@www.example.com (through a relay at mail.example.com) comes to one of my email addresses.

On my old crufty Mac OS X Server mailserver, which runs postfix, I was wondering why mail sent to apache@www.example.com was not getting delivered, even though in /etc/aliases on mail.example.com clearly had the entries

Instead, I was receiving errors like relay=cyrus, delay=0.13, delays=0.02/0.02/0/0.09, dsn=4.3.0, status=deferred (temporary failure. Command output: couldn't connect to lmtpd: No such file or directory_ 421 4.3.0 deliver: couldn't connect to lmtpd_

I was able to fix this by changing the relay from cyrus to local. That was OK since I am not using cyrus anyway; I have no local accounts on the mailserver.

I made the change by clicking a checkbox called "Deliver to "/var/mail" when POP & IMAP are disabled" in the OS X Server GUI.

The change has the effect of changing a line in /etc/postfix/main.cf.

Before:

After:

Postfix will now use the local transport, which actually checks /etc/aliases and delivers accordingly.

And yes, this is my last OS X Server and it's on the list to be migrated to RHEL.

Solution to scp missing on Minimal RHEL6 Install

If you choose the minimal installation of Red Hat Enterprise Linux 6, you do indeed get a minimal install. It's so minimal that things like wget and scp are missing. It's pretty obvious that yum install wget will get you wget but if you want scp:


yum install openssh-clients

Red Hat Enterprise Linux 5 on Dell OptiPlex 990 Initial Impressions

I received a Dell OptiPlex 990 and installed RHEL5 on it. Some preliminary observations:

• I wanted a RAID 1, so I reached for a trusty 3ware 8006-2LP, which is a 64-bit-capable SATA RAID PCI card that I typically run in a 32-bit PCI slot. But the OptiPlex 990 has placed a heatsink directly behind the PCI slot. The card can't physically fit in the slot because the heatsink is in the way. That led to my next adventure.

• I've tried several times over the years to run software RAID on OptiPlex systems both with FreeBSD and Linux. Everything would set up fine but under load the computer would reboot and rebuild. I suspect that the SATA controllers on the motherboard were to blame. But I thought hey, this whole Sandy Bridge thing might have improved things -- let's try again. So I put in the Red Hat installer. That led to my next adventure.
• The onboard Sandy Bridge video is not compatible with the installer. After X tries to start the result is a black screen. Not too surprising, really. Fortunately the installer offers a handy VNC option which worked fine.
• I created two RAID partitions, mirrored them, and assigned the /boot mountpoint to the mirror. Then I created a large LVM volume group and created logical volumes for the rest of my mountpoints. A brief summary of operations:

Create RAID partition of 100MB on /dev/sda
Create RAID partition of 100MB on /dev/sdb
Create RAID device on /dev/md0 using the above two partitions in RAID1. Mount point is /boot.

# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/VolGroup00-LogVol01
65G 2.3G 59G 4% /
/dev/mapper/VolGroup00-LogVol02
9.5G 152M 8.9G 2% /var/log
/dev/md0 99M 13M 82M 14% /boot
tmpfs 902M 0 902M 0% /dev/shm


Now on to test whether the onboard SATA controllers would fall down. I ran two simultaneous instances of bonnie++, one in /var/bonnie and one in /var/log/bonnie (separate logical volumes, if you've been paying attention and haven't fallen asleep yet). No rebooting. I was impressed.

My final observation: the onboard NIC is an Intel 82579LM which does not seem to have an onboard TCP offload engine. Though the 82577LM does in this matrix so I'm not sure. Let's see what ethtool has to say:

# ethtool -k eth0
Offload parameters for eth0:
Cannot get device udp large send offload settings: Operation not supported
rx-checksumming: on
tx-checksumming: on
scatter-gather: on
tcp segmentation offload: on
udp fragmentation offload: off
generic segmentation offload: off
generic-receive-offload: off


Update 2011-06-15: The 990 is definitely noisier than the 980 or the 7xx series due to fan placement and lack of hooded airflow internally. Also, with RHEL6 and BIOS A03 the OptiPlex 990 will shut down but will hang when restarted. In order for it to reboot cleanly it is necessary to add reboot=pci to the kernel options in /etc/grub.conf.

Nonparsing Apache Log Messages Solved

I set up a new Red Hat Enterprise Linux 5 webserver but the nightly emails from logwatch were complaining about logs not being parsed correctly:

--------------------- httpd Begin ------------------------

(Only the first ten are printed; there were a total of 188)
"1.2.3.4 - - [30/Mar/2011:01:49:18 -0500] \"GET /foo/bar/modules/node/node.css HTTP/1.1\" 200 678 \"https://www.example.edu/foo/bar/baz\"
...


I looked in /etc/logwatch/conf/services but there was no httpd.conf in there. In fact, there was nothing there, so the speculation by logwatch as to what the problem was was appreciated but not helpful.

Looking more carefully at the log entries, it became apparent that they were all from referrers that were requests over SSL. When I pored over the logging configuration for the virtual hosts on this box, I discovered that the CustomLog directive for the host listening on 443 was very slightly different than the default host; yet they were writing to the same log file.

Making the two CustomLog directives identical fixed the problem. (I also changed the configuration so that https requests were written to a separate file.)