Solution to Invalid command 'PubcookieAppID'

When setting up Pubcookie, you may encounter an "Internal Server Error" in your browser and the following error in /var/log/httpd/ssl_error_log (or wherever you're keeping your SSL error log):

[alert] [client] /var/www/html/foo/bar/baz/.htaccess: Invalid command 'PubcookieAppID', perhaps misspelled or defined by a module not included in the server configuration


This might be a headscratcher for a while, since you've probably been working hard to make sure that your Pubcookie configuration is nice and tidy and your .htaccess file is set up with PubcookieAppID, like this:

AuthType NetID
PubcookieAppID fribble
require valid-user


In fact, Apache is telling you exactly what you need to know: it can't make sense of the PubcookieAppID directive because the module that interprets that directive is not loading and thus is "not included in the server configuration."

To solve this, make sure that a line like this is actually somewhere in your configuration, normally somewhere like /etc/httpd/conf.d/pubcookie.conf:

LoadModule pubcookie_module modules/mod_pubcookie.so

In my case, I had commented it out while getting SSL to work. Duh.

Red Hat Enterprise Linux 5 on Dell OptiPlex 990 Initial Impressions

I received a Dell OptiPlex 990 and installed RHEL5 on it. Some preliminary observations:

• I wanted a RAID 1, so I reached for a trusty 3ware 8006-2LP, which is a 64-bit-capable SATA RAID PCI card that I typically run in a 32-bit PCI slot. But the OptiPlex 990 has placed a heatsink directly behind the PCI slot. The card can't physically fit in the slot because the heatsink is in the way. That led to my next adventure.

• I've tried several times over the years to run software RAID on OptiPlex systems both with FreeBSD and Linux. Everything would set up fine but under load the computer would reboot and rebuild. I suspect that the SATA controllers on the motherboard were to blame. But I thought hey, this whole Sandy Bridge thing might have improved things -- let's try again. So I put in the Red Hat installer. That led to my next adventure.
• The onboard Sandy Bridge video is not compatible with the installer. After X tries to start the result is a black screen. Not too surprising, really. Fortunately the installer offers a handy VNC option which worked fine.
• I created two RAID partitions, mirrored them, and assigned the /boot mountpoint to the mirror. Then I created a large LVM volume group and created logical volumes for the rest of my mountpoints. A brief summary of operations:

Create RAID partition of 100MB on /dev/sda
Create RAID partition of 100MB on /dev/sdb
Create RAID device on /dev/md0 using the above two partitions in RAID1. Mount point is /boot.

# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/VolGroup00-LogVol01
65G 2.3G 59G 4% /
/dev/mapper/VolGroup00-LogVol02
9.5G 152M 8.9G 2% /var/log
/dev/md0 99M 13M 82M 14% /boot
tmpfs 902M 0 902M 0% /dev/shm


Now on to test whether the onboard SATA controllers would fall down. I ran two simultaneous instances of bonnie++, one in /var/bonnie and one in /var/log/bonnie (separate logical volumes, if you've been paying attention and haven't fallen asleep yet). No rebooting. I was impressed.

My final observation: the onboard NIC is an Intel 82579LM which does not seem to have an onboard TCP offload engine. Though the 82577LM does in this matrix so I'm not sure. Let's see what ethtool has to say:

# ethtool -k eth0
Offload parameters for eth0:
Cannot get device udp large send offload settings: Operation not supported
rx-checksumming: on
tx-checksumming: on
scatter-gather: on
tcp segmentation offload: on
udp fragmentation offload: off
generic segmentation offload: off
generic-receive-offload: off


Update 2011-06-15: The 990 is definitely noisier than the 980 or the 7xx series due to fan placement and lack of hooded airflow internally. Also, with RHEL6 and BIOS A03 the OptiPlex 990 will shut down but will hang when restarted. In order for it to reboot cleanly it is necessary to add reboot=pci to the kernel options in /etc/grub.conf.

Nonparsing Apache Log Messages Solved

I set up a new Red Hat Enterprise Linux 5 webserver but the nightly emails from logwatch were complaining about logs not being parsed correctly:

--------------------- httpd Begin ------------------------

(Only the first ten are printed; there were a total of 188)
"1.2.3.4 - - [30/Mar/2011:01:49:18 -0500] \"GET /foo/bar/modules/node/node.css HTTP/1.1\" 200 678 \"https://www.example.edu/foo/bar/baz\"
...


I looked in /etc/logwatch/conf/services but there was no httpd.conf in there. In fact, there was nothing there, so the speculation by logwatch as to what the problem was was appreciated but not helpful.

Looking more carefully at the log entries, it became apparent that they were all from referrers that were requests over SSL. When I pored over the logging configuration for the virtual hosts on this box, I discovered that the CustomLog directive for the host listening on 443 was very slightly different than the default host; yet they were writing to the same log file.

Making the two CustomLog directives identical fixed the problem. (I also changed the configuration so that https requests were written to a separate file.)

Logrotate configuration file for Apache Solr on RHEL5

I'm running Solr using Jetty and logging to /var/log/solr.log.

This log is growing so I added a logrotate entry at /etc/logrotate.d/solr as follows:

# Logrotate file for /var/log/solr.log
/var/log/solr.log {
rotate 5
compress
size 10M
weekly
postrotate
/sbin/service solr restart
endscript
}


Since this is a new file, the SELinux security context is off for /etc/logrotate.d/solr:

# ls -lZ /etc/logrotate.d
-rw-r--r-- root root system_u:object_r:etc_t acpid
-rw-r--r-- root root system_u:object_r:etc_t conman
-rw-r--r-- root root system_u:object_r:etc_t cups
-rw-r--r-- root root system_u:object_r:etc_t mgetty
-rw-r--r-- root root system_u:object_r:etc_t ppp
-rw-r--r-- root root system_u:object_r:etc_t psacct
-rw-r--r-- root root system_u:object_r:etc_t rpm
-rw-r--r-- root root system_u:object_r:etc_t setroubleshoot
-rw-r--r-- root root root:object_r:etc_t solr
...


That can be fixed by telling SELinux to relabel the solr file using the correct context for /etc/logrotate.d/:

# chcon --reference=/etc/logrotate.d solr
[root@plpt300 logrotate.d]# ls -lZ
-rw-r--r-- root root system_u:object_r:etc_t acpid
-rw-r--r-- root root system_u:object_r:etc_t conman
-rw-r--r-- root root system_u:object_r:etc_t cups
-rw-r--r-- root root system_u:object_r:etc_t mgetty
-rw-r--r-- root root system_u:object_r:etc_t ppp
-rw-r--r-- root root system_u:object_r:etc_t psacct
-rw-r--r-- root root system_u:object_r:etc_t rpm
-rw-r--r-- root root system_u:object_r:etc_t setroubleshoot
-rw-r--r-- root root system_u:object_r:etc_t solr
...


The new config file can be checked using the debug flag for logrotate, e.g.:

# logrotate --debug /etc/logrotate.d/solr
reading config file /etc/logrotate.d/solr
reading config info for /var/log/solr.log

rotating pattern: /var/log/solr.log weekly (5 rotations)
empty log files are rotated, old logs are removed
considering log /var/log/solr.log
log does not need rotating
not running postrotate script, since no logs were rotated


Installing PHP 5.2 (or 5.3) on RHEL5 the easy way

Just replace the stock PHP with one from the IUS Community project.


wget http://dl.iuscommunity.org/pub/ius/stable/Redhat/5.5/x86_64/ius-release-1.0-6.ius.el5.noarch.rpm
rpm -Uvh ius-release-1.0-6.ius.el5.noarch.rpm
yum install yum-plugin-replace
rpm -qa | grep php
yum replace php --replace-with php52
/sbin/service httpd restart


Done!

Moving /tmp to RAM and pleasing SELinux on RHEL5

On a box that had lots of unused memory, I decided to move /tmp to memory. But getting SELinux to stop complaining was a challenge. I would get log entries in /var/log/messages like

kernel: SELinux: security_context_to_sid(system_u:object_r:tmp_t) failed for (dev tmpfs, type tmpfs) errno=-22


Here is the line that I added to /etc/fstab that caused it to mount correctly:

tmpfs /tmp tmpfs defaults,size=4096m,context=system_u:object_r:tmp_t:s0 0


This is what it looks like after a successful boot:

# df -h
Filesystem Size Used Avail Use% Mounted on
...
tmpfs 3.9G 0 3.9G 0% /dev/shm
tmpfs 4.0G 0 4.0G 0% /tmp


SELinux is not entirely happy about the situation, though. It still leaves this log entry:

setroubleshoot: SELinux is preventing the xfs from using potentially mislabeled files (/tmp/.font-unix). For complete SELinux messages. run sealert -l 30ae77a9-fc73-491d-9f84-5e6d2fb982ec

Hyper-V + Opteron 6100 + RHEL5 = Kernel panic - not syncing: Fatal exception

I'm trying out Hyper-V on Windows Server 2008 R2. My first attempt is to create a VM running Red Hat 5.5. This is what I get after the initial screen:

The VM has 2GB of memory which should be plenty (I upped it to 4GB and got the same result). I get a slightly different message when using 32-bit instead of 64-bit:

I'm using a legacy virtual network adapter as the documentation suggests. Hints are welcome. This is on a Dell PowerEdge R815 with two AMD Opteron 6100's and 40GB of RAM.

Update: Thanks to Heine for pointing me to this bug.

Setting Up GeoIP Lookup in awstats on RHEL5

I assume awstats (and awstats-selinux) are already installed.

Step 1: Install the most excellent GeoLite data from MaxMind

mkdir /usr/local/share/GeoIP
cd /usr/local/share/GeoIP
wget http://geolite.maxmind.com/download/geoip/database/GeoLiteCountry/GeoIP.dat.gz
wget http://geolite.maxmind.com/download/geoip/database/GeoLiteCity.dat.gz
wget http://geolite.maxmind.com/download/geoip/database/asnum/GeoIPASNum.dat.gz
gunzip *gz


Step 2: Install the GeoIP C API

There will be a dependency on zlib so make sure zlib-devel is installed:

yum install zlib-devel

Now download and compile:

wget http://geolite.maxmind.com/download/geoip/api/c/GeoIP-1.4.6.tar.gz
tar xvzf GeoIP-1.4.6.tar.gz
cd GeoIP-1.4.6
./configure
make
make check
sudo make install


Make it possible to link against the geoip library:

sudo echo '/usr/local/lib' > /etc/ld.so.conf.d/geoip.conf
sudo ldconfig

Step 3: Install the Geo::IP Perl Module

cd ~
wget http://geolite.maxmind.com/download/geoip/api/perl/Geo-IP-1.38.tar.gz
tar xzvf Geo-IP-1.38.tar.gz
cd Geo-IP-1.38
perl Makefile.PL LIBS='-L/usr/local/lib'
make
make test
sudo make install


Step 4: Enable the GeoIP Plugin

Edit /etc/awstats/awstats.yourdomain.conf to add these lines:

LoadPlugin="geoip GEOIP_STANDARD /usr/local/share/GeoIP/GeoIP.dat"
LoadPlugin="geoip_city_maxmind GEOIP_STANDARD /usr/local/share/GeoIP/GeoLiteCity.dat"
LoadPlugin="geoip_org_maxmind GEOIP_STANDARD /usr/local/share/GeoIP/GeoIPASNum.dat"


Reference: GeoIP Information for AWStats
Speedup Awstats by using GeoIP instead of DNS Lookups

Installing Microsoft TrueType fonts on RHEL5, Step by Step

I'm doing some experimentation with JpGraph on Red Hat Enterprise Linux 5.5. So I wanted to install some TrueType fonts. Here's a step-by-step of how I did it, following the superb instructions given here.

Installed the rpm-build utility.

# yum install rpm-build


Then installed the cabextract utility, available on EPEL.

# yum install cabextract


Downloaded the latest spec file for msttcorefonts:

curl -O http://corefonts.sourceforge.net/msttcorefonts-2.0-1.spec


Created a ~/.rpmmacros file containing one line:
%_topdir %(echo $HOME)/rpmbuild

Then I followed the instructions to create necessary directories:

$ mkdir -p ~/rpmbuild/BUILD
$ mkdir -p ~/rpmbuild/RPMS/noarch


Built the rpm:

$ rpmbuild -bb msttcorefonts-2.0-1.noarch.rpm


For me, it stopped once during the build with a 500 error from Sourceforge:

Resolving hivelocity.dl.sourceforge.net... 74.50.111.26
Connecting to hivelocity.dl.sourceforge.net|74.50.111.26|:80... connected.
HTTP request sent, awaiting response... 500 Internal Server Error
2010-04-12 13:26:10 ERROR 500: Internal Server Error.

RPM build errors:
Bad exit status from /var/tmp/rpm-tmp.51326 (%prep)


I simply ran the previous command again and it worked.

Installed the rpm as root:

# rpm -ivh /home/john/rpmbuild/RPMS/noarch/msttcorefonts-2.0-1.noarch.rpm


I now see that the fonts are in /usr/share/

# find /usr -name *ttf -print

...
/usr/share/fonts/msttcorefonts/georgiaz.ttf
/usr/share/fonts/msttcorefonts/comic.ttf
/usr/share/fonts/msttcorefonts/times.ttf
/usr/share/fonts/msttcorefonts/timesbi.ttf
/usr/share/fonts/msttcorefonts/impact.ttf
/usr/share/fonts/msttcorefonts/verdanaz.ttf
/usr/share/fonts/msttcorefonts/georgiab.ttf
/usr/share/fonts/msttcorefonts/arialbi.ttf
/usr/share/fonts/msttcorefonts/arialbd.ttf
/usr/share/fonts/msttcorefonts/ariblk.ttf
/usr/share/fonts/msttcorefonts/trebucbd.ttf
/usr/share/fonts/msttcorefonts/couri.ttf
/usr/share/fonts/msttcorefonts/trebuc.ttf
/usr/share/fonts/msttcorefonts/timesi.ttf
/usr/share/fonts/msttcorefonts/verdanab.ttf
/usr/share/fonts/msttcorefonts/timesbd.ttf
/usr/share/fonts/msttcorefonts/courbd.ttf
/usr/share/fonts/msttcorefonts/arial.ttf
/usr/share/fonts/msttcorefonts/georgiai.ttf
/usr/share/fonts/msttcorefonts/ariali.ttf
/usr/share/fonts/msttcorefonts/courbi.ttf
/usr/share/fonts/msttcorefonts/comicbd.ttf
/usr/share/fonts/msttcorefonts/webdings.ttf
/usr/share/fonts/msttcorefonts/trebucbi.ttf
/usr/share/fonts/msttcorefonts/georgia.ttf
/usr/share/fonts/msttcorefonts/trebucit.ttf
/usr/share/fonts/msttcorefonts/cour.ttf
/usr/share/fonts/msttcorefonts/tahoma.ttf
/usr/share/fonts/msttcorefonts/verdanai.ttf
/usr/share/fonts/msttcorefonts/verdana.ttf
/usr/share/fonts/msttcorefonts/andalemo.ttf
...


Reference: An easy way to install Microsoft's TrueType core fonts on linux

Solution to 100% CPU Usage by Linux Guest on VMWare Fusion

As part of my testing setup, I have an Intel Mac Pro with Mac OS X 10.6 Server (which runs with the 64-bit kernel) on which I run VMWare Fusion 3.0.1 and several Red Hat virtual machines.

I noticed that even at idle, each VM was taking up a high amount (like 100%!) of a CPU core. Additionally, on one VM top was displaying in near-real-time, which was kind of neat but I doubt the intended behavior. Because of this, I suspected the time management in the kernel was off.

Sure enough, Timekeeping Best Practices for Linux Guests has some hints, and for more information than you'll ever want, try Timekeeping in VMWare Virtual Machines (I was particularly interested in the Clocksource Kernels section).

Making the following modification to /etc/grub.conf on RHEL5 brought my CPU usage down from 100% to barely noticeable:

Before:

kernel /vmlinuz-2.6.18-164.11.1.el5 ro root=/dev/VolGroup00/LogVol00 rhgb quiet

After:

kernel /vmlinuz-2.6.18-164.11.1.el5 ro root=/dev/VolGroup00/LogVol00 rhgb divider=10

I took out quiet because I like to see what's happening when the system boots.

The Note on RHEL 5.4 or CentOS and divider=10 mentions that you do not need this for RHEL 5.4 for accurate timekeeping, but you do need it to prevent the excessive CPU use.

I also modified /etc/ntp.conf as described in the above article, adding

tinker panic 0

to the top of the file and commenting out the following lines:

# Undisciplined Local Clock. This is a fake driver intended for backup
# and when no outside source of synchronized time is available.
#server 127.127.1.0
#fudge 127.127.1.0 stratum 10


My VMs are down from 100 percent CPU use to practically zero. Think of the energy savings!