Setting Up GeoIP Lookup in awstats on RHEL5

I assume awstats (and awstats-selinux) are already installed.

Step 1: Install the most excellent GeoLite data from MaxMind

mkdir /usr/local/share/GeoIP
cd /usr/local/share/GeoIP
wget http://geolite.maxmind.com/download/geoip/database/GeoLiteCountry/GeoIP.dat.gz
wget http://geolite.maxmind.com/download/geoip/database/GeoLiteCity.dat.gz
wget http://geolite.maxmind.com/download/geoip/database/asnum/GeoIPASNum.dat.gz
gunzip *gz


Step 2: Install the GeoIP C API

There will be a dependency on zlib so make sure zlib-devel is installed:

yum install zlib-devel

Now download and compile:

wget http://geolite.maxmind.com/download/geoip/api/c/GeoIP-1.4.6.tar.gz
tar xvzf GeoIP-1.4.6.tar.gz
cd GeoIP-1.4.6
./configure
make
make check
sudo make install


Make it possible to link against the geoip library:

sudo echo '/usr/local/lib' > /etc/ld.so.conf.d/geoip.conf
sudo ldconfig

Step 3: Install the Geo::IP Perl Module

cd ~
wget http://geolite.maxmind.com/download/geoip/api/perl/Geo-IP-1.38.tar.gz
tar xzvf Geo-IP-1.38.tar.gz
cd Geo-IP-1.38
perl Makefile.PL LIBS='-L/usr/local/lib'
make
make test
sudo make install


Step 4: Enable the GeoIP Plugin

Edit /etc/awstats/awstats.yourdomain.conf to add these lines:

LoadPlugin="geoip GEOIP_STANDARD /usr/local/share/GeoIP/GeoIP.dat"
LoadPlugin="geoip_city_maxmind GEOIP_STANDARD /usr/local/share/GeoIP/GeoLiteCity.dat"
LoadPlugin="geoip_org_maxmind GEOIP_STANDARD /usr/local/share/GeoIP/GeoIPASNum.dat"


Reference: GeoIP Information for AWStats
Speedup Awstats by using GeoIP instead of DNS Lookups

Cacti Segfaulting on CentOS5

I was getting Cacti set up and was having trouble with PHP. Specifically, it appeared that enabling the Cacti cron job caused a PHP segfault. I looked in /etc/cron.d/cacti to see what was being run:

cat /etc/cron.d/cacti
#*/5 * * * * cacti /usr/bin/php /usr/share/cacti/poller.php > /dev/null 2>&1


Then I ran it from the command line myself:

# php -v
PHP 5.1.6 (cli) (built: Jan 13 2010 17:13:05)
Copyright (c) 1997-2006 The PHP Group
Zend Engine v2.1.0, Copyright (c) 1998-2006 Zend Technologies

# php /usr/share/cacti/poller.php
Segmentation fault


Sure enough, an ugly error was recorded in /var/log/messages:

kernel: php[5250]: segfault at bf29ce64 ip b788cbf3 sp bf29cdc4 error 6 in libc-2.5.so[b7851000+13f000]


It turns out that this is what happens when you bork the permissions on cacti's MySQL table. After issuing a command like the following to MySQL, my segfault went away.

GRANT ALL ON cacti.* TO 'cactiusername'@'localhost' IDENTIFIED BY 'secret';


Performance presentation, Des Moines Web Geeks

Matt Nuzum, the webmaster of ubuntu.com, invited me to speak at the Des Moines Web Geeks meeting last night. The venue, Impromptu Studios, was a bit hard to find. There were no signs for Impromptu Studios, but after lurking in dark alleyways for a while I eventually discovered what should have been obvious from the beginning: enter the furniture store with the "Closed" sign on the front and go up the stairs past the disused lavatory with a sign on the door saying "Beware of the Leopard."

The group was great and we talked for about an hour and half about bottlenecks and optimization. I even went home with a Django book!

Since this was a diverse group I tried to talk about things that everyone has to deal with (bottlenecks for storage, network, memory and CPU) rather than anything Drupal-specific.

As promised, last night's slides: Performance, Scalability on the Server Side (PDF, 1MB).

Installing mytop on RHEL5

I recently installed mytop on a new 64-bit Red Hat Enterprise Linux 5.4 server. Here are my notes.

The mytop program requires two Perl modules. So let's make sure those are installed first. The perl-DBI package is usually installed as a dependency already. Check if it's there:


# yum list installed *DBI
Loaded plugins: rhnplugin, security
Installed Packages
perl-DBI.x86_64 1.52-2.el5 installed


Install TermReadKey module, also known as Term::ReadKey. I am installing this from the EPEL repository, which I regard as very safe (in terms of compatibility, longevity and maintenance). If you're not set up with EPEL, you can do

# rpm -Uvh http://download.fedora.redhat.com/pub/epel/5/i386/epel-release-5-4.noarch.rpm

Anyway, installing Term::ReadKey from EPEL:


# yum install perl-TermReadKey
Loaded plugins: rhnplugin, security
Setting up Install Process
Resolving Dependencies
--> Running transaction check
---> Package perl-TermReadKey.x86_64 0:2.30-4.el5 set to be updated
--> Finished Dependency Resolution
...
Running rpm_check_debug
Running Transaction Test
Finished Transaction Test
Transaction Test Succeeded
Running Transaction
Installing : perl-TermReadKey 1/1

Installed:
perl-TermReadKey.x86_64 0:2.30-4.el5


Now it's time to get mytop.


# wget http://jeremy.zawodny.com/mysql/mytop/mytop-1.6.tar.gz
# tar zxvf mytop-1.6.tar.gz
# cd mytop-1.6
# perl Makefile.PL
Writing Makefile for mytop

# which mytop
/usr/bin/mytop


yum update missing dependencies on RHEL5

I was getting the following dependency errors on a Red Hat Enterprise Linux 5 box. This was a stock box that has a RHEL subscription and had not used any yum repositories other than Red Hat's own official repositories.


# yum update
Loaded plugins: rhnplugin, security
Skipping security plugin, no data
Setting up Update Process
Resolving Dependencies
Skipping security plugin, no data
--> Running transaction check
---> Package libsoup.i386 0:2.2.98-2.el5_3.1 set to be updated
---> Package ghostscript.i386 0:8.15.2-9.4.el5_3.4 set to be updated
---> Package evolution-data-server.i386 0:1.12.3-10.el5_3.3 set to be updated
---> Package libpng.i386 2:1.2.10-7.1.el5_3.2 set to be updated
--> Processing Dependency: libebook-1.2.so.9 for package: gnome-panel
--> Processing Dependency: libebook-1.2.so.9 for package: control-center
--> Processing Dependency: libecal-1.2.so.7 for package: gnome-panel
--> Processing Dependency: libedataserver-1.2.so.7 for package: gnome-panel
--> Processing Dependency: libedataserver-1.2.so.7 for package: control-center
--> Processing Dependency: libedataserverui-1.2.so.8 for package: gnome-panel
--> Processing Dependency: evolution-data-server >= 1.1.4 for package: gnome-panel
---> Package device-mapper-multipath.i386 0:0.4.7-23.el5_3.1 set to be updated
---> Package lcms.i386 0:1.18-0.1.beta1.el5_3.2 set to be updated
---> Package kpartx.i386 0:0.4.7-23.el5_3.1 set to be updated
--> Processing Dependency: libgs.so.8 for package: ImageMagick
--> Processing Dependency: ghostscript for package: libgnomeprint22
--> Processing Dependency: ghostscript for package: ghostscript-fonts
---> Package firefox.i386 0:3.0.7-1.el5 set to be updated
---> Package xulrunner.i386 0:1.9.0.7-1.el5 set to be updated
---> Package curl.i386 0:7.15.5-2.1.el5_3.4 set to be updated
--> Finished Dependency Resolution
libgnomeprint22-2.12.1-10.el5.i386 from installed has depsolving problems
--> Missing Dependency: ghostscript is needed by package libgnomeprint22-2.12.1-10.el5.i386 (installed)
gnome-panel-2.16.1-7.el5.i386 from installed has depsolving problems
--> Missing Dependency: libecal-1.2.so.7 is needed by package gnome-panel-2.16.1-7.el5.i386 (installed)
1:control-center-2.16.0-16.el5.i386 from installed has depsolving problems
--> Missing Dependency: libedataserver-1.2.so.7 is needed by package 1:control-center-2.16.0-16.el5.i386 (installed)
gnome-panel-2.16.1-7.el5.i386 from installed has depsolving problems
--> Missing Dependency: libedataserver-1.2.so.7 is needed by package gnome-panel-2.16.1-7.el5.i386 (installed)
gnome-panel-2.16.1-7.el5.i386 from installed has depsolving problems
--> Missing Dependency: libebook-1.2.so.9 is needed by package gnome-panel-2.16.1-7.el5.i386 (installed)
gnome-panel-2.16.1-7.el5.i386 from installed has depsolving problems
--> Missing Dependency: libedataserverui-1.2.so.8 is needed by package gnome-panel-2.16.1-7.el5.i386 (installed)
ghostscript-fonts-5.50-13.1.1.noarch from installed has depsolving problems
--> Missing Dependency: ghostscript is needed by package ghostscript-fonts-5.50-13.1.1.noarch (installed)
gnome-panel-2.16.1-7.el5.i386 from installed has depsolving problems
--> Missing Dependency: evolution-data-server >= 1.1.4 is needed by package gnome-panel-2.16.1-7.el5.i386 (installed)
ImageMagick-6.2.8.0-4.el5_1.1.i386 from installed has depsolving problems
--> Missing Dependency: libgs.so.8 is needed by package ImageMagick-6.2.8.0-4.el5_1.1.i386 (installed)
1:control-center-2.16.0-16.el5.i386 from installed has depsolving problems
--> Missing Dependency: libebook-1.2.so.9 is needed by package 1:control-center-2.16.0-16.el5.i386 (installed)
Error: Missing Dependency: libecal-1.2.so.7 is needed by package gnome-panel-2.16.1-7.el5.i386 (installed)
Error: Missing Dependency: ghostscript is needed by package ghostscript-fonts-5.50-13.1.1.noarch (installed)
Error: Missing Dependency: ghostscript is needed by package libgnomeprint22-2.12.1-10.el5.i386 (installed)
Error: Missing Dependency: libebook-1.2.so.9 is needed by package 1:control-center-2.16.0-16.el5.i386 (installed)
Error: Missing Dependency: libgs.so.8 is needed by package ImageMagick-6.2.8.0-4.el5_1.1.i386 (installed)
Error: Missing Dependency: libebook-1.2.so.9 is needed by package gnome-panel-2.16.1-7.el5.i386 (installed)
Error: Missing Dependency: libedataserver-1.2.so.7 is needed by package 1:control-center-2.16.0-16.el5.i386 (installed)
Error: Missing Dependency: libedataserverui-1.2.so.8 is needed by package gnome-panel-2.16.1-7.el5.i386 (installed)
Error: Missing Dependency: evolution-data-server >= 1.1.4 is needed by package gnome-panel-2.16.1-7.el5.i386 (installed)
Error: Missing Dependency: libedataserver-1.2.so.7 is needed by package gnome-panel-2.16.1-7.el5.i386 (installed)


The solution was to clean out yum's local caches with:

yum clean all


and then

yum update


I suspect that yum clean headers alone might have done the trick, but I was too quick on the trigger with yum clean all.

Update: Anonymous says that yum clean dbcache works.

Installing APC on RHEL5

Here's how to install APC on RHEL5.
# yum install php-pear php-devel httpd-devel
# pecl install apc
# echo "extension=apc.so" > /etc/php.d/apc.ini
# service httpd restart


Check for the apc section in your phpinfo() page. If it's not there:

# tail /var/log/httpd/error_log
[Tue Aug 12 15:43:59 2008] [notice] Digest: done
PHP Warning: PHP Startup: Unable to load dynamic library '/usr/lib/php/modules/apc.so' - /usr/lib/php/modules/apc.so: cannot open shared object file: Permission denied in Unknown on line 0


SELinux is preventing the extension from loading, I'll bet. Let's check:

# tail /var/log/messages
Aug 12 15:42:40 mybox yum: Installed: httpd-devel - 2.2.3-11.el5_1.3.i386
Aug 12 15:44:01 mybox setroubleshoot: SELinux is preventing the httpd from using potentially mislabeled files (./apc.so). For complete SELinux messages. run sealert -l 9e4bbfa8-327b-4bb2-94df-f154045a1ef1


Let's view the security contexts for our PHP extensions:

# ls -Z /usr/lib/php/modules
-rwxr-xr-x root root root:object_r:tmp_t apc.so
-rwxr-xr-x root root system_u:object_r:textrel_shlib_t dbase.so
-rwxr-xr-x root root system_u:object_r:textrel_shlib_t gd.so
...


Yup. Notice how the security context is missing (it's tmp_t for apc.so). Let's fix that.

# restorecon /usr/lib/php/modules/apc.so
]# ls -Z /usr/lib/php/modules
-rwxr-xr-x root root system_u:object_r:textrel_shlib_t apc.so
-rwxr-xr-x root root system_u:object_r:textrel_shlib_t dbase.so
-rwxr-xr-x root root system_u:object_r:textrel_shlib_t gd.so
...


Now let's try again:

# service httpd restart


Yay!

P.S. On x86_64, apc.so is in /usr/lib64/php/modules.

TCP Tuning for Busy Apache Webserver on CentOS5

Recently I was in a situation where a very busy webserver was not responding. Strangely, top showed plenty of CPU available. The server was essentially just sitting there. What do do?

Upon further investigation, it turned out that the network queue was saturated. So many incoming connections were being attempted that they were falling off the end. Some TCP tuning was in order. Fortunately the server was not memory-starved so allocating more memory to the network stack was not a problem. Here's what ended up in /etc/sysctl.conf and turned the server back into a faithful workhorse.

# Kernel tuning settings for CentOS5,
# busy webserver with lots of free memory.

# Increase option memory buffers
net.core.optmem_max=20480


There are plenty of other sysctl options to tune, but the above made the most difference.

And netstat -s is your friend.